From patchwork Fri Apr 17 05:57:20 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "H.J. Lu" X-Patchwork-Id: 133237 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [127.0.0.1]) by sourceware.org (Postfix) with ESMTP id BE87C4BAD150 for ; Fri, 17 Apr 2026 05:58:31 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org BE87C4BAD150 Authentication-Results: sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=YseTugbr X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pj1-x1033.google.com (mail-pj1-x1033.google.com [IPv6:2607:f8b0:4864:20::1033]) by sourceware.org (Postfix) with ESMTPS id 9F18C4BAD142 for ; Fri, 17 Apr 2026 05:58:00 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 9F18C4BAD142 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 9F18C4BAD142 Authentication-Results: server2.sourceware.org; arc=pass smtp.remote-ip=2607:f8b0:4864:20::1033 ARC-Seal: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1776405480; cv=pass; b=we3I93yvpN81tGQxyp9k7IuIc84zsJOgLkrzCKD2XMQT/arLwrCQqqB3AI8zKZakvrxxS7GXjp5tOvPfHsgYFaCQb1OA2yF1M3inrcBUcy2WzfgW81FkfmYwXP4RqgpFPgzwPgnl8j7pdPJiPj7O71pbAwNEU/6CRNZaz41yYgs= ARC-Message-Signature: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1776405480; c=relaxed/simple; bh=gGRZgzJpX/aNlo7/COUSXl0gcwaZsm/CIIsL5uo3PNo=; h=DKIM-Signature:MIME-Version:From:Date:Message-ID:Subject:To; b=je6WtI+Mbf6fk+vo5/YHS8XCEHz3XHQeMH9yi+f+Txj0sdsvZ0oE6oQMOoFsF37vTAWspAmHNRXGuqrO59/shIcf92bI289/5hvW9reor1SSNYhR4u74DZlI94JRf/mejIlBylbRN2zxyNkEWy4JKR6LZtlA07YvxyuXaIR5yko= ARC-Authentication-Results: i=2; server2.sourceware.org DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 9F18C4BAD142 Received: by mail-pj1-x1033.google.com with SMTP id 98e67ed59e1d1-35da2d35eccso241838a91.0 for ; Thu, 16 Apr 2026 22:58:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1776405479; cv=none; d=google.com; s=arc-20240605; b=WYFMvQ9zBE9lwa7kcfW20CFJxyckba1jgaRrQN9yoMHZiFHGOCmcQnIO+MgyoWuR43 kBto7gTWXS7q9PBGHsQotBHNFj5Np2RtJGQ57Z8ug9WotEiWSL7hDsHJ8wFihQKCV+to a6h15kfgYOukDH1QI4lSckuCh85C8Z6LvV6iUDiKJJH1QLftefOPNIzGNmsefN63+8lZ ZfnMUq2It2hrHPqrTVUj0ywt+LqMFbSqXRP+uNoNZsQSCJrkNXzHO7DP0gcjsp7i6IUW WsnhYVg63l65OE5BkT3WACnDn7Y1HZyUbpbi1u97zxe5MQ54WfdHS4xy1qxFS1+sgbcv T9aQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :dkim-signature; bh=1+SHVo2vq1dPcrQj6EOCj4+06pdk9+QPvVBQsbWKcL8=; fh=/eXwpVNxBegZfrfOyFtz/n2HvoReWcLst0S3pIk5NP0=; b=Sb9XAYRcy2hcH0ysQX/219UFgDWg3tAnmhjB4h34E3r48xmBQUf9p5Of5NofUmNr8s niiBos2wXM0U9Lqq0yfRo/Y1fsLnyPAAe+Cq0NpGvikUdVnAvbjgb6kNaRcSgCO+m++9 hHQ2hkXvE7+u3eew/uhfd7pFMYCPo6NfaPsvAZQzy0Tl44Mj5+aqFIBTBAViXB3rBJk4 1mb7MdmQPeesEFkDGSWkcaqQ3deRp7zKGuoJ4puAnTVqRHMit+sLq2cVqEdQsbyCgW/Y eOq9me3XUxl9k+oGlHQ39Me4PZE14PHV5i1UrktRc01eXy7mdH4ma3rxuQN1u5dwZipc 6FXg==; darn=sourceware.org ARC-Authentication-Results: i=1; mx.google.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1776405479; x=1777010279; darn=sourceware.org; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :from:to:cc:subject:date:message-id:reply-to; bh=1+SHVo2vq1dPcrQj6EOCj4+06pdk9+QPvVBQsbWKcL8=; b=YseTugbrnOQUCKWZ6rKMVSa5MQX9aFc4jN/kZl05i2EdmErQGN/G7yTZiSUrH+kVkV cWpt1JaoP94K5prk9jgikQw4vW5zb+O3/kjUKXLMZYT2NIBoaiM9SfLzl7ZVmiazMkFP dZDiVarNX+Q3qaIXCnPAn8/BDQVD5HSYYzimf4glDE03tehN/xHnzxXi9+k0ITX/4rmX 7ieHDQfegCgaSFcpzdF449eCEQindNM0wi2Z4H0jsbeevl904XSOVmPSGPtkm96a+sPT 7hAnnqM5h+Go8I3RhH6PLbwirdq1cMlSyAkflMZKXO6vX9aJQTPWtgdaAuu3lwKlj4ha WavQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1776405479; x=1777010279; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :x-gm-gg:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=1+SHVo2vq1dPcrQj6EOCj4+06pdk9+QPvVBQsbWKcL8=; b=lZ8cdHr3k/nQYyR8ZCymBX8aOlD+CASyI8ctMbzW2sTDbzm1MMQzGSYko3j2I+Trl1 CEoVeYdyvg6XoZogZLp3xZziMiFP7yyHOrKGGbMRxO6JiZoGZw1apGS5UYPELpCP6+qi 2jBBgDDQKz7C1rEpai4Ne1qBW+iynrwwlawvfsVGBc/BTMRbKqVcjKTyX2rqZmb64xJr I/j1gXR/RYmem2jZIlWPOWXilBCzPIjIjoXhVZc/kBPMgw4yfNOoYSwGRkVAbQ5XHRkX SFxdpSzh+I3RZKU6H+I3VtC3+rbwL4ff/rwQaJT1W50rsUiF4kQTwA1VSU9O5sjqKVZP VOpw== X-Gm-Message-State: AOJu0YwdCLIdFMiEJ2/dW4uFwQ2Q5ZG3lB/tvPYdzWuIrCYBEd4eL5Sh gSwE9Otxm1xhqwGLHKmxoHSwNuAkaVB1Dvoh4jKWaXHgFU6i51V1+WTDw/CORcNSsBjcJuJ+3ZU +NBcuNqnNmbLqgnvH0MDqjlKbg0f/nMZ52ThdwfA= X-Gm-Gg: AeBDievXwJFGl3SHP7+lMdam+FMvC2K81YCJsbNs4d0Mn73l7/iIaZgaU094T2n8b5B V12gzbmEylhlIqCviXGhGOhPn4PTbZiDSHZPzcH/49edDiFQMAdsg9YYrJCSE/baHS6PkkZ+szz CihtrIsnKT3e4Mpx8Brp+TgbMY7hiaInXC+UbvWt7pR4AymkE3UK88HB9yExtLPIoXdNX6NFjmy qiV0KC3XoyjuiVge4VemLkWeSoMHIJpoDkZfjbb2Si0i2HhP03uELkr0FmeRRafmCPc6sad2lDy b9gUg89vJrwivs8jllM= X-Received: by 2002:a17:90b:518c:b0:35b:e52a:6fe5 with SMTP id 98e67ed59e1d1-361403cbdecmr1473030a91.5.1776405479246; Thu, 16 Apr 2026 22:57:59 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: "H.J. Lu" Date: Fri, 17 Apr 2026 13:57:20 +0800 X-Gm-Features: AQROBzDWdqHGg-M4kS1XNQYWUn_uwsptQA0MqyoJaeG-NK2dFINJXSJUrAdxWUY Message-ID: Subject: [PATCH v2 1/2] Check if linker supports -z separate-code To: GNU C Library , WANG Rui X-Spam-Status: No, score=-3010.8 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces~patchwork=sourceware.org@sourceware.org On Fri, Apr 17, 2026 at 12:09 PM H.J. Lu wrote: > > Set have-separate-code to yes if linker supports -z separate-code. > Unchanged from v1. From c9b366a4b5afa30e0f4dd5022ab74cbc47207ae7 Mon Sep 17 00:00:00 2001 From: "H.J. Lu" Date: Tue, 14 Apr 2026 11:06:31 +0800 Subject: [PATCH v2] x86: Zero ZMM16-31 when zeroing all call used registers When zeroing all call used registers with AVX512F enabled, zero ZMM16-31 explicitly since vzeroall doesn't touch ZMM16-31. Also add a test for zeroing all call used registers with both AVX512F and APX enabled. gcc/ PR target/124876 * config/i386/i386.cc (ix86_zero_call_used_regs): Zero ZMM16-31 if needed. gcc/testsuite/ PR target/124876 * gcc.target/i386/zero-scratch-regs-23.c: Scan vpxord on ZMM16-31. * gcc.target/i386/zero-scratch-regs-33.c: New test. Signed-off-by: H.J. Lu --- gcc/config/i386/i386.cc | 11 ++++ .../gcc.target/i386/zero-scratch-regs-23.c | 16 +++++ .../gcc.target/i386/zero-scratch-regs-33.c | 60 +++++++++++++++++++ 3 files changed, 87 insertions(+) create mode 100644 gcc/testsuite/gcc.target/i386/zero-scratch-regs-33.c diff --git a/gcc/config/i386/i386.cc b/gcc/config/i386/i386.cc index dd017bc7ac3..c93ea383668 100644 --- a/gcc/config/i386/i386.cc +++ b/gcc/config/i386/i386.cc @@ -4078,6 +4078,17 @@ ix86_zero_call_used_regs (HARD_REG_SET need_zeroed_hardregs) { emit_insn (zero_all_vec_insn); all_sse_zeroed = true; + if (TARGET_64BIT && TARGET_AVX512F) + { + rtx zero = CONST0_RTX (V4SFmode); + for (unsigned int regno = XMM16_REG; + regno <= XMM31_REG; + regno++) + { + rtx reg = gen_rtx_REG (V4SFmode, regno); + emit_move_insn (reg, zero); + } + } } /* mm/st registers are shared registers set, we should follow the following diff --git a/gcc/testsuite/gcc.target/i386/zero-scratch-regs-23.c b/gcc/testsuite/gcc.target/i386/zero-scratch-regs-23.c index a3285bed8a0..397893faa6c 100644 --- a/gcc/testsuite/gcc.target/i386/zero-scratch-regs-23.c +++ b/gcc/testsuite/gcc.target/i386/zero-scratch-regs-23.c @@ -7,6 +7,22 @@ foo (void) } /* { dg-final { scan-assembler "vzeroall" } } */ +/* { dg-final { scan-assembler "vpxord\[ \t\]+%zmm16, %zmm16, %zmm16" { target { ! ia32 } } } } */ +/* { dg-final { scan-assembler "vpxord\[ \t\]+%zmm17, %zmm17, %zmm17" { target { ! ia32 } } } } */ +/* { dg-final { scan-assembler "vpxord\[ \t\]+%zmm18, %zmm18, %zmm18" { target { ! ia32 } } } } */ +/* { dg-final { scan-assembler "vpxord\[ \t\]+%zmm19, %zmm19, %zmm19" { target { ! ia32 } } } } */ +/* { dg-final { scan-assembler "vpxord\[ \t\]+%zmm20, %zmm20, %zmm20" { target { ! ia32 } } } } */ +/* { dg-final { scan-assembler "vpxord\[ \t\]+%zmm21, %zmm21, %zmm21" { target { ! ia32 } } } } */ +/* { dg-final { scan-assembler "vpxord\[ \t\]+%zmm22, %zmm22, %zmm22" { target { ! ia32 } } } } */ +/* { dg-final { scan-assembler "vpxord\[ \t\]+%zmm23, %zmm23, %zmm23" { target { ! ia32 } } } } */ +/* { dg-final { scan-assembler "vpxord\[ \t\]+%zmm24, %zmm24, %zmm24" { target { ! ia32 } } } } */ +/* { dg-final { scan-assembler "vpxord\[ \t\]+%zmm25, %zmm25, %zmm25" { target { ! ia32 } } } } */ +/* { dg-final { scan-assembler "vpxord\[ \t\]+%zmm26, %zmm26, %zmm26" { target { ! ia32 } } } } */ +/* { dg-final { scan-assembler "vpxord\[ \t\]+%zmm27, %zmm27, %zmm27" { target { ! ia32 } } } } */ +/* { dg-final { scan-assembler "vpxord\[ \t\]+%zmm28, %zmm28, %zmm28" { target { ! ia32 } } } } */ +/* { dg-final { scan-assembler "vpxord\[ \t\]+%zmm29, %zmm29, %zmm29" { target { ! ia32 } } } } */ +/* { dg-final { scan-assembler "vpxord\[ \t\]+%zmm30, %zmm30, %zmm30" { target { ! ia32 } } } } */ +/* { dg-final { scan-assembler "vpxord\[ \t\]+%zmm31, %zmm31, %zmm31" { target { ! ia32 } } } } */ /* { dg-final { scan-assembler-times "fldz" 8 } } */ /* { dg-final { scan-assembler-times "fstp\[ \t\]+%st\\(0\\)" 8 } } */ /* { dg-final { scan-assembler-not "%xmm" } } */ diff --git a/gcc/testsuite/gcc.target/i386/zero-scratch-regs-33.c b/gcc/testsuite/gcc.target/i386/zero-scratch-regs-33.c new file mode 100644 index 00000000000..f40fe2a5377 --- /dev/null +++ b/gcc/testsuite/gcc.target/i386/zero-scratch-regs-33.c @@ -0,0 +1,60 @@ +/* { dg-do compile { target { *-*-linux* && { ! ia32 } } } } */ +/* { dg-options "-O2 -fzero-call-used-regs=all -march=corei7 -mavx512f -mapxf" } */ + +void +foo (void) +{ +} + +/* { dg-final { scan-assembler "vzeroall" } } */ +/* { dg-final { scan-assembler "vpxord\[ \t\]+%zmm16, %zmm16, %zmm16" } } */ +/* { dg-final { scan-assembler "vpxord\[ \t\]+%zmm17, %zmm17, %zmm17" } } */ +/* { dg-final { scan-assembler "vpxord\[ \t\]+%zmm18, %zmm18, %zmm18" } } */ +/* { dg-final { scan-assembler "vpxord\[ \t\]+%zmm19, %zmm19, %zmm19" } } */ +/* { dg-final { scan-assembler "vpxord\[ \t\]+%zmm20, %zmm20, %zmm20" } } */ +/* { dg-final { scan-assembler "vpxord\[ \t\]+%zmm21, %zmm21, %zmm21" } } */ +/* { dg-final { scan-assembler "vpxord\[ \t\]+%zmm22, %zmm22, %zmm22" } } */ +/* { dg-final { scan-assembler "vpxord\[ \t\]+%zmm23, %zmm23, %zmm23" } } */ +/* { dg-final { scan-assembler "vpxord\[ \t\]+%zmm24, %zmm24, %zmm24" } } */ +/* { dg-final { scan-assembler "vpxord\[ \t\]+%zmm25, %zmm25, %zmm25" } } */ +/* { dg-final { scan-assembler "vpxord\[ \t\]+%zmm26, %zmm26, %zmm26" } } */ +/* { dg-final { scan-assembler "vpxord\[ \t\]+%zmm27, %zmm27, %zmm27" } } */ +/* { dg-final { scan-assembler "vpxord\[ \t\]+%zmm28, %zmm28, %zmm28" } } */ +/* { dg-final { scan-assembler "vpxord\[ \t\]+%zmm29, %zmm29, %zmm29" } } */ +/* { dg-final { scan-assembler "vpxord\[ \t\]+%zmm30, %zmm30, %zmm30" } } */ +/* { dg-final { scan-assembler "vpxord\[ \t\]+%zmm31, %zmm31, %zmm31" } } */ +/* { dg-final { scan-assembler-times "fldz" 8 } } */ +/* { dg-final { scan-assembler-times "fstp\[ \t\]+%st\\(0\\)" 8 } } */ +/* { dg-final { scan-assembler "xorl\[ \t\]+%eax, %eax" } } */ +/* { dg-final { scan-assembler "xorl\[ \t\]+%edx, %edx" } } */ +/* { dg-final { scan-assembler "xorl\[ \t\]+%ecx, %ecx" } } */ +/* { dg-final { scan-assembler "xorl\[ \t\]+%esi, %esi" } } */ +/* { dg-final { scan-assembler "xorl\[ \t\]+%edi, %edi" } } */ +/* { dg-final { scan-assembler "xorl\[ \t\]+%r8d, %r8d" } } */ +/* { dg-final { scan-assembler "xorl\[ \t\]+%r9d, %r9d" } } */ +/* { dg-final { scan-assembler "xorl\[ \t\]+%r10d, %r10d" } } */ +/* { dg-final { scan-assembler "xorl\[ \t\]+%r11d, %r11d" } } */ +/* { dg-final { scan-assembler "kxorw\[ \t\]+%k0, %k0, %k0" } } */ +/* { dg-final { scan-assembler "kxorw\[ \t\]+%k1, %k1, %k1" } } */ +/* { dg-final { scan-assembler "kxorw\[ \t\]+%k2, %k2, %k2" } } */ +/* { dg-final { scan-assembler "kxorw\[ \t\]+%k3, %k3, %k3" } } */ +/* { dg-final { scan-assembler "kxorw\[ \t\]+%k4, %k4, %k4" } } */ +/* { dg-final { scan-assembler "kxorw\[ \t\]+%k5, %k5, %k5" } } */ +/* { dg-final { scan-assembler "kxorw\[ \t\]+%k6, %k6, %k6" } } */ +/* { dg-final { scan-assembler "kxorw\[ \t\]+%k7, %k7, %k7" } } */ +/* { dg-final { scan-assembler "xorl\[ \t\]+%r16d, %r16d" } } */ +/* { dg-final { scan-assembler "xorl\[ \t\]+%r17d, %r17d" } } */ +/* { dg-final { scan-assembler "xorl\[ \t\]+%r18d, %r18d" } } */ +/* { dg-final { scan-assembler "xorl\[ \t\]+%r19d, %r19d" } } */ +/* { dg-final { scan-assembler "xorl\[ \t\]+%r20d, %r20d" } } */ +/* { dg-final { scan-assembler "xorl\[ \t\]+%r21d, %r21d" } } */ +/* { dg-final { scan-assembler "xorl\[ \t\]+%r22d, %r22d" } } */ +/* { dg-final { scan-assembler "xorl\[ \t\]+%r23d, %r23d" } } */ +/* { dg-final { scan-assembler "xorl\[ \t\]+%r24d, %r24d" } } */ +/* { dg-final { scan-assembler "xorl\[ \t\]+%r25d, %r25d" } } */ +/* { dg-final { scan-assembler "xorl\[ \t\]+%r26d, %r26d" } } */ +/* { dg-final { scan-assembler "xorl\[ \t\]+%r27d, %r27d" } } */ +/* { dg-final { scan-assembler "xorl\[ \t\]+%r28d, %r28d" } } */ +/* { dg-final { scan-assembler "xorl\[ \t\]+%r29d, %r29d" } } */ +/* { dg-final { scan-assembler "xorl\[ \t\]+%r30d, %r30d" } } */ +/* { dg-final { scan-assembler "xorl\[ \t\]+%r31d, %r31d" } } */ -- 2.53.0 From patchwork Fri Apr 17 06:00:04 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "H.J. Lu" X-Patchwork-Id: 133238 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [127.0.0.1]) by sourceware.org (Postfix) with ESMTP id 1558C4BAD142 for ; Fri, 17 Apr 2026 06:01:24 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 1558C4BAD142 Authentication-Results: sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=QPBiIXez X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pj1-x102b.google.com (mail-pj1-x102b.google.com [IPv6:2607:f8b0:4864:20::102b]) by sourceware.org (Postfix) with ESMTPS id 4EFC54BAD142 for ; Fri, 17 Apr 2026 06:00:44 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 4EFC54BAD142 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 4EFC54BAD142 Authentication-Results: server2.sourceware.org; arc=pass smtp.remote-ip=2607:f8b0:4864:20::102b ARC-Seal: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1776405644; cv=pass; b=JD6JjtKEUCJdn5BUUL9F5LCLLhA908aHrGd7r4Mhe+cbXYg4v3I+OJLt86bf83CfZfj0cs9UNhdnVp10qa+CBqxv8tgj8+1Bxl7+yWHj8VuKdtuU7uuPJBsS8R58G1BZzRq7I+CVfSek+5xSpuhv5WED3DAJsfv7yx7biQcYewI= ARC-Message-Signature: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1776405644; c=relaxed/simple; bh=D07ys0AdFuVd7TKDCqGgb6ipzdsd1CWTK7GvBsJSB7s=; h=DKIM-Signature:MIME-Version:From:Date:Message-ID:Subject:To; b=ciuIv9+8Z8Z/f5fK9h2eGlQeggIRHSXCJ8YNFWsZ1rlWa5wwHkWBOn3MSME+EFbYB8cdCR44vvNGDqwoMiYWK8CxH1hmmSfhQOudqTokhIyKSgpxwP6sTH9y90qrI0GnjimAnxjmvAEZal8jrcmKP6HfSqXR6fqundUh6nhZVUA= ARC-Authentication-Results: i=2; server2.sourceware.org DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 4EFC54BAD142 Received: by mail-pj1-x102b.google.com with SMTP id 98e67ed59e1d1-358ed696623so156271a91.0 for ; Thu, 16 Apr 2026 23:00:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1776405643; cv=none; d=google.com; s=arc-20240605; b=cMzh2MEusQsMK7U2vIiU4vAv3Lk3vkYQ6LAHCJSGIC4OQwP7bQOjw7PCd76D74y0hQ /4Rv1BW/1+2QenGl1YBUsq3uxEkbeZ7yhI9gB8+PX7sHVRhjYnmw/PUwVk8idWuk35xS 4MBM4eML466pi5TIDDSVWAQ8ACuavV3ECsdwgqtNEHEl4/1e6yjTyb/BnyeazuIAlwr0 2suCQxN5o3yQzS0JUWYM0C/1dP/rlsSLxlfmeCWh8w90WlTW1Jlyz6RIpi4HRjjECQLZ alMF4PPofGaYkC5EsuWvcXkVwZskfOVLY47XUW8l4UXjefZ0qGpMHcrKF/ShxliDXMRd AumA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :dkim-signature; bh=YpfLccHRYkh1RjuDISIeAUpcZvjP0IhHzuuqAqrBD50=; fh=/eXwpVNxBegZfrfOyFtz/n2HvoReWcLst0S3pIk5NP0=; b=LKm5b8tlH8Zb/Yn+T1K30zHB0dCAoIVRFtpsM2D1gzM/8ybRrA09G35ofaKWWtGQNR ZS56zJ0VK+2F3HouHL/8kzyHBKKemBrUWBsFRZMnvVUbkqborbuI7Ln46UPoIpFqusvp wScJZ/PROHU8i5/RedcXX7ozc52C3VQj+MUaiHhdkXNxx8EjDTiBU1D6VviqSIvxoieE eJ7HRbWVJU2V0+WORLns2hmMqd4zwMfGDpMIe/HUwZexx1mXIL9YneDznM3bP+KzEShi KPUMvAHnvwO2wnbQLPQCYDNNiVdwaZsspeg5zJBy/TQ/rjdZJ+hQ9hbCn9kWizap1mwW m3Jw==; darn=sourceware.org ARC-Authentication-Results: i=1; mx.google.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1776405643; x=1777010443; darn=sourceware.org; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :from:to:cc:subject:date:message-id:reply-to; bh=YpfLccHRYkh1RjuDISIeAUpcZvjP0IhHzuuqAqrBD50=; b=QPBiIXez5oED52Py/uLNTw8uWt0oJYZzw4NfrXRx88zexs37sZnzjcCaAkF68zoqMu nLcySSsZK0T6Hr2wkQVOslblAaqwKXCz8WnCH66eGPV6zsZRyiTmapxT2j4xOluziYyg 8uid+qQYmIWPfTfcywh8LhSrYW6LMJ3Kd9smyNBqtX0ghaM7dHuAmcG40l8Q9NfKZRHO UaBcs5aUvIEUJ2v2IHji6kSUgX3OZglbicKW9/IVQRLFTvYLN7rmq6spyLYl6v/YrjJn CPXM++bL/U/731lC4tPX0qstvoxa0ERk+RLR+X59Ct/U6Ig2Ar9JVWIaOBibOd+v7wYz Dz/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1776405643; x=1777010443; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :x-gm-gg:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=YpfLccHRYkh1RjuDISIeAUpcZvjP0IhHzuuqAqrBD50=; b=eYOZ5XtohdFNoxoBEt4TffmEODCpsPVRiTJlP5CUlIB+cFKsQglt/fTKlJRr9iGWiM oWTd/NyGA5vjq5aGHZxmNHRSWo2J8a4cxLkXup9P0Pp7FKY13f/Dh/02bRJWliBh1kqn 13mLAXfjPpnn3DAuSFvH38uuC6CulGA/XrrjIRnXt8gba9zflXo5BvDgPfJxw0voREAd ckFkF4jxushmVntkOL1yZngkGASzc8AIMMieQDFnRfGBAJbuYFPBBu9nQU+iqu5KkR3u GhgTGUz9Hqq8szh891DNjSbBUGHyOYZRNhJ9UoVtB71uu8Wl/VNNs7Mqyqz02ab/tDCf LAOg== X-Gm-Message-State: AOJu0YxCRZxwXz5f7MuhKstN7DFdTShPnl0wWqqAel3+/lEmyiLqjCfo MyBrqeeGr/zWRKzltA4V3LWAQsmXEJ7EvUqkGWCiA7H2GX03HMy7liZTALsGtfpTlNlQTv8labH UiEndxjc90w93Xazey0tYXH5UumF8CnBgkeQG5JpVtQ== X-Gm-Gg: AeBDiesKJfHR/vSvOmwGE8j6kBn6ATUD0DBlysczNO1Oh/8etB5qHJ1ox2mZtGEXmi9 SQ20kcPQPsn9u5V7A/gSW0U4wKt7RL/WRyTzNt2XYFJvZ57cTnT8uWCXJDw/WarTTCUmfu1BsZk Fm0ex9H1knTL49ZxkTJWKi4L4FVP7XncVAXK9arTuij1L7VtrxP5i5eVc+kj+KXKvoZ13xvtNpH MH4t8TwK1GH1lf+ce+gdYOiGEiaQUWk9uePeo8uzOxjrmAfeoW2QAAZuQtr+hjC0yWE/ah/sVc/ otrFLBR85NKhw/sxK8mfo7QBuTeLJg== X-Received: by 2002:a17:90b:164a:b0:35f:b572:ece3 with SMTP id 98e67ed59e1d1-361403bd001mr1555070a91.6.1776405642580; Thu, 16 Apr 2026 23:00:42 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: "H.J. Lu" Date: Fri, 17 Apr 2026 14:00:04 +0800 X-Gm-Features: AQROBzAzEFj5ruxMavL1_N5cGfjZrZAgzz-0tfWE0eoRPmqNg_6Oy5gfpYHrcU4 Message-ID: Subject: [PATCH v2 2/2] elf: Support THP segment load with madvise enabled THP To: GNU C Library , WANG Rui X-Spam-Status: No, score=-3010.6 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces~patchwork=sourceware.org@sourceware.org On Fri, Apr 17, 2026 at 12:11 PM H.J. Lu wrote: > > The current THP segment load approach works only when THP is enabled > with always in the kernel. If THP is enabled with madvise in the > kernel, to enable THP segment load in an application, madvise should > be called with MADV_HUGEPAGE on all THP eligible PT_LOAD segments: > > 1. Remove _dl_map_segment_align. Call _dl_map_segment_adjust to adjust > alignments of PT_LOAD segments, which marks and aligns only THP eligible > PT_LOAD segments to THP page size when loading an object. This fixes > BZ #34079. > 2. Rename DL_MAP_DEFAULT_THP_PAGESIZE to DL_DEFAULT_THP_PAGESIZE. If > DL_DEFAULT_THP_PAGESIZE is defined, assume that THP is enabled with > madvise in the kernel. Otherwise, call __get_thp_size to get THP page > size and call __get_thp_mode to get THP mode. > 3. Call _dl_executable_postprocess in rtld_setup_main_map for dynamic > executables and in LIBC_START_MAIN for static executables, which calls > madvise with MADV_HUGEPAGE on all THP eligible PT_LOAD segments in > executable. This fixes BZ #34080 for both dynamic and static executables. > 4. Call _dl_postprocess_loadcmd_extra in _dl_postprocess_loadcmd, which > calls madvise with MADV_HUGEPAGE on all THP eligible PT_LOAD segments > when loading an object after they have been mapped in. This fixes > BZ #34080 for shared objects. > 5. Add tests to verify that large executable PT_LOAD segments in > executables and shared libraries are mapped at addresses aligned to THP > page size when the kernel is configured to use THP in "always" mode or > "madvise" mode by inspecting /proc/self/maps to check that the mapping > address is aligned to THP page size reported by the kernel [1]. Also > verify that madvise is called with MADV_HUGEPAGE when the glibc tunable > glibc.elf.thp=1 is used and madvise isn't called with MADV_HUGEPAGE when > the glibc tunable glibc.elf.thp=0 is used. > > Skip these tests if THP page size cannot be determined or if THP is not > enabled in "always" mode nor "madvise" mode. > > Quote WANG Rui : > > From benchmarking a clang build of the Linux kernel on x86_64 with > your patch in THP madvise mode, I observed that iTLB misses were > reduced, similar to what we see in THP always mode. > > [1] This is developed by WANG Rui . > > Signed-off-by: H.J. Lu > Co-developed-by: WANG Rui Changes from v1: 1. Rebased against the v10 THP test patch from WANG Rui . 2. Keep DL_MAP_DEFAULT_THP_PAGESIZE. From f5e4ac66240c78ac6e8bc7a52d5763a227420928 Mon Sep 17 00:00:00 2001 From: WANG Rui Date: Fri, 17 Apr 2026 11:55:56 +0800 Subject: [PATCH v2 2/2] elf: Support THP segment load with madvise enabled THP The current THP segment load approach works only when THP is enabled with always in the kernel. If THP is enabled with madvise in the kernel, to enable THP segment load in an application, madvise should be called with MADV_HUGEPAGE on all THP eligible PT_LOAD segments: 1. Remove _dl_map_segment_align. Call _dl_map_segment_adjust to adjust alignments of PT_LOAD segments, which marks and aligns only THP eligible PT_LOAD segments to THP page size when loading an object. This fixes BZ #34079. 2. Call _dl_executable_postprocess in rtld_setup_main_map for dynamic executables and in LIBC_START_MAIN for static executables, which calls madvise with MADV_HUGEPAGE on all THP eligible PT_LOAD segments in executable. This fixes BZ #34080 for both dynamic and static executables. 3. Call _dl_postprocess_loadcmd_extra in _dl_postprocess_loadcmd, which calls madvise with MADV_HUGEPAGE on all THP eligible PT_LOAD segments when loading an object after they have been mapped in. This fixes BZ #34080 for shared objects. 4. Add tests to verify that large executable PT_LOAD segments in executables and shared libraries are mapped at addresses aligned to THP page size when the kernel is configured to use THP in "always" mode or "madvise" mode by inspecting /proc/self/maps to check that the mapping address is aligned to THP page size reported by the kernel [1]. Also verify that madvise is called with MADV_HUGEPAGE when the glibc tunable glibc.elf.thp=1 is used and madvise isn't called with MADV_HUGEPAGE when the glibc tunable glibc.elf.thp=0 is used. Skip these tests if THP page size cannot be determined or if THP is not enabled in "always" mode nor "madvise" mode. Quote WANG Rui : From benchmarking a clang build of the Linux kernel on x86_64 with your patch in THP madvise mode, I observed that iTLB misses were reduced, similar to what we see in THP always mode. [1] This is developed by WANG Rui . Signed-off-by: H.J. Lu Co-developed-by: WANG Rui --- csu/libc-start.c | 4 + elf/dl-load.c | 10 +- elf/dl-load.h | 5 + elf/dl-support.c | 6 + elf/rtld.c | 17 +-- sysdeps/generic/dl-exec-post.h | 34 +++++ ...{dl-map-segment-align.h => dl-load-post.h} | 7 +- sysdeps/generic/dl-map-segment-adjust.h | 30 ++++ sysdeps/generic/hugepages.h | 4 +- sysdeps/generic/ldsodefs.h | 11 ++ sysdeps/unix/sysv/linux/Makefile | 124 +++++++++++++++- sysdeps/unix/sysv/linux/dl-exec-post.h | 95 +++++++++++++ sysdeps/unix/sysv/linux/dl-load-post.h | 32 +++++ .../unix/sysv/linux/dl-map-segment-adjust.c | 67 +++++++++ ...egment-align.h => dl-map-segment-adjust.h} | 10 +- .../unix/sysv/linux/dl-map-segment-align.c | 55 -------- sysdeps/unix/sysv/linux/ldsodefs.h | 3 + sysdeps/unix/sysv/linux/loongarch/Makefile | 3 + ...{dl-map-segment-align.h => dl-exec-post.h} | 4 +- sysdeps/unix/sysv/linux/strace-tst-thp.sh | 56 ++++++++ .../unix/sysv/linux/tst-thp-1-no-s-code-pde.c | 19 +++ .../sysv/linux/tst-thp-1-no-s-code-static.c | 19 +++ sysdeps/unix/sysv/linux/tst-thp-1-no-s-code.c | 19 +++ sysdeps/unix/sysv/linux/tst-thp-1-pde.c | 19 +++ sysdeps/unix/sysv/linux/tst-thp-1-static.c | 19 +++ sysdeps/unix/sysv/linux/tst-thp-1.c | 28 ++++ sysdeps/unix/sysv/linux/tst-thp-align-check.h | 132 ++++++++++++++++++ sysdeps/unix/sysv/linux/tst-thp-align.c | 36 +++++ sysdeps/unix/sysv/linux/tst-thp-size-mod.S | 24 ++++ sysdeps/unix/sysv/linux/x86/dl-exec-post.h | 22 +++ 30 files changed, 829 insertions(+), 85 deletions(-) create mode 100644 sysdeps/generic/dl-exec-post.h rename sysdeps/generic/{dl-map-segment-align.h => dl-load-post.h} (83%) create mode 100644 sysdeps/generic/dl-map-segment-adjust.h create mode 100644 sysdeps/unix/sysv/linux/dl-exec-post.h create mode 100644 sysdeps/unix/sysv/linux/dl-load-post.h create mode 100644 sysdeps/unix/sysv/linux/dl-map-segment-adjust.c rename sysdeps/unix/sysv/linux/{dl-map-segment-align.h => dl-map-segment-adjust.h} (79%) delete mode 100644 sysdeps/unix/sysv/linux/dl-map-segment-align.c rename sysdeps/unix/sysv/linux/loongarch/lp64/{dl-map-segment-align.h => dl-exec-post.h} (90%) create mode 100644 sysdeps/unix/sysv/linux/strace-tst-thp.sh create mode 100644 sysdeps/unix/sysv/linux/tst-thp-1-no-s-code-pde.c create mode 100644 sysdeps/unix/sysv/linux/tst-thp-1-no-s-code-static.c create mode 100644 sysdeps/unix/sysv/linux/tst-thp-1-no-s-code.c create mode 100644 sysdeps/unix/sysv/linux/tst-thp-1-pde.c create mode 100644 sysdeps/unix/sysv/linux/tst-thp-1-static.c create mode 100644 sysdeps/unix/sysv/linux/tst-thp-1.c create mode 100644 sysdeps/unix/sysv/linux/tst-thp-align-check.h create mode 100644 sysdeps/unix/sysv/linux/tst-thp-align.c create mode 100644 sysdeps/unix/sysv/linux/tst-thp-size-mod.S create mode 100644 sysdeps/unix/sysv/linux/x86/dl-exec-post.h diff --git a/csu/libc-start.c b/csu/libc-start.c index 1c58561bce..be60831a42 100644 --- a/csu/libc-start.c +++ b/csu/libc-start.c @@ -205,6 +205,7 @@ call_fini (void *unused) #endif /* !SHARED */ #include +#include STATIC int LIBC_START_MAIN (int (*main) (int, char **, char ** MAIN_AUXVEC_DECL), @@ -305,6 +306,9 @@ LIBC_START_MAIN (int (*main) (int, char **, char ** MAIN_AUXVEC_DECL), __pointer_chk_guard_local = pointer_chk_guard; # endif + struct link_map *main_map = _dl_get_dl_main_map (); + _dl_executable_postprocess (main_map, GL(dl_phdr), GL(dl_phnum)); + #endif /* !SHARED */ /* Register the destructor of the dynamic linker if there is any. */ diff --git a/elf/dl-load.c b/elf/dl-load.c index 48575fff06..800b7ca1a6 100644 --- a/elf/dl-load.c +++ b/elf/dl-load.c @@ -71,7 +71,7 @@ struct filebuf #include #include #include -#include +#include #include #include #include @@ -1173,8 +1173,8 @@ _dl_map_object_from_fd (const char *name, const char *origname, int fd, /* Optimize a common case. */ c->prot = pf_to_prot (ph->p_flags); - /* Architecture-specific adjustment of segment alignment. */ - p_align_max = _dl_map_segment_align (c, p_align_max); + /* Initialize flags. */ + c->flags = 0; break; case PT_TLS: @@ -1229,9 +1229,7 @@ _dl_map_object_from_fd (const char *name, const char *origname, int fd, goto lose; } - /* Align all PT_LOAD segments to the maximum p_align. */ - for (size_t i = 0; i < nloadcmds; i++) - loadcmds[i].mapalign = p_align_max; + _dl_map_segment_adjust (loadcmds, &nloadcmds, p_align_max); /* dlopen of an executable is not valid because it is not possible to perform proper relocations, handle static TLS, or run the diff --git a/elf/dl-load.h b/elf/dl-load.h index 897c4034c5..55edf0323c 100644 --- a/elf/dl-load.h +++ b/elf/dl-load.h @@ -78,8 +78,11 @@ struct loadcmd ElfW(Addr) mapstart, mapend, dataend, allocend, mapalign; ElfW(Off) mapoff; int prot; /* PROT_* bits. */ + /* May be used by _dl_postprocess_loadcmd_extra. */ + unsigned int flags; }; +#include /* This is a subroutine of _dl_map_segments. It should be called for each load command, some time after L->l_addr has been set correctly. It is @@ -95,6 +98,8 @@ _dl_postprocess_loadcmd (struct link_map *l, const ElfW(Ehdr) *header, /* Found the program header in this segment. */ l->l_phdr = (void *) (uintptr_t) (c->mapstart + header->e_phoff - c->mapoff); + + _dl_postprocess_loadcmd_extra (l, c); } diff --git a/elf/dl-support.c b/elf/dl-support.c index 0508d6113b..420e418b40 100644 --- a/elf/dl-support.c +++ b/elf/dl-support.c @@ -179,6 +179,12 @@ int _dl_stack_cache_lock; #endif struct dl_scope_free_list *_dl_scope_free_list; +#ifdef HAVE_THP +int _dl_thp_control = -1; +enum thp_mode_t _dl_thp_mode; +size_t _dl_thp_pagesize; +#endif + #ifdef NEED_DL_SYSINFO /* Needed for improved syscall handling on at least x86/Linux. NB: Don't initialize it here to avoid RELATIVE relocation in static PIE. */ diff --git a/elf/rtld.c b/elf/rtld.c index e926ec73e4..fe8d6f87e3 100644 --- a/elf/rtld.c +++ b/elf/rtld.c @@ -52,6 +52,7 @@ #include #include #include +#include #include @@ -323,6 +324,9 @@ struct rtld_global _rtld_global = /* Generally the default presumption without further information is an * executable stack but this is not true for all platforms. */ ._dl_stack_prot_flags = DEFAULT_STACK_PROT_PERMS, +#ifdef HAVE_THP + ._dl_thp_control = -1, +#endif #ifdef _LIBC_REENTRANT ._dl_load_lock = _RTLD_LOCK_RECURSIVE_INITIALIZER, ._dl_load_write_lock = _RTLD_LOCK_RECURSIVE_INITIALIZER, @@ -1209,14 +1213,8 @@ rtld_setup_main_map (struct link_map *main_map) main_map->l_relro_size = ph->p_memsz; break; } - /* Process program headers again, but scan them backwards since - PT_GNU_PROPERTY is close to the end of program headers. */ - for (const ElfW(Phdr) *ph = &phdr[phnum]; ph != phdr; --ph) - if (ph[-1].p_type == PT_GNU_PROPERTY) - { - _dl_process_pt_gnu_property (main_map, -1, &ph[-1]); - break; - } + + _dl_executable_postprocess (main_map, phdr, phnum); /* Adjust the address of the TLS initialization image in case the executable is actually an ET_DYN object. */ @@ -1589,6 +1587,9 @@ dl_main (const ElfW(Phdr) *phdr, { RTLD_TIMING_VAR (start); rtld_timer_start (&start); +#ifdef HAVE_THP + _dl_get_thp_config (); +#endif _dl_map_object (NULL, rtld_progname, lt_executable, 0, __RTLD_OPENEXEC, LM_ID_BASE); rtld_timer_stop (&load_time, start); diff --git a/sysdeps/generic/dl-exec-post.h b/sysdeps/generic/dl-exec-post.h new file mode 100644 index 0000000000..f5dcdc093a --- /dev/null +++ b/sysdeps/generic/dl-exec-post.h @@ -0,0 +1,34 @@ +/* _dl_executable_postprocess. Generic version. + Copyright (C) 2026 Free Software Foundation, Inc. + Copyright The GNU Toolchain Authors. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +static inline void +_dl_executable_postprocess (struct link_map *main_map, + const ElfW(Phdr) *phdr, ElfW(Word) phnum) +{ +#ifdef SHARED + /* Process program headers again, but scan them backwards since + PT_GNU_PROPERTY is close to the end of program headers. */ + for (const ElfW(Phdr) *ph = &phdr[phnum]; ph != phdr; --ph) + if (ph[-1].p_type == PT_GNU_PROPERTY) + { + _dl_process_pt_gnu_property (main_map, -1, &ph[-1]); + break; + } +#endif +} diff --git a/sysdeps/generic/dl-map-segment-align.h b/sysdeps/generic/dl-load-post.h similarity index 83% rename from sysdeps/generic/dl-map-segment-align.h rename to sysdeps/generic/dl-load-post.h index f4a671f25f..bfaad8ea7c 100644 --- a/sysdeps/generic/dl-map-segment-align.h +++ b/sysdeps/generic/dl-load-post.h @@ -1,4 +1,4 @@ -/* _dl_map_segment_align. Generic version. +/* _dl_postprocess_loadcmd_extra. Generic version. Copyright (C) 2026 Free Software Foundation, Inc. Copyright The GNU Toolchain Authors. This file is part of the GNU C Library. @@ -19,8 +19,7 @@ #include -static inline ElfW(Addr) -_dl_map_segment_align (const struct loadcmd *c, ElfW(Addr) p_align_max) +static inline void +_dl_postprocess_loadcmd_extra (struct link_map *l, const struct loadcmd *c) { - return p_align_max; } diff --git a/sysdeps/generic/dl-map-segment-adjust.h b/sysdeps/generic/dl-map-segment-adjust.h new file mode 100644 index 0000000000..b48057ec26 --- /dev/null +++ b/sysdeps/generic/dl-map-segment-adjust.h @@ -0,0 +1,30 @@ +/* _dl_map_segment_adjust. Generic version. + Copyright (C) 2026 Free Software Foundation, Inc. + Copyright The GNU Toolchain Authors. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include + +static inline void +_dl_map_segment_adjust (struct loadcmd *load_cmds, + size_t *n_load_cmds, ElfW(Addr) p_align_max) +{ + /* Align all PT_LOAD segments to the maximum p_align. */ + size_t n = *n_load_cmds; + for (size_t i = 0; i < n; i++) + load_cmds[i].mapalign = p_align_max; +} diff --git a/sysdeps/generic/hugepages.h b/sysdeps/generic/hugepages.h index 5fc9b5c8de..df9ec4920b 100644 --- a/sysdeps/generic/hugepages.h +++ b/sysdeps/generic/hugepages.h @@ -26,10 +26,10 @@ unsigned long int __get_thp_size (void) attribute_hidden; enum thp_mode_t { + thp_mode_not_supported = 0, thp_mode_always, thp_mode_madvise, - thp_mode_never, - thp_mode_not_supported + thp_mode_never }; enum thp_mode_t __get_thp_mode (void) attribute_hidden; diff --git a/sysdeps/generic/ldsodefs.h b/sysdeps/generic/ldsodefs.h index 15c4659853..36f2e88ae4 100644 --- a/sysdeps/generic/ldsodefs.h +++ b/sysdeps/generic/ldsodefs.h @@ -39,6 +39,7 @@ #include #include #include +#include __BEGIN_DECLS @@ -477,6 +478,16 @@ struct rtld_global EXTERN struct __pthread **_dl_pthread_threads; __mach_rwlock_define (EXTERN, _dl_pthread_threads_lock) #endif +#ifdef HAVE_THP + /* The THP segment load control: + > 0: Enabled by GLIBC_TUNABLES=glibc.elf.thp=1. + 0: Disabled by GLIBC_TUNABLES=glibc.elf.thp=0. + < 0: To be enabled or disabled by GLIBC_TUNABLES. */ + EXTERN int _dl_thp_control; + + EXTERN enum thp_mode_t _dl_thp_mode; + EXTERN size_t _dl_thp_pagesize; +#endif #ifdef SHARED }; # define __rtld_global_attribute__ diff --git a/sysdeps/unix/sysv/linux/Makefile b/sysdeps/unix/sysv/linux/Makefile index b06afbdeaa..41d321c564 100644 --- a/sysdeps/unix/sysv/linux/Makefile +++ b/sysdeps/unix/sysv/linux/Makefile @@ -673,7 +673,7 @@ endif ifeq ($(subdir),elf) dl-routines += \ - dl-map-segment-align \ + dl-map-segment-adjust \ dl-rseq-symbols \ # dl-routines @@ -698,13 +698,19 @@ $(objpfx)pldd: $(objpfx)xmalloc.o tests += \ tst-rseq-tls-range \ tst-rseq-tls-range-4096 \ + tst-thp-1 \ + tst-thp-1-pde \ + tst-thp-1-static \ + tst-thp-align \ # tests tests-static += \ tst-rseq-tls-range-4096-static \ tst-rseq-tls-range-static \ + tst-thp-1-static \ # tests-static modules-names += \ tst-rseq-tls-range-mod \ + tst-thp-size-mod \ # modules-names CFLAGS-tst-rseq-tls-range.c += -DMAIN_TLS_ALIGN=4 CFLAGS-tst-rseq-tls-range-4096.c += -DMAIN_TLS_ALIGN=4096 @@ -724,6 +730,122 @@ tests-special += \ $(objpfx)tst-nolink-libc-2.out \ # tests-special endif + +ifndef THP-PAGE-SIZE +# Align PT_LOAD segments in THP tests to THP page size so that kernel will +# map PIE to the address aligned to THP page size. Default THP page size +# to 2MB which can be overridden in Makefile in subdirectories. +THP-PAGE-SIZE = 0x200000 +endif + +THP-PAGE-SIZE-LDFLAGS = -Wl,-z,max-page-size=$(THP-PAGE-SIZE) + +LDFLAGS-tst-thp-size-mod.so = $(THP-PAGE-SIZE-LDFLAGS) +tst-thp-align-ENV = GLIBC_TUNABLES=glibc.elf.thp=1 +$(objpfx)tst-thp-align.out: $(objpfx)tst-thp-size-mod.so + +ifeq ($(have-separate-code),yes) +tests += \ + tst-thp-1-no-s-code \ + tst-thp-1-no-s-code-pde \ + tst-thp-1-no-s-code-static \ +# tests +tests-static += \ + tst-thp-1-no-s-code-static \ +# tests-static + +LDFLAGS-tst-thp-1 = -Wl,-z,separate-code $(THP-PAGE-SIZE-LDFLAGS) +LDFLAGS-tst-thp-1-pde = -Wl,-z,separate-code $(THP-PAGE-SIZE-LDFLAGS) +LDFLAGS-tst-thp-1-static = -Wl,-z,separate-code $(THP-PAGE-SIZE-LDFLAGS) +LDFLAGS-tst-thp-1-no-s-code = -Wl,-z,noseparate-code \ + $(THP-PAGE-SIZE-LDFLAGS) +LDFLAGS-tst-thp-1-no-s-code-pde = -Wl,-z,noseparate-code \ + $(THP-PAGE-SIZE-LDFLAGS) +LDFLAGS-tst-thp-1-no-s-code-static = -Wl,-z,noseparate-code \ + $(THP-PAGE-SIZE-LDFLAGS) + +$(objpfx)tst-thp-1-no-s-code: $(objpfx)tst-thp-size-mod.o +$(objpfx)tst-thp-1-no-s-code-pde: $(objpfx)tst-thp-size-mod.o +$(objpfx)tst-thp-1-no-s-code-static: $(objpfx)tst-thp-size-mod.o + +tst-thp-1-no-s-code-ENV = GLIBC_TUNABLES=glibc.elf.thp=1 +tst-thp-1-no-s-code-pde-ENV = GLIBC_TUNABLES=glibc.elf.thp=1 +tst-thp-1-no-s-code-static-ENV = GLIBC_TUNABLES=glibc.elf.thp=1 + +tst-thp-1-no-s-code-pde-no-pie = yes +endif + +tst-thp-1-ENV = GLIBC_TUNABLES=glibc.elf.thp=1 +tst-thp-1-pde-ENV = GLIBC_TUNABLES=glibc.elf.thp=1 +tst-thp-1-static-ENV = GLIBC_TUNABLES=glibc.elf.thp=1 + +$(objpfx)tst-thp-1: $(objpfx)tst-thp-size-mod.o +$(objpfx)tst-thp-1-pde: $(objpfx)tst-thp-size-mod.o +$(objpfx)tst-thp-1-static: $(objpfx)tst-thp-size-mod.o + +tst-thp-1-pde-no-pie = yes + +thp-kernel-status = $(shell grep madvise /sys/kernel/mm/transparent_hugepage/enabled) +# Verify that madvise is called with MADV_HUGEPAGE when THP is enabled +# under madvise THP kernel. +ifneq ($(findstring [madvise],$(thp-kernel-status)),) +tests-special += \ + $(objpfx)strace-tst-thp-1-disabled.out \ + $(objpfx)strace-tst-thp-1-enabled.out \ + $(objpfx)strace-tst-thp-1-pde-disabled.out \ + $(objpfx)strace-tst-thp-1-pde-enabled.out \ + $(objpfx)strace-tst-thp-1-static-disabled.out \ + $(objpfx)strace-tst-thp-1-static-enabled.out \ +# tests-special + +$(objpfx)strace-tst-thp-1-enabled.out: \ + $(..)sysdeps/unix/sysv/linux/strace-tst-thp.sh $(objpfx)ld.so \ + $(objpfx)tst-thp-1 + $(SHELL) $< $(objpfx)ld.so '$(test-wrapper-env)' \ + '$(run-program-env) GLIBC_TUNABLES=glibc.elf.thp=1' \ + '$(rpath-link)' $(objpfx)tst-thp-1 > $@; + $(evaluate-test) + +$(objpfx)strace-tst-thp-1-disabled.out: \ + $(..)sysdeps/unix/sysv/linux/strace-tst-thp.sh $(objpfx)ld.so \ + $(objpfx)tst-thp-1 + $(SHELL) $< $(objpfx)ld.so '$(test-wrapper-env)' \ + '$(run-program-env) GLIBC_TUNABLES=glibc.elf.thp=0' \ + '$(rpath-link)' $(objpfx)tst-thp-1 > $@; + $(evaluate-test) + +$(objpfx)strace-tst-thp-1-pde-enabled.out: \ + $(..)sysdeps/unix/sysv/linux/strace-tst-thp.sh $(objpfx)ld.so \ + $(objpfx)tst-thp-1-pde + $(SHELL) $< $(objpfx)ld.so '$(test-wrapper-env)' \ + '$(run-program-env) GLIBC_TUNABLES=glibc.elf.thp=1' \ + '$(rpath-link)' $(objpfx)tst-thp-1-pde > $@; \ + $(evaluate-test) + +$(objpfx)strace-tst-thp-1-pde-disabled.out: \ + $(..)sysdeps/unix/sysv/linux/strace-tst-thp.sh $(objpfx)ld.so \ + $(objpfx)tst-thp-1-pde + $(SHELL) $< $(objpfx)ld.so '$(test-wrapper-env)' \ + '$(run-program-env) GLIBC_TUNABLES=glibc.elf.thp=0' \ + '$(rpath-link)' $(objpfx)tst-thp-1-pde > $@; \ + $(evaluate-test) + +$(objpfx)strace-tst-thp-1-static-enabled.out: \ + $(..)sysdeps/unix/sysv/linux/strace-tst-thp.sh $(objpfx)ld.so \ + $(objpfx)tst-thp-1-static + $(SHELL) $< $(objpfx)ld.so '$(test-wrapper-env)' \ + '$(run-program-env) GLIBC_TUNABLES=glibc.elf.thp=1' \ + '$(rpath-link)' $(objpfx)tst-thp-1-static > $@; \ + $(evaluate-test) + +$(objpfx)strace-tst-thp-1-static-disabled.out: \ + $(..)sysdeps/unix/sysv/linux/strace-tst-thp.sh $(objpfx)ld.so \ + $(objpfx)tst-thp-1-static + $(SHELL) $< $(objpfx)ld.so '$(test-wrapper-env)' \ + '$(run-program-env) GLIBC_TUNABLES=glibc.elf.thp=0' \ + '$(rpath-link)' $(objpfx)tst-thp-1-static > $@; \ + $(evaluate-test) +endif endif # $(subdir) == elf ifeq ($(subdir),rt) diff --git a/sysdeps/unix/sysv/linux/dl-exec-post.h b/sysdeps/unix/sysv/linux/dl-exec-post.h new file mode 100644 index 0000000000..1ef1f6556e --- /dev/null +++ b/sysdeps/unix/sysv/linux/dl-exec-post.h @@ -0,0 +1,95 @@ +/* _dl_executable_postprocess. Linux version. + Copyright (C) 2026 Free Software Foundation, Inc. + Copyright The GNU Toolchain Authors. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +static inline void +_dl_get_thp_config (void) +{ + /* Check if there is GLIBC_TUNABLES=glibc.elf.thp=[0|1]. */ + if (TUNABLE_IS_INITIALIZED_FULL (glibc, elf, thp)) + GL(dl_thp_control) = TUNABLE_GET_FULL (glibc, elf, thp, int32_t, + NULL); + + /* Return if THP is disabled by GLIBC_TUNABLES=glibc.elf.thp=0. */ + if (GL(dl_thp_control) == 0) + return; + +#ifdef DL_MAP_DEFAULT_THP_PAGESIZE + /* NB: Accessing /sys/kernel/mm files is quite expensive and the file + may not be accessible in containers. If DL_MAP_DEFAULT_THP_PAGESIZE + is defined, assume THP mode is madvise and always call madvise. + Since madvise is a fast systemcall, it adds only a small overhead + compared to the cost of accessing /sys/kernel/mm files. */ + GL(dl_thp_pagesize) = DL_MAP_DEFAULT_THP_PAGESIZE; + GL(dl_thp_mode) = thp_mode_madvise; +#else + GL(dl_thp_pagesize) = __get_thp_size (); + GL(dl_thp_mode) = __get_thp_mode (); +#endif +} + +static inline void +_dl_executable_postprocess (struct link_map *main_map, + const ElfW(Phdr) *phdr, ElfW(Word) phnum) +{ + if (GL(dl_thp_control) == -1) + _dl_get_thp_config (); + + /* NB: In static executable, PT_GNU_PROPERTY is processed in target + libc-start.h if it is needed by target. When ld.so is used, if + a target doesn't need PT_GNU_PROPERTY, _dl_process_pt_gnu_property + is an empty function. */ +#ifdef SHARED + /* Process program headers again, but scan them backwards since + PT_GNU_PROPERTY is close to the end of program headers. */ + for (const ElfW(Phdr) *ph = &phdr[phnum]; ph != phdr; --ph) + if (ph[-1].p_type == PT_GNU_PROPERTY) + { + _dl_process_pt_gnu_property (main_map, -1, &ph[-1]); + break; + } +#endif + + /* Return if THP segment load isn't enabled. */ + if (GL(dl_thp_control) <= 0) + return; + +#ifndef DL_MAP_DEFAULT_THP_PAGESIZE + /* NB: If DL_MAP_DEFAULT_THP_PAGESIZE is defined, dl_thp_mode is always + set to thp_mode_madvise. */ + if (GL(dl_thp_mode) != thp_mode_madvise) + return; +#endif + + /* When we get here, the main executable have been mapped in. Call + madvise with MADV_HUGEPAGE for all THP eligible PT_LOAD segments. */ + + const ElfW(Phdr) *ph; + + size_t thp_pagesize = GL(dl_thp_pagesize); + + /* Call __madvise if offset and address of the PT_LOAD segment are + aligned to THP page size and it is read-only. */ + for (ph = phdr; ph < &phdr[phnum]; ++ph) + if (ph->p_type == PT_LOAD + && ph->p_memsz >= thp_pagesize + && ((ph->p_vaddr | ph->p_offset) & (thp_pagesize - 1)) == 0 + && (ph->p_flags & (PF_W | PF_R)) == PF_R) + __madvise ((void *) (main_map->l_addr + ph->p_vaddr), + ph->p_memsz, MADV_HUGEPAGE); +} diff --git a/sysdeps/unix/sysv/linux/dl-load-post.h b/sysdeps/unix/sysv/linux/dl-load-post.h new file mode 100644 index 0000000000..b3c4e16bb7 --- /dev/null +++ b/sysdeps/unix/sysv/linux/dl-load-post.h @@ -0,0 +1,32 @@ +/* _dl_postprocess_loadcmd_extra. Linux version. + Copyright (C) 2026 Free Software Foundation, Inc. + Copyright The GNU Toolchain Authors. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +/* Bits in the flags field of struct loadcmd. */ +#define LOADCMD_POST_MADVISE_THP (0x1 << 0) + +/* After L has been mapped in, call madvise with MADV_HUGEPAGE if L is + THP eligible. */ + +static inline void +_dl_postprocess_loadcmd_extra (struct link_map *l, const struct loadcmd *c) +{ + if ((c->flags & LOADCMD_POST_MADVISE_THP) != 0) + __madvise ((void *) (l->l_addr + c->mapstart), + c->mapend - c->mapstart, MADV_HUGEPAGE); +} diff --git a/sysdeps/unix/sysv/linux/dl-map-segment-adjust.c b/sysdeps/unix/sysv/linux/dl-map-segment-adjust.c new file mode 100644 index 0000000000..90c8ea92fe --- /dev/null +++ b/sysdeps/unix/sysv/linux/dl-map-segment-adjust.c @@ -0,0 +1,67 @@ +/* _dl_map_segment_adjust. Linux version. + Copyright (C) 2026 Free Software Foundation, Inc. + Copyright The GNU Toolchain Authors. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include +#include + +/* Set the mapalign field in entries in LOAD_CMDS to align PT_LOAD + segments for THP. P_ALIGN_MAX is the maximum p_align value in all + PT_LOAD segments. */ + +void +_dl_map_segment_adjust (struct loadcmd *load_cmds, size_t *n_load_cmds, + ElfW(Addr) p_align_max) +{ + size_t n = *n_load_cmds; + size_t i; + enum thp_mode_t thp_mode = GL(dl_thp_mode); + size_t thp_pagesize = GL(dl_thp_pagesize); + struct loadcmd *c = load_cmds; + + if (n == 0) + return; + + /* We cap the huge page size at MAX_THP_PAGESIZE to avoid over-aligning + on systems with very large normal pages (like 64K pages with 512M + huge pages). */ + if (GL(dl_thp_control) <= 0 + || p_align_max >= thp_pagesize + || !(thp_mode == thp_mode_always || thp_mode == thp_mode_madvise)) + { + for (i = 0; i < n; i++, c++) + c->mapalign = p_align_max; + return; + } + + /* Set the mapalign field to THP page size only if offset and address + of the segment are aligned to THP page size, it is read-only and + its size >= THP page size. It helps improve THP eligibility and + reduces TLB pressure. */ + for (i = 0; i < n; i++, c++) + if (((c->mapstart | c->mapoff) & (thp_pagesize - 1)) == 0 + && (c->mapend - c->mapstart) >= thp_pagesize + && (c->prot & PROT_WRITE) == 0) + { + c->mapalign = thp_pagesize; + if (thp_mode == thp_mode_madvise) + c->flags = LOADCMD_POST_MADVISE_THP; + } + else + c->mapalign = p_align_max; +} diff --git a/sysdeps/unix/sysv/linux/dl-map-segment-align.h b/sysdeps/unix/sysv/linux/dl-map-segment-adjust.h similarity index 79% rename from sysdeps/unix/sysv/linux/dl-map-segment-align.h rename to sysdeps/unix/sysv/linux/dl-map-segment-adjust.h index d9b05181b7..2fbfc66d48 100644 --- a/sysdeps/unix/sysv/linux/dl-map-segment-align.h +++ b/sysdeps/unix/sysv/linux/dl-map-segment-adjust.h @@ -1,4 +1,4 @@ -/* _dl_map_segment_align. Linux version. +/* _dl_map_segment_adjust. Linux version. Copyright (C) 2026 Free Software Foundation, Inc. Copyright The GNU Toolchain Authors. This file is part of the GNU C Library. @@ -19,9 +19,5 @@ #include -#ifndef DL_MAP_DEFAULT_THP_PAGESIZE -# define DL_MAP_DEFAULT_THP_PAGESIZE 0 -#endif - -extern ElfW (Addr) _dl_map_segment_align - (const struct loadcmd *, ElfW (Addr)) attribute_hidden; +extern void _dl_map_segment_adjust + (struct loadcmd *, size_t *, ElfW(Addr)) attribute_hidden; diff --git a/sysdeps/unix/sysv/linux/dl-map-segment-align.c b/sysdeps/unix/sysv/linux/dl-map-segment-align.c deleted file mode 100644 index a39e74d91b..0000000000 --- a/sysdeps/unix/sysv/linux/dl-map-segment-align.c +++ /dev/null @@ -1,55 +0,0 @@ -/* _dl_map_segment_align. Linux version. - Copyright (C) 2026 Free Software Foundation, Inc. - Copyright The GNU Toolchain Authors. - This file is part of the GNU C Library. - - The GNU C Library is free software; you can redistribute it and/or - modify it under the terms of the GNU Lesser General Public - License as published by the Free Software Foundation; either - version 2.1 of the License, or (at your option) any later version. - - The GNU C Library is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - Lesser General Public License for more details. - - You should have received a copy of the GNU Lesser General Public - License along with the GNU C Library; if not, see - . */ - -#include -#include -#include - -ElfW (Addr) -_dl_map_segment_align (const struct loadcmd *c, ElfW (Addr) p_align_max) -{ - static enum thp_mode_t thp_mode = thp_mode_not_supported; - static unsigned long int thp_pagesize; - - if (TUNABLE_GET (glibc, elf, thp, int32_t, NULL) == 0) - return p_align_max; - - if (__glibc_unlikely (thp_mode == thp_mode_not_supported - || thp_pagesize == 0)) - { - unsigned long int default_thp_pagesize = DL_MAP_DEFAULT_THP_PAGESIZE; - thp_mode = default_thp_pagesize ? thp_mode_always : __get_thp_mode (); - thp_pagesize = default_thp_pagesize ? : __get_thp_size (); - } - - /* Aligning load segments that are large enough to the PMD size helps - improve THP eligibility and reduces TLB pressure. - We cap the huge page size at MAX_THP_PAGESIZE to avoid over-aligning - on systems with very large normal pages (like 64K pages with 512M - huge pages). */ - if (thp_mode == thp_mode_always - && thp_pagesize <= MAX_THP_PAGESIZE - && ((c->mapstart | c->mapoff) & (thp_pagesize - 1)) == 0 - && (c->mapend - c->mapstart) >= thp_pagesize - && p_align_max < thp_pagesize - && (c->prot & PROT_WRITE) == 0) - return thp_pagesize; - - return p_align_max; -} diff --git a/sysdeps/unix/sysv/linux/ldsodefs.h b/sysdeps/unix/sysv/linux/ldsodefs.h index c63b649432..e39d9afe34 100644 --- a/sysdeps/unix/sysv/linux/ldsodefs.h +++ b/sysdeps/unix/sysv/linux/ldsodefs.h @@ -21,6 +21,9 @@ /* We have the auxiliary vector. */ #define HAVE_AUX_VECTOR +/* We have transparent huge page. */ +#define HAVE_THP + /* Get the real definitions. */ #include_next diff --git a/sysdeps/unix/sysv/linux/loongarch/Makefile b/sysdeps/unix/sysv/linux/loongarch/Makefile index 0d5f087862..d5beb62440 100644 --- a/sysdeps/unix/sysv/linux/loongarch/Makefile +++ b/sysdeps/unix/sysv/linux/loongarch/Makefile @@ -12,3 +12,6 @@ abi-ilp32s-condition := __WORDSIZE == 32 && defined __loongarch_soft_float abi-ilp32d-condition := __WORDSIZE == 32 && defined __loongarch_double_float abi-lp64s-condition := __WORDSIZE == 64 && defined __loongarch_soft_float abi-lp64d-condition := __WORDSIZE == 64 && defined __loongarch_double_float + +# Align THP tests to 32MB. +THP-PAGE-SIZE = 0x2000000 diff --git a/sysdeps/unix/sysv/linux/loongarch/lp64/dl-map-segment-align.h b/sysdeps/unix/sysv/linux/loongarch/lp64/dl-exec-post.h similarity index 90% rename from sysdeps/unix/sysv/linux/loongarch/lp64/dl-map-segment-align.h rename to sysdeps/unix/sysv/linux/loongarch/lp64/dl-exec-post.h index c51ee4ac47..fbc58d662c 100644 --- a/sysdeps/unix/sysv/linux/loongarch/lp64/dl-map-segment-align.h +++ b/sysdeps/unix/sysv/linux/loongarch/lp64/dl-exec-post.h @@ -1,4 +1,4 @@ -/* _dl_map_segment_align. LoongArch64 Linux version. +/* DL_MAP_DEFAULT_THP_PAGESIZE. LoongArch64 Linux version. Copyright (C) 2026 Free Software Foundation, Inc. Copyright The GNU Toolchain Authors. This file is part of the GNU C Library. @@ -19,4 +19,4 @@ #define DL_MAP_DEFAULT_THP_PAGESIZE (32 * 1024 * 1024) -#include_next +#include_next diff --git a/sysdeps/unix/sysv/linux/strace-tst-thp.sh b/sysdeps/unix/sysv/linux/strace-tst-thp.sh new file mode 100644 index 0000000000..8903717fd4 --- /dev/null +++ b/sysdeps/unix/sysv/linux/strace-tst-thp.sh @@ -0,0 +1,56 @@ +#!/bin/sh +# Run THP test under strace to verify control of the THP segment load. +# Copyright (C) 2026 Free Software Foundation, Inc. +# This file is part of the GNU C Library. + +# The GNU C Library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. + +# The GNU C Library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. + +# You should have received a copy of the GNU Lesser General Public +# License along with the GNU C Library; if not, see +# . + +set -e + +rtld="$1" +test_wrapper_env="$2" +run_program_env="$3" +library_path="$4" +test_prog="$5" + +# Test whether strace is available in the test environment. If not, skip +# the test. +${test_wrapper_env} ${run_program_env} \ + /bin/sh -c "command -v strace" || exit 77 + +# Finally the actual test inside the test environment, using the just +# build ld.so and new libraries to run the THP test under strace. +if /bin/sh -c \ + "${test_wrapper_env} ${run_program_env} strace ${rtld} \ + --library-path ${library_path} ${test_prog} 2>&1 \ + | grep -E \"madvise(.*, MADV_HUGEPAGE)\""; then + case x"${run_program_env}" in + *glibc.elf.thp=1*) + exit 0 + ;; + *) + exit 1 + ;; + esac +else + case x"${run_program_env}" in + *glibc.elf.thp=0*) + exit 0 + ;; + *) + exit 1 + ;; + esac +fi diff --git a/sysdeps/unix/sysv/linux/tst-thp-1-no-s-code-pde.c b/sysdeps/unix/sysv/linux/tst-thp-1-no-s-code-pde.c new file mode 100644 index 0000000000..3fd01e9bfe --- /dev/null +++ b/sysdeps/unix/sysv/linux/tst-thp-1-no-s-code-pde.c @@ -0,0 +1,19 @@ +/* Test PDE with THP segment load linked with -Wl,-z,noseparate-code. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include "tst-thp-1.c" diff --git a/sysdeps/unix/sysv/linux/tst-thp-1-no-s-code-static.c b/sysdeps/unix/sysv/linux/tst-thp-1-no-s-code-static.c new file mode 100644 index 0000000000..d0ae0f1ff0 --- /dev/null +++ b/sysdeps/unix/sysv/linux/tst-thp-1-no-s-code-static.c @@ -0,0 +1,19 @@ +/* Test static with THP segment load linked with -Wl,-z,noseparate-code. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include "tst-thp-1.c" diff --git a/sysdeps/unix/sysv/linux/tst-thp-1-no-s-code.c b/sysdeps/unix/sysv/linux/tst-thp-1-no-s-code.c new file mode 100644 index 0000000000..5eb1e005ed --- /dev/null +++ b/sysdeps/unix/sysv/linux/tst-thp-1-no-s-code.c @@ -0,0 +1,19 @@ +/* Test THP segment load linked with -Wl,-z,noseparate-code. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include "tst-thp-1.c" diff --git a/sysdeps/unix/sysv/linux/tst-thp-1-pde.c b/sysdeps/unix/sysv/linux/tst-thp-1-pde.c new file mode 100644 index 0000000000..d854dd43da --- /dev/null +++ b/sysdeps/unix/sysv/linux/tst-thp-1-pde.c @@ -0,0 +1,19 @@ +/* Test PDE with THP segment load linked with -Wl,-z,separate-code. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include "tst-thp-1.c" diff --git a/sysdeps/unix/sysv/linux/tst-thp-1-static.c b/sysdeps/unix/sysv/linux/tst-thp-1-static.c new file mode 100644 index 0000000000..66d7e12954 --- /dev/null +++ b/sysdeps/unix/sysv/linux/tst-thp-1-static.c @@ -0,0 +1,19 @@ +/* Test static with THP segment load linked with -Wl,-z,separate-code. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include "tst-thp-1.c" diff --git a/sysdeps/unix/sysv/linux/tst-thp-1.c b/sysdeps/unix/sysv/linux/tst-thp-1.c new file mode 100644 index 0000000000..49eea7069c --- /dev/null +++ b/sysdeps/unix/sysv/linux/tst-thp-1.c @@ -0,0 +1,28 @@ +/* Test THP segment load linked with -Wl,-z,separate-code. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include "tst-thp-align-check.h" + +static int +do_test (void) +{ + check_align ("tst-thp-1"); + return 0; +} + +#include diff --git a/sysdeps/unix/sysv/linux/tst-thp-align-check.h b/sysdeps/unix/sysv/linux/tst-thp-align-check.h new file mode 100644 index 0000000000..bcb50c96f8 --- /dev/null +++ b/sysdeps/unix/sysv/linux/tst-thp-align-check.h @@ -0,0 +1,132 @@ +/* Test the THP compatible alignment of PT_LOAD segments. + + Copyright (C) 2026 Free Software Foundation, Inc. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#define attribute_hidden +#include /* For enum thp_mode_t and MAX_THP_PAGESIZE. */ +#undef attribute_hidden + +static unsigned long int +get_thp_size (void) +{ + int fd = open ("/sys/kernel/mm/transparent_hugepage/hpage_pmd_size", + O_RDONLY, 0); + if (fd == -1) + return 0; + + char str[INT_BUFSIZE_BOUND (unsigned long int)]; + ssize_t s = read (fd, str, sizeof (str)); + close (fd); + if (s < 0) + return 0; + + unsigned long int r = 0; + for (ssize_t i = 0; i < s; i++) + { + if (str[i] == '\n') + break; + r *= 10; + r += str[i] - '0'; + } + return r; +} + +static enum thp_mode_t +get_thp_mode (void) +{ + int fd = open ("/sys/kernel/mm/transparent_hugepage/enabled", O_RDONLY, 0); + if (fd == -1) + return thp_mode_not_supported; + + static const char mode_always[] = "[always] madvise never\n"; + static const char mode_madvise[] = "always [madvise] never\n"; + static const char mode_never[] = "always madvise [never]\n"; + + char str[sizeof(mode_always)]; + ssize_t s = read (fd, str, sizeof (str)); + if (s >= sizeof str || s < 0) + return thp_mode_not_supported; + str[s] = '\0'; + close (fd); + + if (s == sizeof (mode_always) - 1) + { + if (strcmp (str, mode_always) == 0) + return thp_mode_always; + else if (strcmp (str, mode_madvise) == 0) + return thp_mode_madvise; + else if (strcmp (str, mode_never) == 0) + return thp_mode_never; + } + return thp_mode_not_supported; +} + +static void +check_align (const char *name) +{ + unsigned long int thp_size = get_thp_size (); + enum thp_mode_t thp_mode = get_thp_mode (); + + if (thp_size == 0) + { + FAIL_UNSUPPORTED ("unable to get THP size.\n"); + return; + } + + if (thp_size > MAX_THP_PAGESIZE) + { + FAIL_UNSUPPORTED ("THP size exceeds MAX_THP_PAGESIZE.\n"); + return; + } + + if (thp_mode != thp_mode_always && thp_mode != thp_mode_madvise) + { + FAIL_UNSUPPORTED ("THP mode is not always nor madvise.\n"); + return; + } + + FILE *f = xfopen ("/proc/self/maps", "r"); + char *line = NULL; + size_t len; + + while (xgetline (&line, &len, f)) + { + uintptr_t from, to; + char *prot = NULL, *path = NULL; + int r = sscanf (line, "%" SCNxPTR "-%" SCNxPTR "%ms%*s%*s%*s%ms", + &from, &to, &prot, &path); + + TEST_VERIFY (r == 3 || r == 4); + + if (strstr (prot, "x") && strstr (path, name)) + TEST_COMPARE (from % thp_size, 0); + + free (path); + } + + free (line); + xfclose (f); +} diff --git a/sysdeps/unix/sysv/linux/tst-thp-align.c b/sysdeps/unix/sysv/linux/tst-thp-align.c new file mode 100644 index 0000000000..2e44109ba6 --- /dev/null +++ b/sysdeps/unix/sysv/linux/tst-thp-align.c @@ -0,0 +1,36 @@ +/* Test the THP compatible alignment of PT_LOAD segments. + + Copyright (C) 2026 Free Software Foundation, Inc. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include +#include "tst-thp-align-check.h" + +#define THP_SIZE_MOD_NAME "tst-thp-size-mod.so" + +static int +do_test (void) +{ + void *dl; + + dl = xdlopen (THP_SIZE_MOD_NAME, RTLD_NOW); + check_align (THP_SIZE_MOD_NAME); + xdlclose (dl); + + return 0; +} + +#include diff --git a/sysdeps/unix/sysv/linux/tst-thp-size-mod.S b/sysdeps/unix/sysv/linux/tst-thp-size-mod.S new file mode 100644 index 0000000000..add66dfb52 --- /dev/null +++ b/sysdeps/unix/sysv/linux/tst-thp-size-mod.S @@ -0,0 +1,24 @@ +/* A module that provides THP code size for testing. + + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +/* Create a 32MB text section to enable transparent huge page load. */ + .text + .globl thp_size +thp_size: + .space 32 * 1024 * 1024 diff --git a/sysdeps/unix/sysv/linux/x86/dl-exec-post.h b/sysdeps/unix/sysv/linux/x86/dl-exec-post.h new file mode 100644 index 0000000000..0fa0b5e0e5 --- /dev/null +++ b/sysdeps/unix/sysv/linux/x86/dl-exec-post.h @@ -0,0 +1,22 @@ +/* DL_MAP_DEFAULT_THP_PAGESIZE. Linux/x86 version. + Copyright (C) 2026 Free Software Foundation, Inc. + Copyright The GNU Toolchain Authors. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#define DL_MAP_DEFAULT_THP_PAGESIZE (2 * 1024 * 1024) + +#include_next -- 2.53.0