From patchwork Wed Apr 8 07:47:25 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "H.J. Lu" X-Patchwork-Id: 132792 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [127.0.0.1]) by sourceware.org (Postfix) with ESMTP id 9B5A04BA2E0E for ; Wed, 8 Apr 2026 07:48:41 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 9B5A04BA2E0E Authentication-Results: sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=oWhG3fLN X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pj1-x1032.google.com (mail-pj1-x1032.google.com [IPv6:2607:f8b0:4864:20::1032]) by sourceware.org (Postfix) with ESMTPS id 8F29F4BA5436 for ; Wed, 8 Apr 2026 07:48:04 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 8F29F4BA5436 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 8F29F4BA5436 Authentication-Results: server2.sourceware.org; arc=pass smtp.remote-ip=2607:f8b0:4864:20::1032 ARC-Seal: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1775634484; cv=pass; b=iM0v6lpO2IwnN6cWNUYf3Nf1h+brN3VDr+GXihmnTPW3mxlnJgo0+fMYoB45fG/PXveIfealFOwTBz2g/y8/d3fREtlmqdHq5ZjxsSW7oa2CS3wvt+RFXemXz6ALxXXl3cIka0fmXMb8rD8jp2OXK7YHrbQpM2xGbFCoVEiRAOo= ARC-Message-Signature: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1775634484; c=relaxed/simple; bh=Ts44fJ9sSRRIr7LGaH2FkDkbWKyZgvw/w2UmZhinwB0=; h=DKIM-Signature:MIME-Version:From:Date:Message-ID:Subject:To; b=XBgRydgp/uRsR+CXsIm5ZYTLibxgv6KIeUioAX8msuQaoBcsQO9HoJ8wD/MKXj4BFWv6wQoTX8X+1VqMDHhlwck0DKKYuVynGFTIWzq5UUmUcyoorogJm/8+/alU2IdFsUbwTqd0+uVNvmCxf6KKTUoS3ygWP/ghZ2jCtf03Oak= ARC-Authentication-Results: i=2; server2.sourceware.org DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 8F29F4BA5436 Received: by mail-pj1-x1032.google.com with SMTP id 98e67ed59e1d1-35c206f0481so5669565a91.0 for ; Wed, 08 Apr 2026 00:48:04 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1775634483; cv=none; d=google.com; s=arc-20240605; b=TyNFKQvPkHZW7NvArissP2CKZhBCTBCHoQEfn+PB2YnUXfFn9hsUCt2eW3Y88rnEFO LjucxikiZ60w8nys1X+BJ7E8MAjiHHP9++zB5b3x926A7LaNEffwL1sSaZUPYkJbGtbD zoaAtT7oqRE35MRDmv1QGPrwlxppmWWubCQzz5pcDGwRYcbeTbSPK3ISbRPCTStu8k7R /dDE2FvrlEuQjmVLsX8/TrlOG7eI+91lslqcj8QN39c+OaGw6hPahrlpK3yPVP4YIl1j RgFFdSIa6iXzjM+OFxKPsWmXX3NVuz4LPTcsWY0yyFhpjCNiDz8eDyWPiBPIWAwMzrcQ AD4A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:dkim-signature; bh=CpA1KQC4gOtuMwVcpQVVquRgUA0DBxhFj6fm4wL8eLk=; fh=Kjg0pLtZ3khhLpL1MjCxzW8jTI5bCs/NxpJFONo9Y4Y=; b=YmCbwmkIv71Z1Yg+hvUNmG2Z/x00mr1BMk158WGbfMJN/+llFb3OSo69oDQ1WQG0LU yPFLaySyfeYJXbo44Nc8OBefIxLitTykCVEx8g+4L4GoQ7UNnxpR+/VFujr5obSvjady NY+MDgNzTy8zv2PzZlZ585yVq9InnoweF2D7/X27ugmxoO+Nd5S+Fo15Zfl6fCyb6bmi rh5TKeZ8JFbSVhVUP/DxrD7o6FvsSj6jbEZ8I/u15a3sKBkJD7pz7TmnOVNyfVJgeSZU 4dsUXVLBAKeOzC46CzvLDkaJDHAzSZBSSOURulVphCbbvs45xyzPRo8ss3iXiKQiU3Mb PMTQ==; darn=sourceware.org ARC-Authentication-Results: i=1; mx.google.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1775634483; x=1776239283; darn=sourceware.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=CpA1KQC4gOtuMwVcpQVVquRgUA0DBxhFj6fm4wL8eLk=; b=oWhG3fLNa0ysfhzLi2ETRyCJgIqWOm7/8nLENYFDRcfHI5pFtE3NWq+qq/kvYspGib PAxNmuDQJfcpakps7uMQWw56pn9NpJUI5VrAY0/1m/3++GffX3t+dUHxn/Dx8NbJ0Vx5 9zMqEFyFCxKEq9jxiTodVfUQ1yO7rCrFQcZvsScL1mS24dXA5Ke1nZHfErXGf39bVYgA W4Cl8Yzy+Dw0jJrdzGjBPZ7B8EBF2OP6i7PwcB43o/rh00LyvGHetoQw6td1xR64/Z3S yiiRUKF5/o4dY329xbye1NGaSDyFWJtsYvF0rAF/JRw7GBrsR+czbVqE66UJ5PqUrut8 A0Cw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1775634483; x=1776239283; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=CpA1KQC4gOtuMwVcpQVVquRgUA0DBxhFj6fm4wL8eLk=; b=S6c7ikDlMDX3TUMwNDa0lzeQVmPmHsN3TQLSd10/aKMXBob4iuSondU42h8TkChisD 75nYa4DWBgcfl8keDgB3JL0KIrzPPIsoMti4QVYyaBv6D+y+L07lkuDT2mwbc89JYXkV j3OAqTG/ZaEMxTf4wRQChZUUxmrd3JJzBVThA6c1Mpd6B2DhmmCLFYrNhHbGpBLfZThw VTgIKc+UrHMkn5/+1v5xSkTvWZy6LS7msJN6QPJwDPHktvld9grKk8bP1bgQyac9D1ir MCI1K7yGQ+/v+xyA5rnqIjfs0JQ6Z52FsBTV0+tchxIs6EITdTrPVOJ4CJhErQQtoFOG u4Ew== X-Gm-Message-State: AOJu0YzmPYmupVHb2/BvfrACbFo0XpUP7x2aB8rpCijg3o+D5/Yo2Cuz qG9ZovrGig45X2hLga/VrX7Ptb5LafzcBSbwgaBgEfcsl7sMgpoqDj1hKZA0eq963oid69qNwbx sPSiwa2ugmI+jb/WhA9uvPePKDp+ldOuCID3Fm2Lgkw== X-Gm-Gg: AeBDiett92WG/YHHm/EptvQqdWizlT1ZibDMhOUhsuNAR1hkz75CfzSov0xY+0gthBO RBiT45qyd3uVTXe8SPqtcHqWLmTjbUCW4xAU5LZzgnuC0UG57pxRMn56ZGKGaTBM+vriWi44qIm jfCDRP7tuXy2Q0P6Vu55VFqSQvPZLj8++mAyMTMD3SfD86S66E1ltUxWvXIXjyTrXEZP9gnFke7 YE2W9TnEFq53cGhcbRDUK2czDkhXyyE82kpV+mWHnWgTY/yML329lPJ3ve2o7ih0DcnICfIA8a+ AykllK13 X-Received: by 2002:a17:90b:3e88:b0:35d:9da0:4947 with SMTP id 98e67ed59e1d1-35de67f788amr20305431a91.11.1775634483505; Wed, 08 Apr 2026 00:48:03 -0700 (PDT) MIME-Version: 1.0 References: <20260405035323.558335-1-wangrui@loongson.cn> In-Reply-To: From: "H.J. Lu" Date: Wed, 8 Apr 2026 15:47:25 +0800 X-Gm-Features: AQROBzBguPuFGzss1g4VM6j0ZDG0FU0rDB_uTTrKk-Om_LPJCUC1RE2pBF9JOk4 Message-ID: Subject: [PATCH] elf: Add transparent huge page marker and _dl_map_segment_adjust To: WANG Rui Cc: GNU C Library , Adhemerval Zanella , Dev Jain , Florian Weimer , Wilco Dijkstra , Xi Ruoyao , WANG Xuerui , caiyinyu , mengqinggang , Huacai Chen X-Spam-Status: No, score=-3010.8 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces~patchwork=sourceware.org@sourceware.org On Wed, Apr 8, 2026 at 10:07 AM H.J. Lu wrote: > > On Tue, Apr 7, 2026 at 7:32 AM H.J. Lu wrote: > > > > On Sun, Apr 5, 2026 at 11:54 AM WANG Rui wrote: > > > > > > OK for commit? > > > > > > Changes since [v7]: > > > * Rename tunable glibc.elf.hugetlb to glibc.elf.thp. > > > * Rebase on current master. > > > > > > Changes since [v6]: > > > * Move MAX_THP_PAGESIZE to hugepages. > > > * Skip THP mode probing when DL_MAP_DEFAULT_THP_PAGESIZE is non-zero. > > > > > > Changes since [v5]: > > > * No functional changes. > > > * Add benchmark results to the commit message. > > > > > > Changes since [v4]: > > > * Merge malloc-hugepages into hugepages. > > > * Limit the glibc.elf.hugetlb tunable maximum value to 1. > > > > > > Changes since [v3]: > > > * Rebased on current master. > > > * Resolved conflicts with recently merged changes. > > > * No functional changes intended. > > > > > > Changes since [v2]: > > > * Refactor THP detection into a new generic hugepages abstraction, > > > moving helpers out of malloc-hugepages. > > > * Add a new tunable, `glibc.elf.hugetlb`, to control THP-aware ELF > > > segment alignment. > > > * Move the Linux implementation of `_dl_map_segment_align` to > > > sysdeps/unix/sysv/linux/, making it generic for Linux (including > > > 32-bit) and avoiding wordsize-64-specific overrides. > > > * Use `static inline` instead of `static __always_inline`. > > > > > > Changes since [v1]: > > > * Fix CI build failure (-Wunused-function). > > > > > > This patch series introduces a small extension point in the ELF loader to allow > > > architecture-specific adjustment of load segment alignment, and uses it to > > > improve Transparent Huge Page (THP) usage on Linux. > > > > > > Patch 1 moves Transparent Huge Page helpers into a new generic hugepages > > > abstraction, so THP mode detection and default huge page size probing can be > > > shared between malloc and the dynamic loader. There is no functional change. > > > > > > Patch 2 removes a redundant declaration of `_dl_map_segments` from > > > dl-load.h, avoiding `-Wunused-function` build failures and keeping the > > > prototype colocated with its definition. > > > > > > Patch 3 adds a new helper, `_dl_map_segment_align`, which is called when > > > determining the maximum alignment for ELF load segments. The generic > > > implementation is a no-op and preserves existing behavior. > > > > > > Patch 4 introduces a new tunable, `glibc.elf.hugetlb`, which controls > > > THP-aware alignment of ELF loadable segments. By default, the value is 0 > > > and existing behavior is preserved. > > > > > > Patch 5 provides a Linux implementation of `_dl_map_segment_align` that > > > opportunistically aligns large, suitably aligned, non-writable `PT_LOAD` > > > segments to the system’s default THP page size when THP is configured to > > > be used unconditionally and the tunable is enabled. > > > > > > Patch 6 enables this behavior by default on LoongArch64 Linux and defines > > > the default THP page size (32MB), matching the architecture’s PMD huge page > > > geometry. > > > > > > [v7]: https://sourceware.org/pipermail/libc-alpha/2026-March/175776.html > > > [v6]: https://sourceware.org/pipermail/libc-alpha/2026-March/175737.html > > > [v5]: https://sourceware.org/pipermail/libc-alpha/2026-March/175694.html > > > [v4]: https://sourceware.org/pipermail/libc-alpha/2026-March/175644.html > > > [v3]: https://sourceware.org/pipermail/libc-alpha/2026-February/175464.html > > > [v2]: https://sourceware.org/pipermail/libc-alpha/2026-February/175394.html > > > [v1]: https://sourceware.org/pipermail/libc-alpha/2026-February/175359.html > > > > > > WANG Rui (6): > > > hugepages: Move THP helpers to generic hugepages abstraction > > > elf: Remove redundant _dl_map_segments declaration from dl-load.h > > > elf: Introduce _dl_map_segment_align hook for segment alignment tuning > > > tunables: Add glibc.elf.thp tunable for THP-aware segment alignment > > > elf: Align large load segments to PMD huge page size for THP > > > loongarch: Enable THP-aligned load segments by default on 64-bit > > > > > > elf/dl-load.c | 4 ++ > > > elf/dl-load.h | 5 +- > > > elf/dl-tunables.c | 7 +-- > > > elf/dl-tunables.list | 8 +++ > > > malloc/malloc-internal.h | 2 +- > > > malloc/malloc.c | 27 +++++----- > > > manual/tunables.texi | 24 +++++++++ > > > sysdeps/generic/Makefile | 4 +- > > > sysdeps/generic/dl-map-segment-align.h | 26 +++++++++ > > > .../{malloc-hugepages.c => hugepages.c} | 13 +++-- > > > .../{malloc-hugepages.h => hugepages.h} | 32 ++++++----- > > > sysdeps/unix/sysv/linux/Makefile | 1 + > > > .../{malloc-hugepages.h => hugepages.h} | 4 +- > > > .../unix/sysv/linux/dl-map-segment-align.c | 53 +++++++++++++++++++ > > > .../unix/sysv/linux/dl-map-segment-align.h | 27 ++++++++++ > > > .../linux/{malloc-hugepages.c => hugepages.c} | 33 ++++++------ > > > .../unix/sysv/linux/loongarch/cpu-features.c | 6 +++ > > > .../loongarch/lp64/dl-map-segment-align.h | 22 ++++++++ > > > 18 files changed, 237 insertions(+), 61 deletions(-) > > > create mode 100644 sysdeps/generic/dl-map-segment-align.h > > > rename sysdeps/generic/{malloc-hugepages.c => hugepages.c} (76%) > > > rename sysdeps/generic/{malloc-hugepages.h => hugepages.h} (68%) > > > rename sysdeps/unix/sysv/linux/aarch64/{malloc-hugepages.h => hugepages.h} (91%) > > > create mode 100644 sysdeps/unix/sysv/linux/dl-map-segment-align.c > > > create mode 100644 sysdeps/unix/sysv/linux/dl-map-segment-align.h > > > rename sysdeps/unix/sysv/linux/{malloc-hugepages.c => hugepages.c} (89%) > > > create mode 100644 sysdeps/unix/sysv/linux/loongarch/lp64/dl-map-segment-align.h > > > > > > -- > > > 2.53.0 > > > > > > > Please add some tests for this feature. > > > > I opened: > > https://sourceware.org/bugzilla/show_bug.cgi?id=34056 > > for GNU_PROPERTY_1_NEEDED_TRANSPARENT_HUGEPAGE. > Here is a patch on top of yours to implement GNU_PROPERTY_1_NEEDED_TRANSPARENT_HUGEPAGE and add _dl_map_segment_adjust. From e9ce30429e8e7462fd1da34c550c9c9f895682de Mon Sep 17 00:00:00 2001 From: "H.J. Lu" Date: Tue, 7 Apr 2026 06:46:19 +0800 Subject: [PATCH] elf: Add transparent huge page marker and _dl_map_segment_adjust The current scheme of loading PT_LOAD segments with THP is hard-coded in Linux kernel ELF loader and is hard-coded in ld.so or requires a tunable at run-time. Add GNU_PROPERTY_1_NEEDED_TRANSPARENT_HUGEPAGE: /* Set if the object file should be loaded with transparent huge pages if possible. */ #define GNU_PROPERTY_1_NEEDED_TRANSPARENT_HUGEPAGE (1U << 1) to GNU_PROPERTY_1_NEEDED. Update ld.so to process PT_GNU_PROPERTY segment for GNU_PROPERTY_1_NEEDED before loading PT_LOAD segments and enable THP if the GNU_PROPERTY_1_NEEDED_TRANSPARENT_HUGEPAGE is set in the binary. This fixes BZ #34056. Add _dl_map_segment_adjust to enable THP for a binary with program headers: Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align LOAD 0x000000 0x00400000 0x00400000 0x16d110 0x16d110 R 0x1000 LOAD 0x16e000 0x0056e000 0x0056e000 0x1055d9 0x1055d9 R E 0x1000 LOAD 0x273ebc 0x00674ebc 0x00674ebc 0x02540 0x039fc RW 0x1000 by combining the first 2 PT_LOAD segments, when the following conditions are satisfied: 1. The first segment file offset is 0. 2. Both segments are read-only. 3. The first segment load address is aligned to THP page size. 4. There is no address gap between them. 5. The total size >= THP page size. into a single read-only and executable load segment: LOAD 0x000000 0x00400000 0x00400000 0x2735d9 0x2735d9 R 0x200000 LOAD 0x273ebc 0x00674ebc 0x00674ebc 0x02540 0x039fc RW 0x1000 Signed-off-by: H.J. Lu --- elf/Makefile | 6 ++ elf/dl-load.c | 78 +++++++++++++++-- elf/dl-misc.c | 58 +++++++++---- elf/elf.h | 4 + elf/tst-thp-1-mod.c | 24 ++++++ elf/tst-thp-1.c | 28 ++++++ elf/tst-thp-note.S | 36 ++++++++ sysdeps/generic/dl-map-segment-align.h | 16 +++- sysdeps/generic/ldsodefs.h | 6 ++ .../unix/sysv/linux/dl-map-segment-align.c | 86 +++++++++++++++++-- .../unix/sysv/linux/dl-map-segment-align.h | 8 +- 11 files changed, 310 insertions(+), 40 deletions(-) create mode 100644 elf/tst-thp-1-mod.c create mode 100644 elf/tst-thp-1.c create mode 100644 elf/tst-thp-note.S diff --git a/elf/Makefile b/elf/Makefile index 3927e01d64..72a2abf4f5 100644 --- a/elf/Makefile +++ b/elf/Makefile @@ -474,6 +474,7 @@ tests += \ tst-sonamemove-link \ tst-startup-errno \ tst-thrlock \ + tst-thp-1 \ tst-tls-dlinfo \ tst-tls-ie \ tst-tls-ie-dlmopen \ @@ -1009,6 +1010,7 @@ modules-names += \ tst-sonamemove-runmod1 \ tst-sonamemove-runmod2 \ tst-sprof-mod \ + tst-thp-1-mod \ tst-tls-ie-mod0 \ tst-tls-ie-mod1 \ tst-tls-ie-mod2 \ @@ -2341,6 +2343,10 @@ $(objpfx)tst-thrlock: $(shared-thread-library) $(objpfx)tst-thrlock.out: $(libm) $(objpfx)tst-noload.out: $(libm) +LDFLAGS-tst-thp-1-mod.so = -Wl,-soname,tst-thp-1-mod.so +$(objpfx)tst-thp-1-mod.so: $(objpfx)tst-thp-note.o +$(objpfx)tst-thp-1: $(objpfx)tst-thp-1-mod.so + tst-tst-dlopen-tlsmodid-no-pie = yes $(objpfx)tst-dlopen-tlsmodid: $(shared-thread-library) $(objpfx)tst-dlopen-tlsmodid.out: $(objpfx)tst-dlopen-self diff --git a/elf/dl-load.c b/elf/dl-load.c index f3d943e99c..0ed120278b 100644 --- a/elf/dl-load.c +++ b/elf/dl-load.c @@ -836,12 +836,22 @@ _dl_init_paths (const char *llp, const char *source, PT_LOAD segments are mapped. Only one NT_GNU_PROPERTY_TYPE_0 note is handled which contains processor specific properties. FD is -1 for the kernel mapped main executable otherwise it is - the fd used for loading module L. */ + the fd used for loading module L. If NOTE isn't NULL, only + GNU_PROPERTY_1_NEEDED should be processed. */ -void -_dl_process_pt_gnu_property (struct link_map *l, int fd, const ElfW(Phdr) *ph) +static void +_dl_process_pt_gnu_property_1 (struct link_map *l, int fd, + const ElfW(Phdr) *ph, + const ElfW(Nhdr) *note) { - const ElfW(Nhdr) *note = (const void *) (ph->p_vaddr + l->l_addr); + bool property_1_needed_only; + if (note == NULL) + { + property_1_needed_only = false; + note = (const void *) (ph->p_vaddr + l->l_addr); + } + else + property_1_needed_only = true; const ElfW(Addr) size = ph->p_memsz; const ElfW(Addr) align = ph->p_align; @@ -885,8 +895,18 @@ _dl_process_pt_gnu_property (struct link_map *l, int fd, const ElfW(Phdr) *ph) last_type = type; + if (property_1_needed_only) + { + if (type == GNU_PROPERTY_1_NEEDED) + { + if (datasz == 4) + l->l_1_needed = *(unsigned int *) ptr; + return; + } + } /* Target specific property processing. */ - if (_dl_process_gnu_property (l, fd, type, datasz, ptr) == 0) + else if (_dl_process_gnu_property (l, fd, type, datasz, + ptr) == 0) return; /* Check the next property item. */ @@ -904,6 +924,18 @@ _dl_process_pt_gnu_property (struct link_map *l, int fd, const ElfW(Phdr) *ph) } } +/* Process PT_GNU_PROPERTY program header PH in module L after + PT_LOAD segments are mapped. Only one NT_GNU_PROPERTY_TYPE_0 + note is handled which contains processor specific properties. + FD is -1 for the kernel mapped main executable otherwise it is + the fd used for loading module L. */ + +void +_dl_process_pt_gnu_property (struct link_map *l, int fd, const ElfW(Phdr) *ph) +{ + return _dl_process_pt_gnu_property_1 (l, fd, ph, NULL); +} + static void _dl_notify_new_object (int mode, Lmid_t nsid, struct link_map *l) { @@ -1106,6 +1138,34 @@ _dl_map_object_from_fd (const char *name, const char *origname, int fd, bool empty_dynamic = false; ElfW(Addr) p_align_max = 0; + bool thp_requested + = TUNABLE_GET (glibc, elf, thp, int32_t, NULL) != 0; + + /* If transparent hugt page isn't enabled nor set in tunables, before + PT_LOAD segments are mapped in, process PT_GNU_PROPERTY segment + for GNU_PROPERTY_1_NEEDED. */ + if (!thp_requested && !TUNABLE_IS_INITIALIZED (glibc, elf, thp)) + for (ph = phdr; ph < &phdr[l->l_phnum]; ++ph) + if (ph->p_type == PT_GNU_PROPERTY) + { + size_t prop_size = ph->p_offset + ph->p_memsz; + void *prop = _dl_sysdep_map_file_from_fd (fd, &prop_size, + PROT_READ); + if (prop == MAP_FAILED) + { + errstring = N_("can't map in property segment"); + goto lose; + } + const ElfW(Nhdr) *note + = (const void *) (prop + ph->p_offset); + _dl_process_pt_gnu_property_1 (l, fd, ph, note); + thp_requested + = ((l->l_1_needed + & GNU_PROPERTY_1_NEEDED_TRANSPARENT_HUGEPAGE) != 0); + __munmap (prop, prop_size); + break; + } + /* The struct is initialized to zero so this is not necessary: l->l_ld = 0; l->l_phdr = 0; @@ -1174,7 +1234,8 @@ _dl_map_object_from_fd (const char *name, const char *origname, int fd, c->prot = pf_to_prot (ph->p_flags); /* Architecture-specific adjustment of segment alignment. */ - p_align_max = _dl_map_segment_align (c, p_align_max); + p_align_max = _dl_map_segment_align (c, p_align_max, + thp_requested); break; case PT_TLS: @@ -1229,9 +1290,8 @@ _dl_map_object_from_fd (const char *name, const char *origname, int fd, goto lose; } - /* Align all PT_LOAD segments to the maximum p_align. */ - for (size_t i = 0; i < nloadcmds; i++) - loadcmds[i].mapalign = p_align_max; + _dl_map_segment_adjust (loadcmds, &nloadcmds, p_align_max, + thp_requested); /* dlopen of an executable is not valid because it is not possible to perform proper relocations, handle static TLS, or run the diff --git a/elf/dl-misc.c b/elf/dl-misc.c index 04eb768faf..f4278bb663 100644 --- a/elf/dl-misc.c +++ b/elf/dl-misc.c @@ -27,41 +27,61 @@ #include #include -/* Read the whole contents of FILE into new mmap'd space with given - protections. *SIZEP gets the size of the file. On error MAP_FAILED - is returned. */ +/* Map in the contents of FD with given protections. If *SIZEP is 0, + *SIZEP gets the size of the file. Otherwise, read up to *SIZEP bytes. + On error, MAP_FAILED is returned. */ void * -_dl_sysdep_read_whole_file (const char *file, size_t *sizep, int prot) +_dl_sysdep_map_file_from_fd (int fd, size_t *sizep, int prot) { void *result = MAP_FAILED; struct __stat64_t64 st; - int fd = __open64_nocancel (file, O_RDONLY | O_CLOEXEC); - if (fd >= 0) + if (__fstat64_time64 (fd, &st) >= 0) { - if (__fstat64_time64 (fd, &st) >= 0) - { - *sizep = st.st_size; + size_t size = st.st_size; + if (*sizep == 0) + *sizep = size; + else if (*sizep > size) + return result; + else + size = *sizep; - /* No need to map the file if it is empty. */ - if (*sizep != 0) - /* Map a copy of the file contents. */ - result = __mmap (NULL, *sizep, prot, + /* No need to map the file if it is empty. */ + if (size != 0) + /* Map a copy of the file contents. */ + result = __mmap (NULL, size, prot, #ifdef MAP_COPY - MAP_COPY + MAP_COPY #else - MAP_PRIVATE + MAP_PRIVATE #endif #ifdef MAP_FILE - | MAP_FILE + | MAP_FILE #endif - , fd, 0); - } - __close_nocancel (fd); + , fd, 0); } + return result; } +/* Read the whole contents of FILE into new mmap'd space with given + protections. *SIZEP gets the size of the file. On error MAP_FAILED + is returned. */ + +void * +_dl_sysdep_read_whole_file (const char *file, size_t *sizep, int prot) +{ + int fd = __open64_nocancel (file, O_RDONLY | O_CLOEXEC); + if (fd >= 0) + { + *sizep = 0; + void *result = _dl_sysdep_map_file_from_fd (fd, sizep, prot); + __close_nocancel (fd); + return result; + } + return MAP_FAILED; +} + /* Test whether given NAME matches any of the names of the given object. */ int _dl_name_match_p (const char *name, const struct link_map *map) diff --git a/elf/elf.h b/elf/elf.h index 46a01281cb..217dcea2bc 100644 --- a/elf/elf.h +++ b/elf/elf.h @@ -1380,6 +1380,10 @@ typedef struct cannot be used with copy relocation. */ #define GNU_PROPERTY_1_NEEDED_INDIRECT_EXTERN_ACCESS (1U << 0) +/* Set if the object file should be loaded with transparent huge pages + if possible. */ +#define GNU_PROPERTY_1_NEEDED_TRANSPARENT_HUGEPAGE (1U << 1) + /* Processor-specific semantics, lo */ #define GNU_PROPERTY_LOPROC 0xc0000000 /* Processor-specific semantics, hi */ diff --git a/elf/tst-thp-1-mod.c b/elf/tst-thp-1-mod.c new file mode 100644 index 0000000000..ed7d9f7da5 --- /dev/null +++ b/elf/tst-thp-1-mod.c @@ -0,0 +1,24 @@ +/* Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include + +void +hello (void) +{ + printf ("Hello World\n"); +} diff --git a/elf/tst-thp-1.c b/elf/tst-thp-1.c new file mode 100644 index 0000000000..13573c2ae3 --- /dev/null +++ b/elf/tst-thp-1.c @@ -0,0 +1,28 @@ +/* Test GNU_PROPERTY_1_NEEDED_TRANSPARENT_HUGEPAGE. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +extern void hello (void); + +static int +do_test (void) +{ + hello (); + return 0; +} + +#include diff --git a/elf/tst-thp-note.S b/elf/tst-thp-note.S new file mode 100644 index 0000000000..7fc495eb83 --- /dev/null +++ b/elf/tst-thp-note.S @@ -0,0 +1,36 @@ +/* Add a GNU_PROPERTY_1_NEEDED property with + GNU_PROPERTY_1_NEEDED_TRANSPARENT_HUGEPAGE. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include + + .section ".note.gnu.property", "a" + .balign GCCMACRO__SIZEOF_POINTER__ + .long 1f - 0f /* name length */ + .long 5f - 2f /* data length */ + .long 5 /* note type */ +0: .asciz "GNU" /* vendor name */ +1: + .balign GCCMACRO__SIZEOF_POINTER__ +2: .long 0xb0008000 /* pr_type. */ + .long 4f - 3f /* pr_datasz. */ +3: + .long 0x00000002 +4: + .balign GCCMACRO__SIZEOF_POINTER__ +5: diff --git a/sysdeps/generic/dl-map-segment-align.h b/sysdeps/generic/dl-map-segment-align.h index f4a671f25f..ced6023abf 100644 --- a/sysdeps/generic/dl-map-segment-align.h +++ b/sysdeps/generic/dl-map-segment-align.h @@ -1,4 +1,4 @@ -/* _dl_map_segment_align. Generic version. +/* _dl_map_segment_align and _dl_map_segment_adjust. Generic version. Copyright (C) 2026 Free Software Foundation, Inc. Copyright The GNU Toolchain Authors. This file is part of the GNU C Library. @@ -20,7 +20,19 @@ #include static inline ElfW(Addr) -_dl_map_segment_align (const struct loadcmd *c, ElfW(Addr) p_align_max) +_dl_map_segment_align (const struct loadcmd *c, ElfW(Addr) p_align_max, + bool thp_requested) { return p_align_max; } + +static inline void +_dl_map_segment_adjust (struct loadcmd *load_cmds, + size_t *n_load_cmds, ElfW(Addr) max_align, + bool thp_requested) +{ + /* Align all PT_LOAD segments to the maximum p_align. */ + size_t n = *n_load_cmds; + for (size_t i = 0; i < n; i++) + load_cmds[i].mapalign = max_align; +} diff --git a/sysdeps/generic/ldsodefs.h b/sysdeps/generic/ldsodefs.h index 15c4659853..d953036a71 100644 --- a/sysdeps/generic/ldsodefs.h +++ b/sysdeps/generic/ldsodefs.h @@ -1154,6 +1154,12 @@ extern void _dl_unload_cache (void) attribute_hidden; extern void *_dl_sysdep_read_whole_file (const char *file, size_t *sizep, int prot) attribute_hidden; +/* System-dependent function to map in a file's contents. If *SIZEP is + 0, *SIZEP gets the size of the file. Otherwise, read up to *SIZEP + bytes. On error, MAP_FAILED is returned. */ +extern void * _dl_sysdep_map_file_from_fd (int fd, size_t *sizep, + int prot) attribute_hidden; + /* System-specific function to do initial startup for the dynamic linker. After this, file access calls and getenv must work. This is responsible for setting __libc_enable_secure if we need to be secure (e.g. setuid), diff --git a/sysdeps/unix/sysv/linux/dl-map-segment-align.c b/sysdeps/unix/sysv/linux/dl-map-segment-align.c index 75fb049673..dd62de7f4a 100644 --- a/sysdeps/unix/sysv/linux/dl-map-segment-align.c +++ b/sysdeps/unix/sysv/linux/dl-map-segment-align.c @@ -1,4 +1,4 @@ -/* _dl_map_segment_align. Linux version. +/* _dl_map_segment_align and _dl_map_segment_adjust. Linux version. Copyright (C) 2026 Free Software Foundation, Inc. Copyright The GNU Toolchain Authors. This file is part of the GNU C Library. @@ -18,16 +18,16 @@ . */ #include -#include #include +static enum thp_mode_t thp_mode = thp_mode_not_supported; +static size_t thp_pagesize; + ElfW (Addr) -_dl_map_segment_align (const struct loadcmd *c, ElfW (Addr) p_align_max) +_dl_map_segment_align (const struct loadcmd *c, ElfW (Addr) p_align_max, + bool thp_requested) { - static enum thp_mode_t thp_mode = thp_mode_not_supported; - static unsigned long int thp_pagesize; - - if (TUNABLE_GET (glibc, elf, thp, int32_t, NULL) == 0) + if (!thp_requested) return p_align_max; if (__glibc_unlikely (thp_mode == thp_mode_not_supported @@ -51,3 +51,75 @@ _dl_map_segment_align (const struct loadcmd *c, ElfW (Addr) p_align_max) return p_align_max; } + +void +_dl_map_segment_adjust (struct loadcmd *load_cmds, size_t *n_load_cmds, + ElfW(Addr) max_align, bool thp_requested) + +{ + size_t n = *n_load_cmds; + + if (n == 1) + { + load_cmds[0].mapalign = max_align; + return; + } + + if (!thp_requested || thp_pagesize == 0 || max_align == thp_pagesize) + { + /* If THP isn't enabled or the maximum p_align is the same as THP + page size, align all PT_LOAD segments to the maximum p_align. */ +thp_disabled: + for (size_t i = 0; i < n; i++) + load_cmds[i].mapalign = max_align; + return; + } + + /* For a binary with program headers: + + Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align + LOAD 0x000000 0x00400000 0x00400000 0x16d110 0x16d110 R 0x1000 + LOAD 0x16e000 0x0056e000 0x0056e000 0x1055d9 0x1055d9 R E 0x1000 + LOAD 0x273ebc 0x00674ebc 0x00674ebc 0x02540 0x039fc RW 0x1000 + + combine the first 2 PT_LOAD segments, when the following conditions + are satisfied: + + 1. The first segment file offset is 0. + 2. Both segments are read-only. + 3. The first segment load address is aligned to THP page size. + 4. There is no address gap between them. + 5. The total size >= THP page size. + + into a single read-only and executable load segment: + + LOAD 0x000000 0x00400000 0x00400000 0x2735d9 0x2735d9 R 0x200000 + LOAD 0x273ebc 0x00674ebc 0x00674ebc 0x02540 0x039fc RW 0x1000 + */ + + ElfW(Addr) mapstart = load_cmds[0].mapstart; + ElfW(Addr) mapend = load_cmds[1].mapend; + int prot = load_cmds[0].prot | load_cmds[1].prot; + if (load_cmds[0].mapoff != 0 + || (prot & PROT_WRITE) != 0 + || (mapstart & (thp_pagesize - 1)) != 0 + || load_cmds[0].mapend != load_cmds[1].mapstart + || load_cmds[1].mapoff != (load_cmds[0].mapend - mapstart) + || (mapend - mapstart) < thp_pagesize) + goto thp_disabled; + + /* Combine the first 2 PT_LOAD segments. */ + load_cmds[0].mapend = mapend; + load_cmds[0].dataend = load_cmds[1].dataend; + load_cmds[0].allocend = load_cmds[1].allocend; + load_cmds[0].mapalign = thp_pagesize; + load_cmds[0].prot = prot; + + n--; + *n_load_cmds = n; + for (size_t i = 1; i < n; i++) + { + load_cmds[i] = load_cmds[i + 1]; + load_cmds[i].mapalign = max_align; + } +} diff --git a/sysdeps/unix/sysv/linux/dl-map-segment-align.h b/sysdeps/unix/sysv/linux/dl-map-segment-align.h index 83794a0398..4484cedfe3 100644 --- a/sysdeps/unix/sysv/linux/dl-map-segment-align.h +++ b/sysdeps/unix/sysv/linux/dl-map-segment-align.h @@ -1,4 +1,4 @@ -/* _dl_map_segment_align. Linux version. +/* _dl_map_segment_align and _dl_map_segment_adjust. Linux version. Copyright (C) 2026 Free Software Foundation, Inc. Copyright The GNU Toolchain Authors. This file is part of the GNU C Library. @@ -23,5 +23,7 @@ # define DL_MAP_DEFAULT_THP_PAGESIZE 0 #endif -ElfW (Addr) - _dl_map_segment_align (const struct loadcmd *c, ElfW (Addr) p_align_max); +extern ElfW (Addr) _dl_map_segment_align + (const struct loadcmd *, ElfW (Addr), bool) attribute_hidden; +extern void _dl_map_segment_adjust + (struct loadcmd *, size_t *, ElfW(Addr), bool) attribute_hidden; -- 2.53.0