From patchwork Thu May 14 01:31:01 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "H.J. Lu" X-Patchwork-Id: 134945 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id E1ABE4BB58F2 for ; Thu, 14 May 2026 01:32:30 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org E1ABE4BB58F2 Authentication-Results: sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=L7CEWj9V X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pg1-x533.google.com (mail-pg1-x533.google.com [IPv6:2607:f8b0:4864:20::533]) by sourceware.org (Postfix) with ESMTPS id 2CD9B4BB58D0 for ; Thu, 14 May 2026 01:31:39 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 2CD9B4BB58D0 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 2CD9B4BB58D0 Authentication-Results: sourceware.org; arc=pass smtp.remote-ip=2607:f8b0:4864:20::533 ARC-Seal: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1778722299; cv=pass; b=JzNpudxKNzbdtkrVxHxfvbUieZqBx9FHx4JN0DlyjLrTYb41v8OQGvmVmEiGyiR8y+QNsi+HAttAytey4JHKJ84IvLnuFhw3MdLPjpWe+dqcsRzqRlC08/j+dyHt1GOPJ7YpzXiI+aCTNwAmIaF1Juhu2auksIIYVpH8D/3r44Y= ARC-Message-Signature: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1778722299; c=relaxed/simple; bh=azjyUB5SV2ng3C/eK4MMmN7Ik42ySXw4oF9fFgy9bgA=; h=DKIM-Signature:MIME-Version:From:Date:Message-ID:Subject:To; b=lRLUypbgHMaTGkMCKeAwg+igG9k9Y7lnkr1xsSMZur/X4zn62LzPAZx+yuGTn5DKA62LYFjZP4R9/jo2+QEyOON1bQX6VdpFEyHlfTPE6rFAw5OusRncJvVbyji8pC27Y0rZ9s86s+Wc7zBQ5BG63haGySsERk0kCI2dVTZc5ys= ARC-Authentication-Results: i=2; sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=L7CEWj9V DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 2CD9B4BB58D0 Received: by mail-pg1-x533.google.com with SMTP id 41be03b00d2f7-c80167f56cdso3101758a12.3 for ; Wed, 13 May 2026 18:31:39 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1778722298; cv=none; d=google.com; s=arc-20240605; b=iGDTUMbLwdsMf5dJmVC/47h6rCbrNOlgjWm6TckfXweaOmLrWchgs0r8asgnawNRLR QO3ej8VbGKiiTvWTz+HSTboOlx3rz6fdF7krQ4xJ/fuYzzw8l24NP7ygv+OqvW6NEOW7 jsBh7DfOEKbef9YB7hCMWoFNZxXeqkgOrJ0Rxj48u90ZJbdjFrC2kaN9P10CNZaO+j1Z QPkHXcuziSRu0n0PVDq1wBoD2JmMty3FFqT4zJOWFyq14WPWEklFhDTEYdPB2rJBjWKP 7aQzpw5bse4W4qBeeM/ZHmXJdQ/OW15jQ0ekrma5/JIWaX9WwdUtikWo13RzwHYzYbwR Cvxg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=to:subject:message-id:date:from:mime-version:dkim-signature; bh=jDmv1DS/Ya7UvltF35i8l5NnWj8TTWy6f2MCQEs043Q=; fh=HkPdYZUhtUQiwenBYUZ4at1N5s/qMSOArfxSk3HulVc=; b=lI+LbGwb3sESV03xX6kvvVbtXJLBrpTe42pgD9hWFvsfYi4TaKEuvZlelC6Vlflzls +UTUwbV6Wx4D0k2cGmhWy3vS0CN2cpYT3s62+epoKytOEPoQsn6Nhql1947bzppXPdYZ FXw5LpPQLXzA9cmH3vbmUeRoEWoEr/k7gPi0NHVkyjqTqzfDf/SqBFb2UehCTzbln+y+ nIkN5AiJQfYnXZN9bzeZthHwPXP6WQBNzVnvE8jMPbz/LCCD7KAQppDmtP5KTXYl1HWg jjOAQ4MPo/6JS0ubx2NWSQdxezH5YjAbAUxoDi2AQ4C1VCl5PZg9mrI0U1R24Lfun1e6 Gmfg==; darn=sourceware.org ARC-Authentication-Results: i=1; mx.google.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1778722298; x=1779327098; darn=sourceware.org; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=jDmv1DS/Ya7UvltF35i8l5NnWj8TTWy6f2MCQEs043Q=; b=L7CEWj9VBb3HXt8VuSinmFL2jZiPLD0WUAkL3CR9VZoFXOclEMSQzUQqQ27RG1Qp1T BHv9DyacZwvCDqk++mzttgFVkvCstD7B/CJX+KF2fKAW3Oy0XAYV4J2RvbCqQo+uLJk8 uLyBC1iNEiZErZCpIqouiXdo0cwuiGJvjePEedG20yZ9gCnCvelgB+QhXlxGtICqBELv mHUDX8ktK3Xfp7LJntBnfFQAwPiFmq2w+jtM2P7ni7SgGqkK/p0o5tuwFNLBMz1wW5M5 76IFGLEe8iBuSQwq7Kd6h2L8MyAxndNzvb7VhaV7Ra+JAMOv/0li3MwFxRNHlaa/ZwWg duEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778722298; x=1779327098; h=to:subject:message-id:date:from:mime-version:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=jDmv1DS/Ya7UvltF35i8l5NnWj8TTWy6f2MCQEs043Q=; b=g4q0+Ndsyo2Em+4iXWVinLMDmMQMtVYl7izBswFM0q0vGcKwK4yfea1TILgqrkIHzq TRG1+GS+Y9vvQ2nObcrXEhW9bspeXWODidbuTFt3+AndTuueT2O4VgeZG5Uz9hLA7IoA AENkt51ANRDSZrjWMi79zY4w6xYuvxt6kQJojnySkJRAMWCfrLm6KJ8jjUA1JPh/xcmG A5KlmEpZ5sECYyH9YyDqK4XEE0i8zn/vNrDllf5muUspNC+962uYDxnmj0rUTT5z57W3 evyprYYeK519KkCoNInUqIEEzu3BtHsGMO85rm7Ol44Gl+J8HzU2ML9c8OF9l1Q6RItI 9ciw== X-Gm-Message-State: AOJu0Yw5dYNFAr9q6E2qa+jtQR2bIseGtqUm9uo/PQrXL1SJyOm7RVlo F3cge1eKhYJe57rCDw6PwWl7ZiF7rCWd69Wz8h8kpKD0xpM+lqSVc1gHYOA9VgTHf80Qk2PihYv 6+tgTW6uSk5L5uR2tPHaABP5wspX9SCZbBFuKVUaGTg== X-Gm-Gg: Acq92OF7O6k+Jk4nSVJyx03P+lwq1AbPgJbHfnIkp6OgdKrpelfBphCHKxl2E07nv+2 LFBmHrCgT+SIohA3p862keQzs7lpppYgj07Fov0fzs7Ff6Y5s+n+EsgFlZo7Z2ju+eeLinhiwdt G27foEd/ZttECKBzYvEG8Fqo22x0dmfg9ztsA/zHiFz0X7ieuKLjDenLd4WJxvoW9+lp9YO2LOi Ssei8h6QxchJ+2IASjn7RrE9xO5+10Y8Mum2LdTpPbHO1uJnYSp+sNCxUCoE7XU8z6ob04Oykl4 phA4Fo0= X-Received: by 2002:a05:6a20:7355:b0:398:78c0:6dca with SMTP id adf61e73a8af0-3afaf5223bamr5773776637.15.1778722297679; Wed, 13 May 2026 18:31:37 -0700 (PDT) MIME-Version: 1.0 From: "H.J. Lu" Date: Thu, 14 May 2026 09:31:01 +0800 X-Gm-Features: AVHnY4LPWnbka9jDBE-yL29jmEOn_Q14_vUsRAwjCLpbDxT51wbd9AITfbH3tx4 Message-ID: Subject: [PATCH v7] elf: Support THP segment load with madvise enabled THP To: GNU C Library , DJ Delorie , Adhemerval Zanella X-Spam-Status: No, score=-3010.3 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces~patchwork=sourceware.org@sourceware.org Changes in v7: Rebased against commit 389ecf52bc75e6378b2881415b03f3c00a3ef891 Author: Shamil Abdulaev Date: Wed May 13 07:52:40 2026 +0200 libio: Fix race in _IO_new_file_init_internal initialization order [BZ #33785] --- H.J. -- The current THP segment load approach works only when THP is enabled with always in the kernel. If THP is enabled with madvise in the kernel, to enable THP segment load in an application, madvise should be called with MADV_HUGEPAGE on all THP eligible PT_LOAD segments: 1. Define DL_MAP_DEFAULT_THP_PAGESIZE in hugepages.h and default it to 0. 2. Update _dl_map_segment_align to support madvise THP mode. This fixes BZ #34079. 3. Call _dl_executable_postprocess in rtld_setup_main_map for dynamic executables and in LIBC_START_MAIN for static executables, which calls madvise with MADV_HUGEPAGE on all THP eligible PT_LOAD segments in executable. This fixes BZ #34080 for both dynamic and static executables. 4. Call _dl_postprocess_loadcmd_extra in _dl_postprocess_loadcmd, which calls madvise with MADV_HUGEPAGE on all THP eligible PT_LOAD segments when loading an object after they have been mapped in. This fixes BZ #34080 for shared objects. 5. Set the maximum page alignment on THP tests to THP page size as the default maximum page alignment may be smaller than THP page size. 6. Add tests to verify that large executable PT_LOAD segments in executables are mapped at addresses aligned to THP page size when the kernel is configured to use THP in "always" mode or "madvise" mode by inspecting /proc/self/maps to check that the mapping address is aligned to THP page size reported by the kernel. Also verify that madvise is called with MADV_HUGEPAGE when the glibc tunable glibc.elf.thp=1 is used and madvise isn't called with MADV_HUGEPAGE when the glibc tunable glibc.elf.thp=0 is used. Skip these tests if THP page size cannot be determined or if THP is not enabled in "always" mode nor "madvise" mode. Quote WANG Rui : From benchmarking a clang build of the Linux kernel on x86_64 with your patch in THP madvise mode, I observed that iTLB misses were reduced, similar to what we see in THP always mode. NB: Some THP tests fail on arm due to limitations of arm32 kABI: https://sourceware.org/bugzilla/show_bug.cgi?id=34096 From 0172a61ad9a870882dcec0e1a82e646ed23c5614 Mon Sep 17 00:00:00 2001 From: "H.J. Lu" Date: Mon, 13 Apr 2026 08:23:05 +0800 Subject: [PATCH v7] elf: Support THP segment load with madvise enabled THP The current THP segment load approach works only when THP is enabled with always in the kernel. If THP is enabled with madvise in the kernel, to enable THP segment load in an application, madvise should be called with MADV_HUGEPAGE on all THP eligible PT_LOAD segments: 1. Define DL_MAP_DEFAULT_THP_PAGESIZE in hugepages.h and default it to 0. 2. Update _dl_map_segment_align to support madvise THP mode. This fixes BZ #34079. 3. Call _dl_executable_postprocess in rtld_setup_main_map for dynamic executables and in LIBC_START_MAIN for static executables, which calls madvise with MADV_HUGEPAGE on all THP eligible PT_LOAD segments in executable. This fixes BZ #34080 for both dynamic and static executables. 4. Call _dl_postprocess_loadcmd_extra in _dl_postprocess_loadcmd, which calls madvise with MADV_HUGEPAGE on all THP eligible PT_LOAD segments when loading an object after they have been mapped in. This fixes BZ #34080 for shared objects. 5. Set the maximum page alignment on THP tests to THP page size as the default maximum page alignment may be smaller than THP page size. 6. Add tests to verify that large executable PT_LOAD segments in executables are mapped at addresses aligned to THP page size when the kernel is configured to use THP in "always" mode or "madvise" mode by inspecting /proc/self/maps to check that the mapping address is aligned to THP page size reported by the kernel. Also verify that madvise is called with MADV_HUGEPAGE when the glibc tunable glibc.elf.thp=1 is used and madvise isn't called with MADV_HUGEPAGE when the glibc tunable glibc.elf.thp=0 is used. Skip these tests if THP page size cannot be determined or if THP is not enabled in "always" mode nor "madvise" mode. Quote WANG Rui : From benchmarking a clang build of the Linux kernel on x86_64 with your patch in THP madvise mode, I observed that iTLB misses were reduced, similar to what we see in THP always mode. NB: Some THP tests fail on arm due to limitations of arm32 kABI: https://sourceware.org/bugzilla/show_bug.cgi?id=34096 Signed-off-by: H.J. Lu --- csu/libc-start.c | 4 + elf/dl-load.h | 6 +- elf/dl-map-segments.h | 4 +- elf/dl-support.c | 6 + elf/rtld.c | 17 +-- sysdeps/generic/dl-exec-post.h | 34 +++++ sysdeps/generic/dl-load-post.h | 31 +++++ sysdeps/generic/hugepages.h | 8 +- sysdeps/generic/ldsodefs.h | 12 ++ sysdeps/unix/sysv/linux/Makefile | 125 +++++++++++++++++- sysdeps/unix/sysv/linux/arm/Makefile | 7 + sysdeps/unix/sysv/linux/dl-exec-post.h | 108 +++++++++++++++ sysdeps/unix/sysv/linux/dl-load-post.h | 37 ++++++ .../unix/sysv/linux/dl-map-segment-align.c | 36 ++--- .../unix/sysv/linux/dl-map-segment-align.h | 17 ++- sysdeps/unix/sysv/linux/ldsodefs.h | 3 + sysdeps/unix/sysv/linux/loongarch/Makefile | 3 + .../{dl-map-segment-align.h => hugepages.h} | 4 +- sysdeps/unix/sysv/linux/strace-tst-thp.sh | 56 ++++++++ .../unix/sysv/linux/tst-thp-1-no-s-code-pde.c | 19 +++ .../sysv/linux/tst-thp-1-no-s-code-static.c | 19 +++ sysdeps/unix/sysv/linux/tst-thp-1-no-s-code.c | 19 +++ sysdeps/unix/sysv/linux/tst-thp-1-pde.c | 19 +++ sysdeps/unix/sysv/linux/tst-thp-1-static.c | 19 +++ sysdeps/unix/sysv/linux/tst-thp-1.c | 28 ++++ sysdeps/unix/sysv/linux/tst-thp-align-check.h | 124 +++++++++++++++++ sysdeps/unix/sysv/linux/tst-thp-align.c | 123 +---------------- sysdeps/unix/sysv/linux/x86/hugepages.h | 22 +++ 28 files changed, 744 insertions(+), 166 deletions(-) create mode 100644 sysdeps/generic/dl-exec-post.h create mode 100644 sysdeps/generic/dl-load-post.h create mode 100644 sysdeps/unix/sysv/linux/dl-exec-post.h create mode 100644 sysdeps/unix/sysv/linux/dl-load-post.h rename sysdeps/unix/sysv/linux/loongarch/lp64/{dl-map-segment-align.h => hugepages.h} (90%) create mode 100644 sysdeps/unix/sysv/linux/strace-tst-thp.sh create mode 100644 sysdeps/unix/sysv/linux/tst-thp-1-no-s-code-pde.c create mode 100644 sysdeps/unix/sysv/linux/tst-thp-1-no-s-code-static.c create mode 100644 sysdeps/unix/sysv/linux/tst-thp-1-no-s-code.c create mode 100644 sysdeps/unix/sysv/linux/tst-thp-1-pde.c create mode 100644 sysdeps/unix/sysv/linux/tst-thp-1-static.c create mode 100644 sysdeps/unix/sysv/linux/tst-thp-1.c create mode 100644 sysdeps/unix/sysv/linux/tst-thp-align-check.h create mode 100644 sysdeps/unix/sysv/linux/x86/hugepages.h diff --git a/csu/libc-start.c b/csu/libc-start.c index 1c58561bce..be60831a42 100644 --- a/csu/libc-start.c +++ b/csu/libc-start.c @@ -205,6 +205,7 @@ call_fini (void *unused) #endif /* !SHARED */ #include +#include STATIC int LIBC_START_MAIN (int (*main) (int, char **, char ** MAIN_AUXVEC_DECL), @@ -305,6 +306,9 @@ LIBC_START_MAIN (int (*main) (int, char **, char ** MAIN_AUXVEC_DECL), __pointer_chk_guard_local = pointer_chk_guard; # endif + struct link_map *main_map = _dl_get_dl_main_map (); + _dl_executable_postprocess (main_map, GL(dl_phdr), GL(dl_phnum)); + #endif /* !SHARED */ /* Register the destructor of the dynamic linker if there is any. */ diff --git a/elf/dl-load.h b/elf/dl-load.h index e58028038c..2f5784b46b 100644 --- a/elf/dl-load.h +++ b/elf/dl-load.h @@ -83,6 +83,7 @@ struct loadcmd int prot; /* PROT_* bits. */ }; +#include /* Iterator for PT_LOAD program header segments. It should be initialized by _dl_pt_load_iterator_init once, then _dl_pt_load_iterator_next @@ -142,7 +143,7 @@ _dl_pt_load_iterator_next (struct dl_pt_load_iterator *it, struct loadcmd *c) responsible for setting the l_phdr fields */ static __always_inline void _dl_postprocess_loadcmd (struct link_map *l, const ElfW(Ehdr) *header, - const struct loadcmd *c) + const struct loadcmd *c, bool need_extra) { if (l->l_phdr == NULL && c->mapoff <= header->e_phoff @@ -151,6 +152,9 @@ _dl_postprocess_loadcmd (struct link_map *l, const ElfW(Ehdr) *header, /* Found the program header in this segment. */ l->l_phdr = (void *) (uintptr_t) (c->mapstart + header->e_phoff - c->mapoff); + + if (need_extra) + _dl_postprocess_loadcmd_extra (l, c); } diff --git a/elf/dl-map-segments.h b/elf/dl-map-segments.h index 1854fab7dc..c3ba7b93c5 100644 --- a/elf/dl-map-segments.h +++ b/elf/dl-map-segments.h @@ -84,6 +84,8 @@ _dl_map_segments (struct link_map *l, int fd, struct loadcmd c = { 0 }; _dl_pt_load_iterator_next (it, &c); + bool need_exta = _dl_postprocess_loadcmd_extra_needed (); + if (__glibc_likely (type == ET_DYN)) { /* This is a position-independent shared object. We can let the @@ -147,7 +149,7 @@ _dl_map_segments (struct link_map *l, int fd, return DL_MAP_SEGMENTS_ERROR_MAP_SEGMENT; postmap: - _dl_postprocess_loadcmd (l, header, &c); + _dl_postprocess_loadcmd (l, header, &c, need_exta); if (c.allocend > c.dataend) { diff --git a/elf/dl-support.c b/elf/dl-support.c index 0508d6113b..a8114de003 100644 --- a/elf/dl-support.c +++ b/elf/dl-support.c @@ -179,6 +179,12 @@ int _dl_stack_cache_lock; #endif struct dl_scope_free_list *_dl_scope_free_list; +#ifdef HAVE_THP +int _dl_elf_thp_control = -1; +enum thp_mode_t _dl_thp_mode; +size_t _dl_elf_thp_pagesize; +#endif + #ifdef NEED_DL_SYSINFO /* Needed for improved syscall handling on at least x86/Linux. NB: Don't initialize it here to avoid RELATIVE relocation in static PIE. */ diff --git a/elf/rtld.c b/elf/rtld.c index e926ec73e4..68e5476189 100644 --- a/elf/rtld.c +++ b/elf/rtld.c @@ -52,6 +52,7 @@ #include #include #include +#include #include @@ -323,6 +324,9 @@ struct rtld_global _rtld_global = /* Generally the default presumption without further information is an * executable stack but this is not true for all platforms. */ ._dl_stack_prot_flags = DEFAULT_STACK_PROT_PERMS, +#ifdef HAVE_THP + ._dl_elf_thp_control = -1, +#endif #ifdef _LIBC_REENTRANT ._dl_load_lock = _RTLD_LOCK_RECURSIVE_INITIALIZER, ._dl_load_write_lock = _RTLD_LOCK_RECURSIVE_INITIALIZER, @@ -1209,14 +1213,8 @@ rtld_setup_main_map (struct link_map *main_map) main_map->l_relro_size = ph->p_memsz; break; } - /* Process program headers again, but scan them backwards since - PT_GNU_PROPERTY is close to the end of program headers. */ - for (const ElfW(Phdr) *ph = &phdr[phnum]; ph != phdr; --ph) - if (ph[-1].p_type == PT_GNU_PROPERTY) - { - _dl_process_pt_gnu_property (main_map, -1, &ph[-1]); - break; - } + + _dl_executable_postprocess (main_map, phdr, phnum); /* Adjust the address of the TLS initialization image in case the executable is actually an ET_DYN object. */ @@ -1589,6 +1587,9 @@ dl_main (const ElfW(Phdr) *phdr, { RTLD_TIMING_VAR (start); rtld_timer_start (&start); +#ifdef HAVE_THP + _dl_get_thp_config (); +#endif _dl_map_object (NULL, rtld_progname, lt_executable, 0, __RTLD_OPENEXEC, LM_ID_BASE); rtld_timer_stop (&load_time, start); diff --git a/sysdeps/generic/dl-exec-post.h b/sysdeps/generic/dl-exec-post.h new file mode 100644 index 0000000000..f5dcdc093a --- /dev/null +++ b/sysdeps/generic/dl-exec-post.h @@ -0,0 +1,34 @@ +/* _dl_executable_postprocess. Generic version. + Copyright (C) 2026 Free Software Foundation, Inc. + Copyright The GNU Toolchain Authors. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +static inline void +_dl_executable_postprocess (struct link_map *main_map, + const ElfW(Phdr) *phdr, ElfW(Word) phnum) +{ +#ifdef SHARED + /* Process program headers again, but scan them backwards since + PT_GNU_PROPERTY is close to the end of program headers. */ + for (const ElfW(Phdr) *ph = &phdr[phnum]; ph != phdr; --ph) + if (ph[-1].p_type == PT_GNU_PROPERTY) + { + _dl_process_pt_gnu_property (main_map, -1, &ph[-1]); + break; + } +#endif +} diff --git a/sysdeps/generic/dl-load-post.h b/sysdeps/generic/dl-load-post.h new file mode 100644 index 0000000000..00de62a31f --- /dev/null +++ b/sysdeps/generic/dl-load-post.h @@ -0,0 +1,31 @@ +/* _dl_postprocess_loadcmd_extra. Generic version. + Copyright (C) 2026 Free Software Foundation, Inc. + Copyright The GNU Toolchain Authors. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include + +static inline bool +_dl_postprocess_loadcmd_extra_needed (void) +{ + return false; +} + +static inline void +_dl_postprocess_loadcmd_extra (struct link_map *l, const struct loadcmd *c) +{ +} diff --git a/sysdeps/generic/hugepages.h b/sysdeps/generic/hugepages.h index 5fc9b5c8de..f7f4957e79 100644 --- a/sysdeps/generic/hugepages.h +++ b/sysdeps/generic/hugepages.h @@ -26,10 +26,10 @@ unsigned long int __get_thp_size (void) attribute_hidden; enum thp_mode_t { + thp_mode_not_supported = 0, thp_mode_always, thp_mode_madvise, - thp_mode_never, - thp_mode_not_supported + thp_mode_never }; enum thp_mode_t __get_thp_mode (void) attribute_hidden; @@ -45,6 +45,10 @@ void __get_hugepage_config (size_t requested, size_t *pagesize, int *flags) # define MALLOC_DEFAULT_THP_PAGESIZE 0 #endif +#ifndef DL_MAP_DEFAULT_THP_PAGESIZE +# define DL_MAP_DEFAULT_THP_PAGESIZE 0 +#endif + #ifndef MAX_THP_PAGESIZE # define MAX_THP_PAGESIZE (32 * 1024 * 1024) #endif diff --git a/sysdeps/generic/ldsodefs.h b/sysdeps/generic/ldsodefs.h index 15c4659853..733b7d6727 100644 --- a/sysdeps/generic/ldsodefs.h +++ b/sysdeps/generic/ldsodefs.h @@ -39,6 +39,7 @@ #include #include #include +#include __BEGIN_DECLS @@ -477,6 +478,17 @@ struct rtld_global EXTERN struct __pthread **_dl_pthread_threads; __mach_rwlock_define (EXTERN, _dl_pthread_threads_lock) #endif +#ifdef HAVE_THP + /* The THP segment load control: + > 0: Enabled by GLIBC_TUNABLES=glibc.elf.thp=1. + 0: Disabled by GLIBC_TUNABLES=glibc.elf.thp=0. + < 0: To be enabled or disabled by GLIBC_TUNABLES. */ + EXTERN int _dl_elf_thp_control; + /* The kernel THP mode. */ + EXTERN enum thp_mode_t _dl_thp_mode; + /* Page size used for THP segment load. */ + EXTERN size_t _dl_elf_thp_pagesize; +#endif #ifdef SHARED }; # define __rtld_global_attribute__ diff --git a/sysdeps/unix/sysv/linux/Makefile b/sysdeps/unix/sysv/linux/Makefile index 63e7046cb3..8a654f66d4 100644 --- a/sysdeps/unix/sysv/linux/Makefile +++ b/sysdeps/unix/sysv/linux/Makefile @@ -698,11 +698,15 @@ $(objpfx)pldd: $(objpfx)xmalloc.o tests += \ tst-rseq-tls-range \ tst-rseq-tls-range-4096 \ + tst-thp-1 \ + tst-thp-1-pde \ + tst-thp-1-static \ tst-thp-align \ # tests tests-static += \ tst-rseq-tls-range-4096-static \ tst-rseq-tls-range-static \ + tst-thp-1-static \ # tests-static modules-names += \ tst-rseq-tls-range-mod \ @@ -712,15 +716,12 @@ CFLAGS-tst-rseq-tls-range.c += -DMAIN_TLS_ALIGN=4 CFLAGS-tst-rseq-tls-range-4096.c += -DMAIN_TLS_ALIGN=4096 CFLAGS-tst-rseq-tls-range-static.c += -DMAIN_TLS_ALIGN=4 CFLAGS-tst-rseq-tls-range-4096-static.c += -DMAIN_TLS_ALIGN=4096 -LDFLAGS-tst-thp-size-mod.so += -Wl,-z,noseparate-code $(objpfx)tst-rseq-tls-range.out: $(objpfx)tst-rseq-tls-range-mod.so $(objpfx)tst-rseq-tls-range-4096.out: $(objpfx)tst-rseq-tls-range-mod.so $(objpfx)tst-rseq-tls-range-static.out: $(objpfx)tst-rseq-tls-range-mod.so $(objpfx)tst-rseq-tls-range-4096-static.out: $(objpfx)tst-rseq-tls-range-mod.so -$(objpfx)tst-thp-align.out: $(objpfx)tst-thp-size-mod.so tst-rseq-tls-range-static-ENV = LD_LIBRARY_PATH=$(objpfx):$(common-objpfx) tst-rseq-tls-range-4096-static-ENV = LD_LIBRARY_PATH=$(objpfx):$(common-objpfx) -tst-thp-align-ENV = GLIBC_TUNABLES=glibc.elf.thp=1 test-internal-extras += tst-nolink-libc ifeq ($(run-built-tests),yes) @@ -729,6 +730,124 @@ tests-special += \ $(objpfx)tst-nolink-libc-2.out \ # tests-special endif + +ifndef THP-PAGE-SIZE +# Align PT_LOAD segments in THP tests to THP page size so that kernel will +# map PIE to the address aligned to THP page size. Default THP page size +# to 2MB which can be overridden in Makefile in subdirectories. +THP-PAGE-SIZE = 0x200000 +endif + +THP-PAGE-SIZE-LDFLAGS = -Wl,-z,max-page-size=$(THP-PAGE-SIZE) + +LDFLAGS-tst-thp-size-mod.so = -Wl,-z,noseparate-code \ + $(THP-PAGE-SIZE-LDFLAGS) +tst-thp-align-ENV = GLIBC_TUNABLES=glibc.elf.thp=1 +$(objpfx)tst-thp-align.out: $(objpfx)tst-thp-size-mod.so + +tests += \ + tst-thp-1-no-s-code \ + tst-thp-1-no-s-code-pde \ + tst-thp-1-no-s-code-static \ +# tests +tests-static += \ + tst-thp-1-no-s-code-static \ +# tests-static + +LDFLAGS-tst-thp-1 = -Wl,-z,separate-code $(THP-PAGE-SIZE-LDFLAGS) +LDFLAGS-tst-thp-1-pde = -Wl,-z,separate-code $(THP-PAGE-SIZE-LDFLAGS) +LDFLAGS-tst-thp-1-static = -Wl,-z,separate-code $(THP-PAGE-SIZE-LDFLAGS) +LDFLAGS-tst-thp-1-no-s-code = -Wl,-z,noseparate-code \ + $(THP-PAGE-SIZE-LDFLAGS) +LDFLAGS-tst-thp-1-no-s-code-pde = -Wl,-z,noseparate-code \ + $(THP-PAGE-SIZE-LDFLAGS) +LDFLAGS-tst-thp-1-no-s-code-static = -Wl,-z,noseparate-code \ + $(THP-PAGE-SIZE-LDFLAGS) + +$(objpfx)tst-thp-1-no-s-code: $(objpfx)tst-thp-size-mod.o +$(objpfx)tst-thp-1-no-s-code-pde: $(objpfx)tst-thp-size-mod.o +$(objpfx)tst-thp-1-no-s-code-static: $(objpfx)tst-thp-size-mod.o + +tst-thp-1-no-s-code-ENV = GLIBC_TUNABLES=glibc.elf.thp=1 +tst-thp-1-no-s-code-pde-ENV = GLIBC_TUNABLES=glibc.elf.thp=1 +tst-thp-1-no-s-code-static-ENV = GLIBC_TUNABLES=glibc.elf.thp=1 + +tst-thp-1-no-s-code-pde-no-pie = yes + +tst-thp-1-ENV = GLIBC_TUNABLES=glibc.elf.thp=1 +tst-thp-1-pde-ENV = GLIBC_TUNABLES=glibc.elf.thp=1 +tst-thp-1-static-ENV = GLIBC_TUNABLES=glibc.elf.thp=1 + +$(objpfx)tst-thp-1: $(objpfx)tst-thp-size-mod.o +$(objpfx)tst-thp-1-pde: $(objpfx)tst-thp-size-mod.o +$(objpfx)tst-thp-1-static: $(objpfx)tst-thp-size-mod.o + +tst-thp-1-pde-no-pie = yes + +# Don't run strace tests for cross-compiling. +ifeq (no,$(cross-compiling)) +thp-kernel-status = $(shell grep madvise /sys/kernel/mm/transparent_hugepage/enabled) +# Verify that madvise is called with MADV_HUGEPAGE when THP is enabled +# under madvise THP kernel. +ifneq ($(findstring [madvise],$(thp-kernel-status)),) +tests-special += \ + $(objpfx)strace-tst-thp-1-disabled.out \ + $(objpfx)strace-tst-thp-1-enabled.out \ + $(objpfx)strace-tst-thp-1-pde-disabled.out \ + $(objpfx)strace-tst-thp-1-pde-enabled.out \ + $(objpfx)strace-tst-thp-1-static-disabled.out \ + $(objpfx)strace-tst-thp-1-static-enabled.out \ +# tests-special + +$(objpfx)strace-tst-thp-1-enabled.out: \ + $(..)sysdeps/unix/sysv/linux/strace-tst-thp.sh $(objpfx)ld.so \ + $(objpfx)tst-thp-1 + $(SHELL) $< $(objpfx)ld.so '$(test-wrapper-env)' \ + '$(run-program-env) GLIBC_TUNABLES=glibc.elf.thp=1' \ + '$(rpath-link)' $(objpfx)tst-thp-1 > $@; \ + $(evaluate-test) + +$(objpfx)strace-tst-thp-1-disabled.out: \ + $(..)sysdeps/unix/sysv/linux/strace-tst-thp.sh $(objpfx)ld.so \ + $(objpfx)tst-thp-1 + $(SHELL) $< $(objpfx)ld.so '$(test-wrapper-env)' \ + '$(run-program-env) GLIBC_TUNABLES=glibc.elf.thp=0' \ + '$(rpath-link)' $(objpfx)tst-thp-1 > $@; \ + $(evaluate-test) + +$(objpfx)strace-tst-thp-1-pde-enabled.out: \ + $(..)sysdeps/unix/sysv/linux/strace-tst-thp.sh $(objpfx)ld.so \ + $(objpfx)tst-thp-1-pde + $(SHELL) $< $(objpfx)ld.so '$(test-wrapper-env)' \ + '$(run-program-env) GLIBC_TUNABLES=glibc.elf.thp=1' \ + '$(rpath-link)' $(objpfx)tst-thp-1-pde > $@; \ + $(evaluate-test) + +$(objpfx)strace-tst-thp-1-pde-disabled.out: \ + $(..)sysdeps/unix/sysv/linux/strace-tst-thp.sh $(objpfx)ld.so \ + $(objpfx)tst-thp-1-pde + $(SHELL) $< $(objpfx)ld.so '$(test-wrapper-env)' \ + '$(run-program-env) GLIBC_TUNABLES=glibc.elf.thp=0' \ + '$(rpath-link)' $(objpfx)tst-thp-1-pde > $@; \ + $(evaluate-test) + +$(objpfx)strace-tst-thp-1-static-enabled.out: \ + $(..)sysdeps/unix/sysv/linux/strace-tst-thp.sh $(objpfx)ld.so \ + $(objpfx)tst-thp-1-static + $(SHELL) $< $(objpfx)ld.so '$(test-wrapper-env)' \ + '$(run-program-env) GLIBC_TUNABLES=glibc.elf.thp=1' \ + '$(rpath-link)' $(objpfx)tst-thp-1-static > $@; \ + $(evaluate-test) + +$(objpfx)strace-tst-thp-1-static-disabled.out: \ + $(..)sysdeps/unix/sysv/linux/strace-tst-thp.sh $(objpfx)ld.so \ + $(objpfx)tst-thp-1-static + $(SHELL) $< $(objpfx)ld.so '$(test-wrapper-env)' \ + '$(run-program-env) GLIBC_TUNABLES=glibc.elf.thp=0' \ + '$(rpath-link)' $(objpfx)tst-thp-1-static > $@; \ + $(evaluate-test) +endif # [madvise] +endif # $(cross-compiling) endif # $(subdir) == elf ifeq ($(subdir),rt) diff --git a/sysdeps/unix/sysv/linux/arm/Makefile b/sysdeps/unix/sysv/linux/arm/Makefile index e73ce4f811..1ee8bec9b9 100644 --- a/sysdeps/unix/sysv/linux/arm/Makefile +++ b/sysdeps/unix/sysv/linux/arm/Makefile @@ -3,6 +3,13 @@ sysdep-rtld-routines += aeabi_read_tp libc-do-syscall # The test uses INTERNAL_SYSCALL_CALL. In thumb mode, this uses # an undefined reference to __libc_do_syscall. CFLAGS-tst-nolink-libc.c += -marm + +# These tests fail on arm due to limitations of arm32 kABI: +# https://sourceware.org/bugzilla/show_bug.cgi?id=34096 +test-xfail-tst-thp-1-no-s-code-pde = yes +test-xfail-tst-thp-1-no-s-code-static = yes +test-xfail-tst-thp-1-pde = yes +test-xfail-tst-thp-1-static = yes endif ifeq ($(subdir),misc) diff --git a/sysdeps/unix/sysv/linux/dl-exec-post.h b/sysdeps/unix/sysv/linux/dl-exec-post.h new file mode 100644 index 0000000000..367b20ae2a --- /dev/null +++ b/sysdeps/unix/sysv/linux/dl-exec-post.h @@ -0,0 +1,108 @@ +/* _dl_executable_postprocess. Linux version. + Copyright (C) 2026 Free Software Foundation, Inc. + Copyright The GNU Toolchain Authors. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +static inline void +_dl_get_thp_config (void) +{ + /* Check if there is GLIBC_TUNABLES=glibc.elf.thp=[0|1]. */ + if (TUNABLE_IS_INITIALIZED_FULL (glibc, elf, thp)) + GL(dl_elf_thp_control) = TUNABLE_GET_FULL (glibc, elf, thp, int32_t, + NULL); + + /* Return if THP is disabled by GLIBC_TUNABLES=glibc.elf.thp=0. */ + if (GL(dl_elf_thp_control) == 0) + return; + + _Static_assert (DL_MAP_DEFAULT_THP_PAGESIZE <= MAX_THP_PAGESIZE, + "DL_MAP_DEFAULT_THP_PAGESIZE <= MAX_THP_PAGESIZE"); + + /* NB: Accessing /sys/kernel/mm files is quite expensive and the file + may not be accessible in containers. If DL_MAP_DEFAULT_THP_PAGESIZE + is non-zero, assume THP mode is madvise and always call madvise. + Since madvise is a fast systemcall, it adds only a small overhead + compared to the cost of accessing /sys/kernel/mm files. */ + if (DL_MAP_DEFAULT_THP_PAGESIZE != 0) + { + GL(dl_elf_thp_pagesize) = DL_MAP_DEFAULT_THP_PAGESIZE; + GL(dl_thp_mode) = thp_mode_madvise; + } + else + { + GL(dl_elf_thp_pagesize) = __get_thp_size (); + GL(dl_thp_mode) = __get_thp_mode (); + /* We cap the huge page size at MAX_THP_PAGESIZE to avoid + over-aligning on systems with very large normal pages + (like 64K pages with 512M huge pages). */ + if (GL(dl_elf_thp_pagesize) > MAX_THP_PAGESIZE) + GL(dl_elf_thp_control) = 0; + /* NB: Enable THP if THP is always enabled in the kernel. */ + else if (GL(dl_thp_mode) == thp_mode_always) + GL(dl_elf_thp_control) = 1; + } +} + +static inline void +_dl_executable_postprocess (struct link_map *main_map, + const ElfW(Phdr) *phdr, ElfW(Word) phnum) +{ + if (GL(dl_elf_thp_control) == -1) + _dl_get_thp_config (); + + /* NB: In static executable, PT_GNU_PROPERTY is processed in target + libc-start.h if it is needed by target. When ld.so is used, if + a target doesn't need PT_GNU_PROPERTY, _dl_process_pt_gnu_property + is an empty function. */ +#ifdef SHARED + /* Process program headers again, but scan them backwards since + PT_GNU_PROPERTY is close to the end of program headers. */ + for (const ElfW(Phdr) *ph = &phdr[phnum]; ph != phdr; --ph) + if (ph[-1].p_type == PT_GNU_PROPERTY) + { + _dl_process_pt_gnu_property (main_map, -1, &ph[-1]); + break; + } +#endif + + /* Return if THP segment load isn't enabled. */ + if (GL(dl_elf_thp_control) <= 0) + return; + + /* NB: If DL_MAP_DEFAULT_THP_PAGESIZE is non-zero, dl_thp_mode is set + to thp_mode_madvise. */ + if (DL_MAP_DEFAULT_THP_PAGESIZE == 0 + && GL(dl_thp_mode) != thp_mode_madvise) + return; + + /* When we get here, the main executable have been mapped in. Call + madvise with MADV_HUGEPAGE for all THP eligible PT_LOAD segments. */ + + const ElfW(Phdr) *ph; + + size_t thp_pagesize = GL(dl_elf_thp_pagesize); + + /* Call __madvise if offset and address of the PT_LOAD segment are + aligned to THP page size and it is read-only. */ + for (ph = phdr; ph < &phdr[phnum]; ++ph) + if (ph->p_type == PT_LOAD + && ph->p_memsz >= thp_pagesize + && ((ph->p_vaddr | ph->p_offset) & (thp_pagesize - 1)) == 0 + && (ph->p_flags & (PF_W | PF_R)) == PF_R) + __madvise ((void *) (main_map->l_addr + ph->p_vaddr), + ph->p_memsz, MADV_HUGEPAGE); +} diff --git a/sysdeps/unix/sysv/linux/dl-load-post.h b/sysdeps/unix/sysv/linux/dl-load-post.h new file mode 100644 index 0000000000..1fa86bb9f2 --- /dev/null +++ b/sysdeps/unix/sysv/linux/dl-load-post.h @@ -0,0 +1,37 @@ +/* _dl_postprocess_loadcmd_extra. Linux version. + Copyright (C) 2026 Free Software Foundation, Inc. + Copyright The GNU Toolchain Authors. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +static bool _dl_segment_thp_eligible (const struct loadcmd *, size_t); + +static inline bool +_dl_postprocess_loadcmd_extra_needed (void) +{ + return GL(dl_thp_mode) == thp_mode_madvise; +} + +/* After L has been mapped in, call madvise with MADV_HUGEPAGE if L is + THP eligible. */ + +static inline void +_dl_postprocess_loadcmd_extra (struct link_map *l, const struct loadcmd *c) +{ + if (_dl_segment_thp_eligible (c, GL(dl_elf_thp_pagesize))) + __madvise ((void *) (l->l_addr + c->mapstart), + c->mapend - c->mapstart, MADV_HUGEPAGE); +} diff --git a/sysdeps/unix/sysv/linux/dl-map-segment-align.c b/sysdeps/unix/sysv/linux/dl-map-segment-align.c index a39e74d91b..bbfd88bc5e 100644 --- a/sysdeps/unix/sysv/linux/dl-map-segment-align.c +++ b/sysdeps/unix/sysv/linux/dl-map-segment-align.c @@ -17,38 +17,26 @@ License along with the GNU C Library; if not, see . */ +#include #include -#include -#include + +/* Return the alignment of the PT_LOAD segment for THP. P_ALIGN_MAX is + the maximum p_align value in the PT_LOAD segment. */ ElfW (Addr) _dl_map_segment_align (const struct loadcmd *c, ElfW (Addr) p_align_max) { - static enum thp_mode_t thp_mode = thp_mode_not_supported; - static unsigned long int thp_pagesize; + enum thp_mode_t thp_mode = GL(dl_thp_mode); + size_t thp_pagesize = GL(dl_elf_thp_pagesize); - if (TUNABLE_GET (glibc, elf, thp, int32_t, NULL) == 0) + if (GL(dl_elf_thp_control) <= 0 + || p_align_max >= thp_pagesize + || !(thp_mode == thp_mode_always || thp_mode == thp_mode_madvise)) return p_align_max; - if (__glibc_unlikely (thp_mode == thp_mode_not_supported - || thp_pagesize == 0)) - { - unsigned long int default_thp_pagesize = DL_MAP_DEFAULT_THP_PAGESIZE; - thp_mode = default_thp_pagesize ? thp_mode_always : __get_thp_mode (); - thp_pagesize = default_thp_pagesize ? : __get_thp_size (); - } - - /* Aligning load segments that are large enough to the PMD size helps - improve THP eligibility and reduces TLB pressure. - We cap the huge page size at MAX_THP_PAGESIZE to avoid over-aligning - on systems with very large normal pages (like 64K pages with 512M - huge pages). */ - if (thp_mode == thp_mode_always - && thp_pagesize <= MAX_THP_PAGESIZE - && ((c->mapstart | c->mapoff) & (thp_pagesize - 1)) == 0 - && (c->mapend - c->mapstart) >= thp_pagesize - && p_align_max < thp_pagesize - && (c->prot & PROT_WRITE) == 0) + /* Return true if the segment is THP eligible. It helps improve THP + eligibility and reduces TLB pressure. */ + if (_dl_segment_thp_eligible (c, thp_pagesize)) return thp_pagesize; return p_align_max; diff --git a/sysdeps/unix/sysv/linux/dl-map-segment-align.h b/sysdeps/unix/sysv/linux/dl-map-segment-align.h index d9b05181b7..b904e128d8 100644 --- a/sysdeps/unix/sysv/linux/dl-map-segment-align.h +++ b/sysdeps/unix/sysv/linux/dl-map-segment-align.h @@ -19,9 +19,18 @@ #include -#ifndef DL_MAP_DEFAULT_THP_PAGESIZE -# define DL_MAP_DEFAULT_THP_PAGESIZE 0 -#endif - extern ElfW (Addr) _dl_map_segment_align (const struct loadcmd *, ElfW (Addr)) attribute_hidden; + +/* Return true only if the loadcmd C is THP eligible with THP page size + THP_PAGESIZE, which means that it is read-only, its size >= THP page + size, its offset and address of the loadcmd C are aligned to THP page + size. */ + +static inline bool +_dl_segment_thp_eligible (const struct loadcmd *c, size_t thp_pagesize) +{ + return ((c->prot & PROT_WRITE) == 0 + && (c->mapend - c->mapstart) >= thp_pagesize + && ((c->mapstart | c->mapoff) & (thp_pagesize - 1)) == 0); +} diff --git a/sysdeps/unix/sysv/linux/ldsodefs.h b/sysdeps/unix/sysv/linux/ldsodefs.h index c63b649432..e39d9afe34 100644 --- a/sysdeps/unix/sysv/linux/ldsodefs.h +++ b/sysdeps/unix/sysv/linux/ldsodefs.h @@ -21,6 +21,9 @@ /* We have the auxiliary vector. */ #define HAVE_AUX_VECTOR +/* We have transparent huge page. */ +#define HAVE_THP + /* Get the real definitions. */ #include_next diff --git a/sysdeps/unix/sysv/linux/loongarch/Makefile b/sysdeps/unix/sysv/linux/loongarch/Makefile index 0d5f087862..d5beb62440 100644 --- a/sysdeps/unix/sysv/linux/loongarch/Makefile +++ b/sysdeps/unix/sysv/linux/loongarch/Makefile @@ -12,3 +12,6 @@ abi-ilp32s-condition := __WORDSIZE == 32 && defined __loongarch_soft_float abi-ilp32d-condition := __WORDSIZE == 32 && defined __loongarch_double_float abi-lp64s-condition := __WORDSIZE == 64 && defined __loongarch_soft_float abi-lp64d-condition := __WORDSIZE == 64 && defined __loongarch_double_float + +# Align THP tests to 32MB. +THP-PAGE-SIZE = 0x2000000 diff --git a/sysdeps/unix/sysv/linux/loongarch/lp64/dl-map-segment-align.h b/sysdeps/unix/sysv/linux/loongarch/lp64/hugepages.h similarity index 90% rename from sysdeps/unix/sysv/linux/loongarch/lp64/dl-map-segment-align.h rename to sysdeps/unix/sysv/linux/loongarch/lp64/hugepages.h index c51ee4ac47..30252a9b86 100644 --- a/sysdeps/unix/sysv/linux/loongarch/lp64/dl-map-segment-align.h +++ b/sysdeps/unix/sysv/linux/loongarch/lp64/hugepages.h @@ -1,4 +1,4 @@ -/* _dl_map_segment_align. LoongArch64 Linux version. +/* Huge Page support. LoongArch64 Linux version. Copyright (C) 2026 Free Software Foundation, Inc. Copyright The GNU Toolchain Authors. This file is part of the GNU C Library. @@ -19,4 +19,4 @@ #define DL_MAP_DEFAULT_THP_PAGESIZE (32 * 1024 * 1024) -#include_next +#include_next diff --git a/sysdeps/unix/sysv/linux/strace-tst-thp.sh b/sysdeps/unix/sysv/linux/strace-tst-thp.sh new file mode 100644 index 0000000000..8903717fd4 --- /dev/null +++ b/sysdeps/unix/sysv/linux/strace-tst-thp.sh @@ -0,0 +1,56 @@ +#!/bin/sh +# Run THP test under strace to verify control of the THP segment load. +# Copyright (C) 2026 Free Software Foundation, Inc. +# This file is part of the GNU C Library. + +# The GNU C Library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. + +# The GNU C Library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. + +# You should have received a copy of the GNU Lesser General Public +# License along with the GNU C Library; if not, see +# . + +set -e + +rtld="$1" +test_wrapper_env="$2" +run_program_env="$3" +library_path="$4" +test_prog="$5" + +# Test whether strace is available in the test environment. If not, skip +# the test. +${test_wrapper_env} ${run_program_env} \ + /bin/sh -c "command -v strace" || exit 77 + +# Finally the actual test inside the test environment, using the just +# build ld.so and new libraries to run the THP test under strace. +if /bin/sh -c \ + "${test_wrapper_env} ${run_program_env} strace ${rtld} \ + --library-path ${library_path} ${test_prog} 2>&1 \ + | grep -E \"madvise(.*, MADV_HUGEPAGE)\""; then + case x"${run_program_env}" in + *glibc.elf.thp=1*) + exit 0 + ;; + *) + exit 1 + ;; + esac +else + case x"${run_program_env}" in + *glibc.elf.thp=0*) + exit 0 + ;; + *) + exit 1 + ;; + esac +fi diff --git a/sysdeps/unix/sysv/linux/tst-thp-1-no-s-code-pde.c b/sysdeps/unix/sysv/linux/tst-thp-1-no-s-code-pde.c new file mode 100644 index 0000000000..3fd01e9bfe --- /dev/null +++ b/sysdeps/unix/sysv/linux/tst-thp-1-no-s-code-pde.c @@ -0,0 +1,19 @@ +/* Test PDE with THP segment load linked with -Wl,-z,noseparate-code. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include "tst-thp-1.c" diff --git a/sysdeps/unix/sysv/linux/tst-thp-1-no-s-code-static.c b/sysdeps/unix/sysv/linux/tst-thp-1-no-s-code-static.c new file mode 100644 index 0000000000..d0ae0f1ff0 --- /dev/null +++ b/sysdeps/unix/sysv/linux/tst-thp-1-no-s-code-static.c @@ -0,0 +1,19 @@ +/* Test static with THP segment load linked with -Wl,-z,noseparate-code. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include "tst-thp-1.c" diff --git a/sysdeps/unix/sysv/linux/tst-thp-1-no-s-code.c b/sysdeps/unix/sysv/linux/tst-thp-1-no-s-code.c new file mode 100644 index 0000000000..5eb1e005ed --- /dev/null +++ b/sysdeps/unix/sysv/linux/tst-thp-1-no-s-code.c @@ -0,0 +1,19 @@ +/* Test THP segment load linked with -Wl,-z,noseparate-code. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include "tst-thp-1.c" diff --git a/sysdeps/unix/sysv/linux/tst-thp-1-pde.c b/sysdeps/unix/sysv/linux/tst-thp-1-pde.c new file mode 100644 index 0000000000..d854dd43da --- /dev/null +++ b/sysdeps/unix/sysv/linux/tst-thp-1-pde.c @@ -0,0 +1,19 @@ +/* Test PDE with THP segment load linked with -Wl,-z,separate-code. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include "tst-thp-1.c" diff --git a/sysdeps/unix/sysv/linux/tst-thp-1-static.c b/sysdeps/unix/sysv/linux/tst-thp-1-static.c new file mode 100644 index 0000000000..66d7e12954 --- /dev/null +++ b/sysdeps/unix/sysv/linux/tst-thp-1-static.c @@ -0,0 +1,19 @@ +/* Test static with THP segment load linked with -Wl,-z,separate-code. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include "tst-thp-1.c" diff --git a/sysdeps/unix/sysv/linux/tst-thp-1.c b/sysdeps/unix/sysv/linux/tst-thp-1.c new file mode 100644 index 0000000000..49eea7069c --- /dev/null +++ b/sysdeps/unix/sysv/linux/tst-thp-1.c @@ -0,0 +1,28 @@ +/* Test THP segment load linked with -Wl,-z,separate-code. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include "tst-thp-align-check.h" + +static int +do_test (void) +{ + check_align ("tst-thp-1"); + return 0; +} + +#include diff --git a/sysdeps/unix/sysv/linux/tst-thp-align-check.h b/sysdeps/unix/sysv/linux/tst-thp-align-check.h new file mode 100644 index 0000000000..8f1efc0ef1 --- /dev/null +++ b/sysdeps/unix/sysv/linux/tst-thp-align-check.h @@ -0,0 +1,124 @@ +/* Test the THP compatible alignment of PT_LOAD segments. + + Copyright (C) 2026 Free Software Foundation, Inc. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#undef attribute_hidden +#define attribute_hidden +#include /* For enum thp_mode_t and MAX_THP_PAGESIZE. */ +#undef attribute_hidden + +static unsigned long int +get_thp_size (void) +{ + int fd = open ("/sys/kernel/mm/transparent_hugepage/hpage_pmd_size", + O_RDONLY, 0); + if (fd == -1) + return 0; + + char str[INT_BUFSIZE_BOUND (unsigned long int)]; + ssize_t s = read (fd, str, sizeof (str)); + close (fd); + if (s < 0) + return 0; + + unsigned long int r = 0; + for (ssize_t i = 0; i < s; i++) + { + if (str[i] == '\n') + break; + r *= 10; + r += str[i] - '0'; + } + return r; +} + +static enum thp_mode_t +get_thp_mode (void) +{ + int fd = open ("/sys/kernel/mm/transparent_hugepage/enabled", O_RDONLY, 0); + if (fd == -1) + return thp_mode_not_supported; + + static const char mode_always[] = "[always] madvise never\n"; + static const char mode_madvise[] = "always [madvise] never\n"; + static const char mode_never[] = "always madvise [never]\n"; + + char str[sizeof(mode_always)]; + ssize_t s = read (fd, str, sizeof (str)); + if (s >= sizeof str || s < 0) + return thp_mode_not_supported; + str[s] = '\0'; + close (fd); + + if (s == sizeof (mode_always) - 1) + { + if (strcmp (str, mode_always) == 0) + return thp_mode_always; + else if (strcmp (str, mode_madvise) == 0) + return thp_mode_madvise; + else if (strcmp (str, mode_never) == 0) + return thp_mode_never; + } + return thp_mode_not_supported; +} + +static void +check_align (const char *name) +{ + unsigned long int thp_size = get_thp_size (); + enum thp_mode_t thp_mode = get_thp_mode (); + + if (thp_size == 0) + FAIL_UNSUPPORTED ("unable to get THP size.\n"); + + if (thp_size > MAX_THP_PAGESIZE) + FAIL_UNSUPPORTED ("THP size exceeds MAX_THP_PAGESIZE.\n"); + + if (thp_mode != thp_mode_always && thp_mode != thp_mode_madvise) + FAIL_UNSUPPORTED ("THP mode is not always nor madvise.\n"); + + FILE *f = xfopen ("/proc/self/maps", "r"); + char *line = NULL; + size_t len; + + while (xgetline (&line, &len, f)) + { + uintptr_t from, to; + char *prot = NULL, *path = NULL; + int r = sscanf (line, "%" SCNxPTR "-%" SCNxPTR "%ms%*s%*s%*s%ms", + &from, &to, &prot, &path); + + TEST_VERIFY (r == 3 || r == 4); + + if (strstr (prot, "x") && strstr (path, name)) + TEST_COMPARE (from % thp_size, 0); + + free (path); + } + + free (line); + xfclose (f); +} diff --git a/sysdeps/unix/sysv/linux/tst-thp-align.c b/sysdeps/unix/sysv/linux/tst-thp-align.c index 0b3f18e000..2e44109ba6 100644 --- a/sysdeps/unix/sysv/linux/tst-thp-align.c +++ b/sysdeps/unix/sysv/linux/tst-thp-align.c @@ -16,129 +16,10 @@ License along with the GNU C Library; if not, see . */ -#include -#include -#include -#include -#include -#include -#include #include -#include -#include +#include "tst-thp-align-check.h" #define THP_SIZE_MOD_NAME "tst-thp-size-mod.so" -#define MAX_THP_PAGESIZE (32 * 1024 * 1024) - -enum thp_mode_t -{ - thp_mode_always, - thp_mode_madvise, - thp_mode_never, - thp_mode_not_supported -}; - -static unsigned long int -get_thp_size (void) -{ - int fd = open ("/sys/kernel/mm/transparent_hugepage/hpage_pmd_size", - O_RDONLY, 0); - if (fd == -1) - return 0; - - char str[INT_BUFSIZE_BOUND (unsigned long int)]; - ssize_t s = read (fd, str, sizeof (str)); - close (fd); - if (s < 0) - return 0; - - unsigned long int r = 0; - for (ssize_t i = 0; i < s; i++) - { - if (str[i] == '\n') - break; - r *= 10; - r += str[i] - '0'; - } - return r; -} - -static enum thp_mode_t -get_thp_mode (void) -{ - int fd = open ("/sys/kernel/mm/transparent_hugepage/enabled", O_RDONLY, 0); - if (fd == -1) - return thp_mode_not_supported; - - static const char mode_always[] = "[always] madvise never\n"; - static const char mode_madvise[] = "always [madvise] never\n"; - static const char mode_never[] = "always madvise [never]\n"; - - char str[sizeof(mode_always)]; - ssize_t s = read (fd, str, sizeof (str)); - if (s >= sizeof str || s < 0) - return thp_mode_not_supported; - str[s] = '\0'; - close (fd); - - if (s == sizeof (mode_always) - 1) - { - if (strcmp (str, mode_always) == 0) - return thp_mode_always; - else if (strcmp (str, mode_madvise) == 0) - return thp_mode_madvise; - else if (strcmp (str, mode_never) == 0) - return thp_mode_never; - } - return thp_mode_not_supported; -} - -static void -check_align (void) -{ - unsigned long int thp_size = get_thp_size (); - enum thp_mode_t thp_mode = get_thp_mode (); - - if (thp_size == 0) - { - FAIL_UNSUPPORTED ("unable to get THP size.\n"); - return; - } - - if (thp_size > MAX_THP_PAGESIZE) - { - FAIL_UNSUPPORTED ("THP size exceeds MAX_THP_PAGESIZE.\n"); - return; - } - - if (thp_mode != thp_mode_always) - { - FAIL_UNSUPPORTED ("THP mode is not always.\n"); - return; - } - - FILE *f = xfopen ("/proc/self/maps", "r"); - char *line = NULL; - size_t len; - - while (xgetline (&line, &len, f)) - { - uintptr_t from, to; - char *prot = NULL, *path = NULL; - int r = sscanf (line, "%" SCNxPTR "-%" SCNxPTR "%ms%*s%*s%*s%ms", - &from, &to, &prot, &path); - - TEST_VERIFY (r == 3 || r == 4); - - if (strstr (prot, "x") && strstr (path, THP_SIZE_MOD_NAME)) - TEST_COMPARE (from % thp_size, 0); - - free (path); - } - - free (line); - xfclose (f); -} static int do_test (void) @@ -146,7 +27,7 @@ do_test (void) void *dl; dl = xdlopen (THP_SIZE_MOD_NAME, RTLD_NOW); - check_align (); + check_align (THP_SIZE_MOD_NAME); xdlclose (dl); return 0; diff --git a/sysdeps/unix/sysv/linux/x86/hugepages.h b/sysdeps/unix/sysv/linux/x86/hugepages.h new file mode 100644 index 0000000000..1a8c370969 --- /dev/null +++ b/sysdeps/unix/sysv/linux/x86/hugepages.h @@ -0,0 +1,22 @@ +/* Huge Page support. Linux/x86 version. + Copyright (C) 2026 Free Software Foundation, Inc. + Copyright The GNU Toolchain Authors. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#define DL_MAP_DEFAULT_THP_PAGESIZE (2 * 1024 * 1024) + +#include_next -- 2.54.0