From patchwork Thu Aug 28 21:01:59 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: DJ Delorie X-Patchwork-Id: 119126 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id E83623858C74 for ; Thu, 28 Aug 2025 21:02:44 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org E83623858C74 Authentication-Results: sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=QnZd3q3W X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by sourceware.org (Postfix) with ESMTP id D46E33858C74 for ; Thu, 28 Aug 2025 21:02:05 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org D46E33858C74 Authentication-Results: sourceware.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=redhat.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org D46E33858C74 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1756414926; cv=none; b=d+4RYXQ+8WKqO53P+ent01KJVLpQCD7gOBs5LuVxfwlQMHtI178pWz1KqjScoyEdcQ7yiFLCZO96OP6cgyiq4wOx62acP8rxH/wykLKpZ3FgH77lnLZS811l3PSwQVdj5S5eQrjcB8KmvrQACE6ri0Ug+UuAVgTN79O3dkv+u5M= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1756414926; c=relaxed/simple; bh=iQ2DjZfoIm2DGlCGOT7TkSuJOEsg3Gnqgwtx6QpH8no=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=Fj4VhyjQOaBjOTTbNKKe+jKTQ0/Vohy3qG6tu56BVKx5LsZWoXiU4QCZkD3ioXIAIhJo5409ReDksra/lAbImByIxiENfzpa0BqfdgMKR1ccx9XXshSW+/zNiDuRpK92i9hdwk+Yg8Fd1cS0yXnl2weFX51zUr1ObOmPasJ/vnM= ARC-Authentication-Results: i=1; server2.sourceware.org DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org D46E33858C74 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1756414925; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to; bh=iwbikgteeQ0S1fJPCF6z2ubY/pPBXKOUmtNZ2YS5lXc=; b=QnZd3q3Ww/0nZON7+oBTIzYFOwlqjHd5KBkUiuniRCL7GJ8vWZzPfI/gZSBJaaLvEuCuwU 88FErF1NMyb4TZ+7FMWS6KnIb4x1HlrUTHY4+xzrO6EZTx1TWDRTYcYqDpp12QSV941yBH bMyWwRPlmZ5ztox7UiNNWoJLww1oUNc= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-568-Y0PgyPvWNP-3jSverkR1Qg-1; Thu, 28 Aug 2025 17:02:02 -0400 X-MC-Unique: Y0PgyPvWNP-3jSverkR1Qg-1 X-Mimecast-MFC-AGG-ID: Y0PgyPvWNP-3jSverkR1Qg_1756414921 Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id B7FE41956096 for ; Thu, 28 Aug 2025 21:02:01 +0000 (UTC) Received: from greed.delorie.com (unknown [10.22.88.240]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 6E40519560BC for ; Thu, 28 Aug 2025 21:02:01 +0000 (UTC) Received: from greed.delorie.com.redhat.com (localhost [127.0.0.1]) by greed.delorie.com (8.16.1/8.16.1) with ESMTP id 57SL1xF7451831 for ; Thu, 28 Aug 2025 17:01:59 -0400 From: DJ Delorie To: libc-alpha@sourceware.org Subject: [patch v2] malloc: avoid need for tcache == NULL checks In-Reply-To: (message from Cupertino Miranda on Fri, 8 Aug 2025 10:36:39 +0100) Date: Thu, 28 Aug 2025 17:01:59 -0400 Message-ID: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: IAEo9RGNXf9Z3I1Kxn8U4JN1SFhdN71E8E-tQSdGSxc_1756414921 X-Mimecast-Originator: redhat.com X-Spam-Status: No, score=-10.7 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H5, RCVD_IN_MSPIKE_WL, RCVD_IN_VALIDITY_RPBL_BLOCKED, RCVD_IN_VALIDITY_SAFE_BLOCKED, SPF_HELO_PASS, SPF_NONE, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces~patchwork=sourceware.org@sourceware.org Avoid needing to check for tcache == NULL by initializing it to a dummy read-only tcache structure. This dummy is all zeros, so logically it is both full (when you want to put) and empty (when you want to get). Also, there are two dummies, one used for "not yet initialized" and one for "tunables say we shouldn't have a tcache". The net result is twofold: 1. Checks for tcache == NULL may be removed from the fast path. Whether this makes the fast path faster when tcache is disabled is TBD, but the normal case is tcache enabled. 2. no memory for tcache is allocated if tunables disable caching. Co-authored-by: Florian Weimer -- >8 -- v2: removed inits from malloc/calloc hot paths. Added to malloc slow path. Redid tcache_init to avoid recursion. diff --git a/malloc/malloc.c b/malloc/malloc.c index e08873cad5..cf1bbd9a26 100644 --- a/malloc/malloc.c +++ b/malloc/malloc.c @@ -3076,8 +3076,35 @@ typedef struct tcache_perthread_struct tcache_entry *entries[TCACHE_MAX_BINS]; } tcache_perthread_struct; +static const union +{ + struct tcache_perthread_struct inactive; + struct + { + char pad; + struct tcache_perthread_struct disabled; + }; +} __tcache_dummy; + static __thread bool tcache_shutting_down = false; -static __thread tcache_perthread_struct *tcache = NULL; + +/* TCACHE is never NULL; it's either "live" or points to one of the + above dummy entries. The dummy entries are all zero so act like an + empty/unusable tcache. */ +static __thread tcache_perthread_struct *tcache = (tcache_perthread_struct *) &__tcache_dummy.inactive; + +/* This is the default, and means "check to see if a real tcache + should be allocated." */ +#define TCACHE_INACTIVE() (tcache == &__tcache_dummy.inactive) +/* This means "the user has disabled the tcache but we have to point + to something." */ +#define TCACHE_DISABLED() (tcache == &__tcache_dummy.disabled) +/* This means the tcache is active. */ +#define TCACHE_ENABLED() (!TCACHE_INACTIVE() && !TCACHE_DISABLED()) +/* Sets the tcache to INACTIVE state. */ +#define TCACHE_SET_INACTIVE() tcache = (tcache_perthread_struct *) &__tcache_dummy.inactive; +/* Sets the tcache to DISABLED state. */ +#define TCACHE_SET_DISABLED() tcache = (tcache_perthread_struct *) &__tcache_dummy.disabled; /* Process-wide key to try and catch a double-free in the same thread. */ static uintptr_t tcache_key; @@ -3237,7 +3264,7 @@ tcache_get_align (size_t nb, size_t alignment) { if (nb < mp_.tcache_max_bytes) { - if (__glibc_unlikely (tcache == NULL)) + if (__glibc_unlikely (TCACHE_INACTIVE ())) { tcache_init (); return NULL; @@ -3305,14 +3332,14 @@ tcache_thread_shutdown (void) { int i; tcache_perthread_struct *tcache_tmp = tcache; + int need_free = TCACHE_ENABLED (); tcache_shutting_down = true; - if (!tcache) - return; - /* Disable the tcache and prevent it from being reinitialized. */ - tcache = NULL; + TCACHE_SET_DISABLED (); + if (! need_free) + return; /* Free all of the entries and the tcache itself back to the arena heap for coalescing. */ @@ -3345,10 +3372,20 @@ tcache_init (void) if (MAX_TCACHE_SMALL_SIZE >= GLRO (dl_pagesize) / 2) malloc_printerr ("max tcache size too large"); + /* Set this unconditionally to avoid infinite loops in + __libc_malloc2 below. */ + TCACHE_SET_DISABLED (); + if (mp_.tcache_count == 0) + return; + size_t bytes = sizeof (tcache_perthread_struct); tcache = (tcache_perthread_struct *) __libc_malloc2 (bytes); - if (tcache != NULL) + if (tcache == NULL) + { + TCACHE_SET_INACTIVE (); + } + else { memset (tcache, 0, bytes); for (int i = 0; i < TCACHE_MAX_BINS; i++) @@ -3356,20 +3393,6 @@ tcache_init (void) } } -static void * __attribute_noinline__ -tcache_calloc_init (size_t bytes) -{ - tcache_init (); - return __libc_calloc2 (bytes); -} - -static void * __attribute_noinline__ -tcache_malloc_init (size_t bytes) -{ - tcache_init (); - return __libc_malloc2 (bytes); -} - #else /* !USE_TCACHE */ static void @@ -3388,6 +3411,11 @@ __libc_malloc2 (size_t bytes) mstate ar_ptr; void *victim; +#if USE_TCACHE + if (__glibc_unlikely (TCACHE_INACTIVE ())) + tcache_init (); +#endif + if (SINGLE_THREAD_P) { victim = tag_new_usable (_int_malloc (&main_arena, bytes)); @@ -3427,8 +3455,6 @@ __libc_malloc (size_t bytes) if (nb < mp_.tcache_max_bytes) { size_t tc_idx = csize2tidx (nb); - if(__glibc_unlikely (tcache == NULL)) - return tcache_malloc_init (bytes); if (__glibc_likely (tc_idx < TCACHE_SMALL_BINS)) { @@ -3473,7 +3499,7 @@ __libc_free (void *mem) return malloc_printerr_tail ("free(): invalid pointer"); #if USE_TCACHE - if (__glibc_likely (size < mp_.tcache_max_bytes && tcache != NULL)) + if (__glibc_likely (size < mp_.tcache_max_bytes)) { /* Check to see if it's already in the tcache. */ tcache_entry *e = (tcache_entry *) chunk2mem (p); @@ -3860,9 +3886,6 @@ __libc_calloc (size_t n, size_t elem_size) if (nb < mp_.tcache_max_bytes) { - if (__glibc_unlikely (tcache == NULL)) - return tcache_calloc_init (bytes); - size_t tc_idx = csize2tidx (nb); if (__glibc_unlikely (tc_idx < TCACHE_SMALL_BINS)) @@ -3994,7 +4017,7 @@ _int_malloc (mstate av, size_t bytes) /* While we're here, if we see other chunks of the same size, stash them in the tcache. */ size_t tc_idx = csize2tidx (nb); - if (tcache != NULL && tc_idx < mp_.tcache_small_bins) + if (tc_idx < mp_.tcache_small_bins) { mchunkptr tc_victim; @@ -4054,7 +4077,7 @@ _int_malloc (mstate av, size_t bytes) /* While we're here, if we see other chunks of the same size, stash them in the tcache. */ size_t tc_idx = csize2tidx (nb); - if (tcache != NULL && tc_idx < mp_.tcache_small_bins) + if (tc_idx < mp_.tcache_small_bins) { mchunkptr tc_victim; @@ -4116,7 +4139,7 @@ _int_malloc (mstate av, size_t bytes) #if USE_TCACHE INTERNAL_SIZE_T tcache_nb = 0; size_t tc_idx = csize2tidx (nb); - if (tcache != NULL && tc_idx < mp_.tcache_small_bins) + if (tc_idx < mp_.tcache_small_bins) tcache_nb = nb; int return_cached = 0;