From patchwork Thu Apr 25 20:43:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Andr=C3=A9_Almeida?= X-Patchwork-Id: 89018 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id EA2DD385840F for ; Thu, 25 Apr 2024 20:45:14 +0000 (GMT) X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from fanzine2.igalia.com (fanzine.igalia.com [178.60.130.6]) by sourceware.org (Postfix) with ESMTPS id B0A913858D20 for ; Thu, 25 Apr 2024 20:44:41 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org B0A913858D20 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=igalia.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=igalia.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org B0A913858D20 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=178.60.130.6 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1714077885; cv=none; b=Q3hjBj6rwOu0s/AZnVMFyPLp0P0IQ0x+CbAS5P27WZFfS+rhFkLnyvDPKUQS3k01szq2gmBiLLbMYvUgnBGShZJS6wBDHT9P3GtYHlJSE6Wn2VOxDCX3ykxJyBCRnonI760e5OFj3dQRw7VvgsGzhv1rT16qQpcjvteZKuJ5H2A= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1714077885; c=relaxed/simple; bh=8sxkJ4DWuOJCZutr1xA6A89cSCURGEeXGMrk/zUulQw=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=nEG/zfoh5HWJJNHof0ylFl7pp9HAhg0RHMs1PF+SSVVDxijUdGEXk7T1U60olRGoWL2gZJZxWtIDlafbw71x53fp2bXB6SXORBI1+QnLP/AhdXgw7kFtzbXzcbIjmVm85gebn70NQj32aBbPjBsG/K0ZWc8oA0LMCo1bOYXw5HU= ARC-Authentication-Results: i=1; server2.sourceware.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=igalia.com; s=20170329; h=Content-Transfer-Encoding:Content-Type:MIME-Version:References: In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=oHann4TH2Ah2IPapN8sqWvlYSxh8ekOAJ6wfcLQixXk=; b=jkyRxfd3GcBbNr6BqHzzx6A5Jo 5z4OjIrxefhD6R+pJufuaS8wwJQFKeYzajwbQteZqnubgHo7Iluz+SJdp1unxCtq0nOwcPuGe38TM S5F59RMbkqSzwbeQJNSL9GoXuDx3kaHOQGyNQvgtdt9Es4QxZQqbIxdEMdgbBWqyRmQrNQWHptUJT gJ8ji4/bm2GmNbsbWPzFs7EMChXhk6od8B1Y5170ZgRcfAV+tBPpg17FC73qt84+F4nqgNkEBMJu8 bNyLehdvELH/kHa/hye3FmD20+Ho69kLs8J+5IJ7+q64O8MhbHE5Rzvyg6W7rOE5XyzyQkAXyLylq qcg26LPg==; Received: from 201-42-129-95.dsl.telesp.net.br ([201.42.129.95] helo=steammachine.lan) by fanzine2.igalia.com with esmtpsa (Cipher TLS1.3:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim) id 1s05wd-008iV5-Tt; Thu, 25 Apr 2024 22:43:52 +0200 From: =?utf-8?q?Andr=C3=A9_Almeida?= To: Mathieu Desnoyers , Peter Zijlstra , Thomas Gleixner Cc: linux-kernel@vger.kernel.org, "Paul E . McKenney" , "Boqun Feng" , "H . Peter Anvin" , "Paul Turner" , linux-api@vger.kernel.org, "Christian Brauner" , "Florian Weimer" , David.Laight@ACULAB.COM, carlos@redhat.com, "Peter Oskolkov" , "Alexander Mikhalitsyn" , "Chris Kennelly" , "Ingo Molnar" , "Darren Hart" , "Davidlohr Bueso" , =?utf-8?q?Andr=C3=A9_Almeida?= , libc-alpha@sourceware.org, "Steven Rostedt" , "Jonathan Corbet" , "Noah Goldstein" , "Daniel Colascione" , longman@redhat.com, kernel-dev@igalia.com Subject: [RFC PATCH 1/1] futex: Add FUTEX_SPIN operation Date: Thu, 25 Apr 2024 17:43:32 -0300 Message-ID: <20240425204332.221162-2-andrealmeid@igalia.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240425204332.221162-1-andrealmeid@igalia.com> References: <20240425204332.221162-1-andrealmeid@igalia.com> MIME-Version: 1.0 X-Spam-Status: No, score=-8.5 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_BARRACUDACENTRAL, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces+patchwork=sourceware.org@sourceware.org Add a new futex mode for futex wait, the futex spin. Given the FUTEX2_SPIN flag, parse the futex value as the PID of the lock owner. Then, before going to the normal wait path, spins while the lock owner is running in a different CPU, to avoid the whole context switch operation and to quickly return to userspace. If the lock owner is not running, just sleep as the normal futex wait path. The check for the owner to be running or not is important to avoid spinning for something that won't be released quickly. Userspace is responsible on providing the proper PID, the kernel does a basic check. Signed-off-by: André Almeida --- include/uapi/linux/futex.h | 2 +- kernel/futex/futex.h | 6 ++- kernel/futex/waitwake.c | 79 +++++++++++++++++++++++++++++++++++++- 3 files changed, 83 insertions(+), 4 deletions(-) diff --git a/include/uapi/linux/futex.h b/include/uapi/linux/futex.h index d2ee625ea189..d77d692ffac2 100644 --- a/include/uapi/linux/futex.h +++ b/include/uapi/linux/futex.h @@ -63,7 +63,7 @@ #define FUTEX2_SIZE_U32 0x02 #define FUTEX2_SIZE_U64 0x03 #define FUTEX2_NUMA 0x04 - /* 0x08 */ +#define FUTEX2_SPIN 0x08 /* 0x10 */ /* 0x20 */ /* 0x40 */ diff --git a/kernel/futex/futex.h b/kernel/futex/futex.h index 8b195d06f4e8..180c1c10dc81 100644 --- a/kernel/futex/futex.h +++ b/kernel/futex/futex.h @@ -37,6 +37,7 @@ #define FLAGS_HAS_TIMEOUT 0x0040 #define FLAGS_NUMA 0x0080 #define FLAGS_STRICT 0x0100 +#define FLAGS_SPIN 0x0200 /* FUTEX_ to FLAGS_ */ static inline unsigned int futex_to_flags(unsigned int op) @@ -52,7 +53,7 @@ static inline unsigned int futex_to_flags(unsigned int op) return flags; } -#define FUTEX2_VALID_MASK (FUTEX2_SIZE_MASK | FUTEX2_PRIVATE) +#define FUTEX2_VALID_MASK (FUTEX2_SIZE_MASK | FUTEX2_PRIVATE | FUTEX2_SPIN) /* FUTEX2_ to FLAGS_ */ static inline unsigned int futex2_to_flags(unsigned int flags2) @@ -65,6 +66,9 @@ static inline unsigned int futex2_to_flags(unsigned int flags2) if (flags2 & FUTEX2_NUMA) flags |= FLAGS_NUMA; + if (flags2 & FUTEX2_SPIN) + flags |= FLAGS_SPIN; + return flags; } diff --git a/kernel/futex/waitwake.c b/kernel/futex/waitwake.c index 3a10375d9521..94feac92cf4f 100644 --- a/kernel/futex/waitwake.c +++ b/kernel/futex/waitwake.c @@ -372,6 +372,78 @@ void futex_wait_queue(struct futex_hash_bucket *hb, struct futex_q *q, __set_current_state(TASK_RUNNING); } +static inline bool task_on_cpu(struct task_struct *p) +{ +#ifdef CONFIG_SMP + return !!(p->on_cpu); +#else + return false; +#endif +} + +static int futex_spin(struct futex_hash_bucket *hb, struct futex_q *q, + struct hrtimer_sleeper *timeout, void __user *uaddr, u32 val) +{ + struct task_struct *p; + u32 pid, uval; + unsigned int i = 0; + + if (futex_get_value_locked(&uval, uaddr)) + return -EFAULT; + + pid = uval; + + p = find_get_task_by_vpid(pid); + if (!p) { + printk("%s: no task found with PID %d\n", __func__, pid); + return -EAGAIN; + } + + if (unlikely(p->flags & PF_KTHREAD)) { + put_task_struct(p); + printk("%s: can't spin in a kernel task\n", __func__); + return -EPERM; + } + + futex_queue(q, hb); + + if (timeout) + hrtimer_sleeper_start_expires(timeout, HRTIMER_MODE_ABS); + + while (1) { + if (likely(!plist_node_empty(&q->list))) { + if (timeout && !timeout->task) + return 0; + + /* spin */ + if (task_on_cpu(p)) { + i++; + continue; + /* task is not running, sleep */ + } else { + break; + } + } else { + printk("%s: woke after %d spins\n", __func__, i); + return 0; + } + } + + printk("%s: spinned %d times, sleeping\n", __func__, i); + + /* spinning didn't work, go to the normal path */ + set_current_state(TASK_INTERRUPTIBLE|TASK_FREEZABLE); + + if (likely(!plist_node_empty(&q->list))) { + if (!timeout || timeout->task) + schedule(); + } + + __set_current_state(TASK_RUNNING); + + return 0; +} + /** * futex_unqueue_multiple - Remove various futexes from their hash bucket * @v: The list of futexes to unqueue @@ -665,8 +737,11 @@ int __futex_wait(u32 __user *uaddr, unsigned int flags, u32 val, if (ret) return ret; - /* futex_queue and wait for wakeup, timeout, or a signal. */ - futex_wait_queue(hb, &q, to); + if (flags & FLAGS_SPIN) + futex_spin(hb, &q, to, uaddr, val); + else + /* futex_queue and wait for wakeup, timeout, or a signal. */ + futex_wait_queue(hb, &q, to); /* If we were woken (and unqueued), we succeeded, whatever. */ if (!futex_unqueue(&q))