diff options
| author | Uros Bizjak <ubizjak@gmail.com> | 2024-01-24 13:58:16 +0300 |
|---|---|---|
| committer | Ingo Molnar <mingo@kernel.org> | 2024-03-01 14:54:25 +0300 |
| commit | e807c2a37044a51de89d6d4f8a1f5ecfb3752f36 (patch) | |
| tree | 6bbce6951a8ff33b095fea22a79f6f9dc762b555 | |
| parent | f3e3620f1a97fcd02a5f3606fa63888dbcffd82c (diff) | |
| download | linux-e807c2a37044a51de89d6d4f8a1f5ecfb3752f36.tar.xz | |
locking/x86: Implement local_xchg() using CMPXCHG without the LOCK prefix
Implement local_xchg() using the CMPXCHG instruction without the LOCK prefix.
XCHG is expensive due to the implied LOCK prefix. The processor
cannot prefetch cachelines if XCHG is used.
Signed-off-by: Uros Bizjak <ubizjak@gmail.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Waiman Long <longman@redhat.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Paul E. McKenney <paulmck@kernel.org>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Link: https://lore.kernel.org/r/20240124105816.612670-1-ubizjak@gmail.com
| -rw-r--r-- | arch/x86/include/asm/local.h | 16 |
1 files changed, 14 insertions, 2 deletions
diff --git a/arch/x86/include/asm/local.h b/arch/x86/include/asm/local.h index 73dba8b94443..59aa966dc212 100644 --- a/arch/x86/include/asm/local.h +++ b/arch/x86/include/asm/local.h @@ -131,8 +131,20 @@ static inline bool local_try_cmpxchg(local_t *l, long *old, long new) (typeof(l->a.counter) *) old, new); } -/* Always has a lock prefix */ -#define local_xchg(l, n) (xchg(&((l)->a.counter), (n))) +/* + * Implement local_xchg using CMPXCHG instruction without the LOCK prefix. + * XCHG is expensive due to the implied LOCK prefix. The processor + * cannot prefetch cachelines if XCHG is used. + */ +static __always_inline long +local_xchg(local_t *l, long n) +{ + long c = local_read(l); + + do { } while (!local_try_cmpxchg(l, &c, n)); + + return c; +} /** * local_add_unless - add unless the number is already a given value |