summaryrefslogtreecommitdiff
path: root/include/linux/compiler.h
diff options
context:
space:
mode:
authorWill Deacon <will@kernel.org>2019-12-19 17:22:31 +0300
committerWill Deacon <will@kernel.org>2020-04-16 14:28:34 +0300
commitdee081bf8f824cabeb7c7495367d5dad0a444eb1 (patch)
treeeb19a451dd6f475428f2775d4d06c491b4ee4b01 /include/linux/compiler.h
parent9e343b467c70379e66b8b771d96f03ae23eba351 (diff)
downloadlinux-dee081bf8f824cabeb7c7495367d5dad0a444eb1.tar.xz
READ_ONCE: Drop pointer qualifiers when reading from scalar types
Passing a volatile-qualified pointer to READ_ONCE() is an absolute trainwreck for code generation: the use of 'typeof()' to define a temporary variable inside the macro means that the final evaluation in macro scope ends up forcing a read back from the stack. When stack protector is enabled (the default for arm64, at least), this causes the compiler to vomit up all sorts of junk. Unfortunately, dropping pointer qualifiers inside the macro poses quite a challenge, especially since the pointed-to type is permitted to be an aggregate, and this is relied upon by mm/ code accessing things like 'pmd_t'. Based on numerous hacks and discussions on the mailing list, this is the best I've managed to come up with. Introduce '__unqual_scalar_typeof()' which takes an expression and, if the expression is an optionally qualified 8, 16, 32 or 64-bit scalar type, evaluates to the unqualified type. Other input types, including aggregates, remain unchanged. Hopefully READ_ONCE() on volatile aggregate pointers isn't something we do on a fast-path. Cc: Peter Zijlstra <peterz@infradead.org> Cc: Arnd Bergmann <arnd@arndb.de> Suggested-by: Linus Torvalds <torvalds@linux-foundation.org> Reported-by: Michael Ellerman <mpe@ellerman.id.au> Signed-off-by: Will Deacon <will@kernel.org>
Diffstat (limited to 'include/linux/compiler.h')
-rw-r--r--include/linux/compiler.h6
1 files changed, 3 insertions, 3 deletions
diff --git a/include/linux/compiler.h b/include/linux/compiler.h
index 50bb2461648f..c363d8debc43 100644
--- a/include/linux/compiler.h
+++ b/include/linux/compiler.h
@@ -203,13 +203,13 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,
* atomicity or dependency ordering guarantees. Note that this may result
* in tears!
*/
-#define __READ_ONCE(x) (*(const volatile typeof(x) *)&(x))
+#define __READ_ONCE(x) (*(const volatile __unqual_scalar_typeof(x) *)&(x))
#define __READ_ONCE_SCALAR(x) \
({ \
- typeof(x) __x = __READ_ONCE(x); \
+ __unqual_scalar_typeof(x) __x = __READ_ONCE(x); \
smp_read_barrier_depends(); \
- __x; \
+ (typeof(x))__x; \
})
#define READ_ONCE(x) \