summaryrefslogtreecommitdiff
path: root/fs/bcachefs/six.c
AgeCommit message (Collapse)AuthorFilesLines
2024-01-01bcachefs: six locks: Simplify optimistic spinningKent Overstreet1-85/+32
osq lock maintainers don't want it to be used outside of kernel/locking/ - but, we can do better. Since we have lock handoff signalled via waitlist entries, there's no reason for optimistic spinning to have to look at the lock at all - aside from checking lock-owner; we can just spin looking at our waitlist entry. Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
2023-11-15bcachefs: six locks: Fix lost wakeupKent Overstreet1-2/+5
In percpu reader mode, trylock() for read had a lost wakeup: on failure to get the lock, we may have caused a writer to fail to get the lock, because we temporarily elevated the reader count. We need to check for waiters after decrementing the read count - not before. Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
2023-10-31six locks: Lock contended tracepointsKent Overstreet1-2/+6
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
2023-10-23bcachefs: Fix W=12 build errorsKent Overstreet1-1/+0
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
2023-10-23bcachefs: six locks: Guard against wakee exiting in __six_lock_wakeup()Kent Overstreet1-1/+8
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
2023-10-23bcachefs: six locks: Fix missing barrier on wait->lock_acquiredKent Overstreet1-8/+25
Six locks do lock handoff via the wakeup path: the thread doing the wakeup also takes the lock on behalf of the waiter, which means the waiter only has to look at its waitlist entry, and doesn't have to touch the lock cacheline while another thread is using it. Linus noticed that this needs a real barrier, which this patch fixes. Also add a comment for the should_sleep_fn() error path. Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: linux-bcachefs@vger.kernel.org Cc: linux-kernel@vger.kernel.org
2023-10-23six locks: Disable percpu read lock mode in userspaceKent Overstreet1-0/+6
When running in userspace, we currently don't have a real percpu implementation available - at least in bcachefs-tools, which is where this code is currently used in userspace. Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
2023-10-23six locks: Use atomic_try_cmpxchg_acquire()Kent Overstreet1-11/+6
This switches to a newer cmpxchg variant which updates @old for us on failure, simplifying the cmpxchg loops a bit and supposedly generating better code. Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
2023-10-23six locks: Fix an unitialized varKent Overstreet1-2/+1
In the conversion to atomic_t, six_lock_slowpath() ended up calling six_lock_wakeup() in the failure path with a state variable that was never initialized - whoops. Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
2023-10-23six locks: Delete redundant commentKent Overstreet1-11/+0
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
2023-10-23six locks: Tiny bit more tidyingKent Overstreet1-34/+30
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
2023-10-23six locks: Seq now only incremented on unlockKent Overstreet1-5/+0
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
2023-10-23six locks: Split out seq, use atomic_t instead of atomic64_tKent Overstreet1-69/+55
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
2023-10-23six locks: Single instance of six_lock_valsKent Overstreet1-25/+20
Since we're not generating different versions of the lock functions for each lock type, the constant propagation we were trying to do before is no longer useful - this is now a small code size decrease. Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
2023-10-23six_locks: Kill test_bit()/set_bit() usageKent Overstreet1-59/+4
This deletes the crazy cast-atomic-to-unsigned-long, and replaces them with atomic_and() and atomic_or(). Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
2023-10-23six locks: lock->state.seq no longer used for write lock heldKent Overstreet1-43/+30
lock->state.seq is shortly being moved out of lock->state, to kill the depedency on atomic64; in preparation for that, we change the write locking bit to write locked. Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
2023-10-23six locks: Simplify six_relock()Kent Overstreet1-42/+5
The next patch is going to move lock->seq out of lock->state. This replaces six_relock() with a much simpler implementation based on trylock. Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
2023-10-23six locks: Improve spurious wakeup handling in pcpu reader modeKent Overstreet1-14/+27
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
2023-10-23six locks: Documentation, renamingKent Overstreet1-42/+176
- Expanded and revamped overview documentation in six.h, giving an overview of all features - docbook-comments for all external interfaces - Rename some functions for simplicity, i.e. six_lock_ip_type() -> six_lock_ip() Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
2023-10-23six locks: Kill six_lock_state unionKent Overstreet1-116/+196
As suggested by Linus, this drops the six_lock_state union in favor of raw bitmasks. On the one hand, bitfields give more type-level structure to the code. However, a significant amount of the code was working with six_lock_state as a u64/atomic64_t, and the conversions from the bitfields to the u64 were deemed a bit too out-there. More significantly, because bitfield order is poorly defined (#ifdef __LITTLE_ENDIAN_BITFIELD can be used, but is gross), incrementing the sequence number would overflow into the rest of the bitfield if the compiler didn't put the sequence number at the high end of the word. The new code is a bit saner when we're on an architecture without real atomic64_t support - all accesses to lock->state now go through atomic64_*() operations. On architectures with real atomic64_t support, we additionally use atomic bit ops for setting/clearing individual bits. Text size: 7467 bytes -> 4649 bytes - compilers still suck at bitfields. Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
2023-10-23six locks: Simplify dispatchKent Overstreet1-69/+17
Originally, we used inlining/flattening to cause the compiler to generate different versions of lock/trylock/relock/unlock for each lock type - read, intent, and write. This made the individual functions smaller and let the compiler eliminate table lookups: however, as the code has gotten more complicated these optimizations have gotten less worthwhile, and all the tricky inlining and dispatching made the code less readable. Text size: 11015 bytes -> 7467 bytes, and benchmarks show no loss of performance. Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
2023-10-23six locks: Centralize setting of waiting bitKent Overstreet1-12/+3
Originally, the waiting bit was always set by trylock() on failure: however, it's now set by __six_lock_type_slowpath(), with wait_lock held - which is the more correct place to do it. That made setting the waiting bit in trylock redundant, so this patch deletes that. Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
2023-10-23six locks: Remove hacks for percpu mode lost wakeupKent Overstreet1-10/+1
The lost wakeup bug hasn't been observed in awhile, and we're trying to provoke it and determine if it still exists. This patch removes some defenses that were added to attempt to track it down; if it still exists, this should make it easier to see it. Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
2023-10-23six locks: Kill six_lock_pcpu_(alloc|free)Kent Overstreet1-19/+34
six_lock_pcpu_alloc() is an unsafe interface: it's not safe to allocate or free the percpu reader count on an existing lock that's in use, the only safe time to allocate percpu readers is when the lock is first being initialized. This patch adds a flags parameter to six_lock_init(), and instead of six_lock_pcpu_free() we now expose six_lock_exit(), which does the same thing but is less likely to be misused. Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
2023-10-23six locks: six_lock_readers_add()Kent Overstreet1-0/+13
This moves a helper out of the bcachefs code that shouldn't have been there, since it touches six lock internals. Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
2023-10-23six locks: be more careful about lost wakeupsKent Overstreet1-3/+11
This is a workaround for a lost wakeup bug we've been seeing - we still need to discover the actual bug. Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
2023-10-23six locks: Simplify six_lock_counts()Kent Overstreet1-10/+3
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
2023-10-23six locks: Improved optimistic spinningKent Overstreet1-15/+37
This adds a threshold for the maximum spin time, similar to the rwsem code, and a flag to the lock itself indicating when we've spun too long so other threads also refrain from spinning. Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
2023-10-23six locks: Expose tracepoint IPKent Overstreet1-36/+43
This adds _ip variations of the various lock functions that allow an IP to be passed in, which is used by lockstat. Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
2023-10-23six locks: Wakeup now takes lock on behalf of waiterKent Overstreet1-105/+158
This brings back an important optimization, to avoid touching the wait lists an extra time, while preserving the property that a thread is on a lock waitlist iff it is waiting - it is never removed from the waitlist until it has the lock. Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
2023-10-23six locks: Fix a lost wakeupKent Overstreet1-3/+8
There was a lost wakeup between a read unlock in percpu mode and a write lock. The unlock path unlocks, then executes a barrier, then checks for waiters; correspondingly, the lock side should set the wait bit and execute a barrier, then attempt to take the lock. Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
2023-10-23six locks: Enable lockdepKent Overstreet1-6/+6
Now that we have lockdep_set_no_check_recursion(), we can enable lockdep checking. Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
2023-10-23six locks: Add start_time to six_lock_waiterKent Overstreet1-0/+14
This is needed by the cycle detector in bcachefs - we need a way to iterater over waitlist entries while dropping and retaking the waitlist lock. Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
2023-10-23six locks: six_lock_waiter()Kent Overstreet1-9/+27
This allows passing in the wait list entry - to be used for a deadlock cycle detector. Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
2023-10-23six locks: Simplify wait listsKent Overstreet1-67/+40
This switches to a single list of waiters, instead of separate lists for read and intent, and switches write locks to also use the wait lists instead of being handled differently. Also, removal from the wait list is now done by the process waiting on the lock, not the process doing the wakeup. This is needed for the new deadlock cycle detector - we need tasks to stay on the waitlist until they've successfully acquired the lock. Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
2023-10-23six locks: Delete six_lock_pcpu_free_rcu()Kent Overstreet1-28/+0
Didn't have any users, and wasn't a good idea to begin with - delete it. Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
2023-10-23six locks: Improve six_lock_countKent Overstreet1-3/+7
six_lock_count now counts up whether a write lock held, and this patch now also correctly counts six_lock->intent_lock_recurse. Signed-off-by: Kent Overstreet <kent.overstreet@gmail.com>
2023-10-23bcachefs: Initial commitKent Overstreet1-0/+780
Initially forked from drivers/md/bcache, bcachefs is a new copy-on-write filesystem with every feature you could possibly want. Website: https://bcachefs.org Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
generated by cgit v1.2.3 (git 2.39.0) at 2024-10-07 12:18:55 +0300