diff options
author | Frederic Weisbecker <frederic@kernel.org> | 2023-03-29 19:02:02 +0300 |
---|---|---|
committer | Paul E. McKenney <paulmck@kernel.org> | 2023-05-10 03:26:59 +0300 |
commit | b96a8b0b5be40f9bc9e45819f14b32ea9cdce73f (patch) | |
tree | c9025e181cc3904376f20d24fdbfdacb7e700bd7 /kernel/rcu/tree_nocb.h | |
parent | 7625926086765123251f765d91fc3a70617d334d (diff) | |
download | linux-b96a8b0b5be40f9bc9e45819f14b32ea9cdce73f.tar.xz |
rcu/nocb: Recheck lazy callbacks under the ->nocb_lock from shrinker
The ->lazy_len is only checked locklessly. Recheck again under the
->nocb_lock to avoid spending more time on flushing/waking if not
necessary. The ->lazy_len can still increment concurrently (from 1 to
infinity) but under the ->nocb_lock we at least know for sure if there
are lazy callbacks at all (->lazy_len > 0).
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Diffstat (limited to 'kernel/rcu/tree_nocb.h')
-rw-r--r-- | kernel/rcu/tree_nocb.h | 14 |
1 files changed, 11 insertions, 3 deletions
diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index c321fce2af8e..dfa9c10d6727 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1358,12 +1358,20 @@ lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) if (!rcu_rdp_is_offloaded(rdp)) continue; - _count = READ_ONCE(rdp->lazy_len); - - if (_count == 0) + if (!READ_ONCE(rdp->lazy_len)) continue; rcu_nocb_lock_irqsave(rdp, flags); + /* + * Recheck under the nocb lock. Since we are not holding the bypass + * lock we may still race with increments from the enqueuer but still + * we know for sure if there is at least one lazy callback. + */ + _count = READ_ONCE(rdp->lazy_len); + if (!_count) { + rcu_nocb_unlock_irqrestore(rdp, flags); + continue; + } WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies, false)); rcu_nocb_unlock_irqrestore(rdp, flags); wake_nocb_gp(rdp, false); |