649 lines
32 KiB
ReStructuredText
649 lines
32 KiB
ReStructuredText
|
======================================================
|
|||
|
|
A Tour Through TREE_RCU's Grace-Period Memory Ordering
|
|||
|
|
======================================================
|
|||
|
|
|
|||
|
|
August 8, 2017
|
|||
|
|
|
|||
|
|
This article was contributed by Paul E. McKenney
|
|||
|
|
|
|||
|
|
Introduction
|
|||
|
|
============
|
|||
|
|
|
|||
|
|
This document gives a rough visual overview of how Tree RCU's
|
|||
|
|
grace-period memory ordering guarantee is provided.
|
|||
|
|
|
|||
|
|
What Is Tree RCU's Grace Period Memory Ordering Guarantee?
|
|||
|
|
==========================================================
|
|||
|
|
|
|||
|
|
RCU grace periods provide extremely strong memory-ordering guarantees
|
|||
|
|
for non-idle non-offline code.
|
|||
|
|
Any code that happens after the end of a given RCU grace period is guaranteed
|
|||
|
|
to see the effects of all accesses prior to the beginning of that grace
|
|||
|
|
period that are within RCU read-side critical sections.
|
|||
|
|
Similarly, any code that happens before the beginning of a given RCU grace
|
|||
|
|
period is guaranteed to not see the effects of all accesses following the end
|
|||
|
|
of that grace period that are within RCU read-side critical sections.
|
|||
|
|
|
|||
|
|
Note well that RCU-sched read-side critical sections include any region
|
|||
|
|
of code for which preemption is disabled.
|
|||
|
|
Given that each individual machine instruction can be thought of as
|
|||
|
|
an extremely small region of preemption-disabled code, one can think of
|
|||
|
|
``synchronize_rcu()`` as ``smp_mb()`` on steroids.
|
|||
|
|
|
|||
|
|
RCU updaters use this guarantee by splitting their updates into
|
|||
|
|
two phases, one of which is executed before the grace period and
|
|||
|
|
the other of which is executed after the grace period.
|
|||
|
|
In the most common use case, phase one removes an element from
|
|||
|
|
a linked RCU-protected data structure, and phase two frees that element.
|
|||
|
|
For this to work, any readers that have witnessed state prior to the
|
|||
|
|
phase-one update (in the common case, removal) must not witness state
|
|||
|
|
following the phase-two update (in the common case, freeing).
|
|||
|
|
|
|||
|
|
The RCU implementation provides this guarantee using a network
|
|||
|
|
of lock-based critical sections, memory barriers, and per-CPU
|
|||
|
|
processing, as is described in the following sections.
|
|||
|
|
|
|||
|
|
Tree RCU Grace Period Memory Ordering Building Blocks
|
|||
|
|
=====================================================
|
|||
|
|
|
|||
|
|
The workhorse for RCU's grace-period memory ordering is the
|
|||
|
|
critical section for the ``rcu_node`` structure's
|
|||
|
|
``->lock``. These critical sections use helper functions for lock
|
|||
|
|
acquisition, including ``raw_spin_lock_rcu_node()``,
|
|||
|
|
``raw_spin_lock_irq_rcu_node()``, and ``raw_spin_lock_irqsave_rcu_node()``.
|
|||
|
|
Their lock-release counterparts are ``raw_spin_unlock_rcu_node()``,
|
|||
|
|
``raw_spin_unlock_irq_rcu_node()``, and
|
|||
|
|
``raw_spin_unlock_irqrestore_rcu_node()``, respectively.
|
|||
|
|
For completeness, a ``raw_spin_trylock_rcu_node()`` is also provided.
|
|||
|
|
The key point is that the lock-acquisition functions, including
|
|||
|
|
``raw_spin_trylock_rcu_node()``, all invoke ``smp_mb__after_unlock_lock()``
|
|||
|
|
immediately after successful acquisition of the lock.
|
|||
|
|
|
|||
|
|
Therefore, for any given ``rcu_node`` structure, any access
|
|||
|
|
happening before one of the above lock-release functions will be seen
|
|||
|
|
by all CPUs as happening before any access happening after a later
|
|||
|
|
one of the above lock-acquisition functions.
|
|||
|
|
Furthermore, any access happening before one of the
|
|||
|
|
above lock-release function on any given CPU will be seen by all
|
|||
|
|
CPUs as happening before any access happening after a later one
|
|||
|
|
of the above lock-acquisition functions executing on that same CPU,
|
|||
|
|
even if the lock-release and lock-acquisition functions are operating
|
|||
|
|
on different ``rcu_node`` structures.
|
|||
|
|
Tree RCU uses these two ordering guarantees to form an ordering
|
|||
|
|
network among all CPUs that were in any way involved in the grace
|
|||
|
|
period, including any CPUs that came online or went offline during
|
|||
|
|
the grace period in question.
|
|||
|
|
|
|||
|
|
The following litmus test exhibits the ordering effects of these
|
|||
|
|
lock-acquisition and lock-release functions::
|
|||
|
|
|
|||
|
|
1 int x, y, z;
|
|||
|
|
2
|
|||
|
|
3 void task0(void)
|
|||
|
|
4 {
|
|||
|
|
5 raw_spin_lock_rcu_node(rnp);
|
|||
|
|
6 WRITE_ONCE(x, 1);
|
|||
|
|
7 r1 = READ_ONCE(y);
|
|||
|
|
8 raw_spin_unlock_rcu_node(rnp);
|
|||
|
|
9 }
|
|||
|
|
10
|
|||
|
|
11 void task1(void)
|
|||
|
|
12 {
|
|||
|
|
13 raw_spin_lock_rcu_node(rnp);
|
|||
|
|
14 WRITE_ONCE(y, 1);
|
|||
|
|
15 r2 = READ_ONCE(z);
|
|||
|
|
16 raw_spin_unlock_rcu_node(rnp);
|
|||
|
|
17 }
|
|||
|
|
18
|
|||
|
|
19 void task2(void)
|
|||
|
|
20 {
|
|||
|
|
21 WRITE_ONCE(z, 1);
|
|||
|
|
22 smp_mb();
|
|||
|
|
23 r3 = READ_ONCE(x);
|
|||
|
|
24 }
|
|||
|
|
25
|
|||
|
|
26 WARN_ON(r1 == 0 && r2 == 0 && r3 == 0);
|
|||
|
|
|
|||
|
|
The ``WARN_ON()`` is evaluated at "the end of time",
|
|||
|
|
after all changes have propagated throughout the system.
|
|||
|
|
Without the ``smp_mb__after_unlock_lock()`` provided by the
|
|||
|
|
acquisition functions, this ``WARN_ON()`` could trigger, for example
|
|||
|
|
on PowerPC.
|
|||
|
|
The ``smp_mb__after_unlock_lock()`` invocations prevent this
|
|||
|
|
``WARN_ON()`` from triggering.
|
|||
|
|
|
|||
|
|
+-----------------------------------------------------------------------+
|
|||
|
|
| **Quick Quiz**: |
|
|||
|
|
+-----------------------------------------------------------------------+
|
|||
|
|
| But the chain of rcu_node-structure lock acquisitions guarantees |
|
|||
|
|
| that new readers will see all of the updater's pre-grace-period |
|
|||
|
|
| accesses and also guarantees that the updater's post-grace-period |
|
|||
|
|
| accesses will see all of the old reader's accesses. So why do we |
|
|||
|
|
| need all of those calls to smp_mb__after_unlock_lock()? |
|
|||
|
|
+-----------------------------------------------------------------------+
|
|||
|
|
| **Answer**: |
|
|||
|
|
+-----------------------------------------------------------------------+
|
|||
|
|
| Because we must provide ordering for RCU's polling grace-period |
|
|||
|
|
| primitives, for example, get_state_synchronize_rcu() and |
|
|||
|
|
| poll_state_synchronize_rcu(). Consider this code:: |
|
|||
|
|
| |
|
|||
|
|
| CPU 0 CPU 1 |
|
|||
|
|
| ---- ---- |
|
|||
|
|
| WRITE_ONCE(X, 1) WRITE_ONCE(Y, 1) |
|
|||
|
|
| g = get_state_synchronize_rcu() smp_mb() |
|
|||
|
|
| while (!poll_state_synchronize_rcu(g)) r1 = READ_ONCE(X) |
|
|||
|
|
| continue; |
|
|||
|
|
| r0 = READ_ONCE(Y) |
|
|||
|
|
| |
|
|||
|
|
| RCU guarantees that the outcome r0 == 0 && r1 == 0 will not |
|
|||
|
|
| happen, even if CPU 1 is in an RCU extended quiescent state |
|
|||
|
|
| (idle or offline) and thus won't interact directly with the RCU |
|
|||
|
|
| core processing at all. |
|
|||
|
|
+-----------------------------------------------------------------------+
|
|||
|
|
|
|||
|
|
This approach must be extended to include idle CPUs, which need
|
|||
|
|
RCU's grace-period memory ordering guarantee to extend to any
|
|||
|
|
RCU read-side critical sections preceding and following the current
|
|||
|
|
idle sojourn.
|
|||
|
|
This case is handled by calls to the strongly ordered
|
|||
|
|
``atomic_add_return()`` read-modify-write atomic operation that
|
|||
|
|
is invoked within ``rcu_dynticks_eqs_enter()`` at idle-entry
|
|||
|
|
time and within ``rcu_dynticks_eqs_exit()`` at idle-exit time.
|
|||
|
|
The grace-period kthread invokes ``rcu_dynticks_snap()`` and
|
|||
|
|
``rcu_dynticks_in_eqs_since()`` (both of which invoke
|
|||
|
|
an ``atomic_add_return()`` of zero) to detect idle CPUs.
|
|||
|
|
|
|||
|
|
+-----------------------------------------------------------------------+
|
|||
|
|
| **Quick Quiz**: |
|
|||
|
|
+-----------------------------------------------------------------------+
|
|||
|
|
| But what about CPUs that remain offline for the entire grace period? |
|
|||
|
|
+-----------------------------------------------------------------------+
|
|||
|
|
| **Answer**: |
|
|||
|
|
+-----------------------------------------------------------------------+
|
|||
|
|
| Such CPUs will be offline at the beginning of the grace period, so |
|
|||
|
|
| the grace period won't expect quiescent states from them. Races |
|
|||
|
|
| between grace-period start and CPU-hotplug operations are mediated |
|
|||
|
|
| by the CPU's leaf ``rcu_node`` structure's ``->lock`` as described |
|
|||
|
|
| above. |
|
|||
|
|
+-----------------------------------------------------------------------+
|
|||
|
|
|
|||
|
|
The approach must be extended to handle one final case, that of waking a
|
|||
|
|
task blocked in ``synchronize_rcu()``. This task might be affinitied to
|
|||
|
|
a CPU that is not yet aware that the grace period has ended, and thus
|
|||
|
|
might not yet be subject to the grace period's memory ordering.
|
|||
|
|
Therefore, there is an ``smp_mb()`` after the return from
|
|||
|
|
``wait_for_completion()`` in the ``synchronize_rcu()`` code path.
|
|||
|
|
|
|||
|
|
+-----------------------------------------------------------------------+
|
|||
|
|
| **Quick Quiz**: |
|
|||
|
|
+-----------------------------------------------------------------------+
|
|||
|
|
| What? Where??? I don't see any ``smp_mb()`` after the return from |
|
|||
|
|
| ``wait_for_completion()``!!! |
|
|||
|
|
+-----------------------------------------------------------------------+
|
|||
|
|
| **Answer**: |
|
|||
|
|
+-----------------------------------------------------------------------+
|
|||
|
|
| That would be because I spotted the need for that ``smp_mb()`` during |
|
|||
|
|
| the creation of this documentation, and it is therefore unlikely to |
|
|||
|
|
| hit mainline before v4.14. Kudos to Lance Roy, Will Deacon, Peter |
|
|||
|
|
| Zijlstra, and Jonathan Cameron for asking questions that sensitized |
|
|||
|
|
| me to the rather elaborate sequence of events that demonstrate the |
|
|||
|
|
| need for this memory barrier. |
|
|||
|
|
+-----------------------------------------------------------------------+
|
|||
|
|
|
|||
|
|
Tree RCU's grace--period memory-ordering guarantees rely most heavily on
|
|||
|
|
the ``rcu_node`` structure's ``->lock`` field, so much so that it is
|
|||
|
|
necessary to abbreviate this pattern in the diagrams in the next
|
|||
|
|
section. For example, consider the ``rcu_prepare_for_idle()`` function
|
|||
|
|
shown below, which is one of several functions that enforce ordering of
|
|||
|
|
newly arrived RCU callbacks against future grace periods:
|
|||
|
|
|
|||
|
|
::
|
|||
|
|
|
|||
|
|
1 static void rcu_prepare_for_idle(void)
|
|||
|
|
2 {
|
|||
|
|
3 bool needwake;
|
|||
|
|
4 struct rcu_data *rdp = this_cpu_ptr(&rcu_data);
|
|||
|
|
5 struct rcu_node *rnp;
|
|||
|
|
6 int tne;
|
|||
|
|
7
|
|||
|
|
8 lockdep_assert_irqs_disabled();
|
|||
|
|
9 if (rcu_rdp_is_offloaded(rdp))
|
|||
|
|
10 return;
|
|||
|
|
11
|
|||
|
|
12 /* Handle nohz enablement switches conservatively. */
|
|||
|
|
13 tne = READ_ONCE(tick_nohz_active);
|
|||
|
|
14 if (tne != rdp->tick_nohz_enabled_snap) {
|
|||
|
|
15 if (!rcu_segcblist_empty(&rdp->cblist))
|
|||
|
|
16 invoke_rcu_core(); /* force nohz to see update. */
|
|||
|
|
17 rdp->tick_nohz_enabled_snap = tne;
|
|||
|
|
18 return;
|
|||
|
|
19 }
|
|||
|
|
20 if (!tne)
|
|||
|
|
21 return;
|
|||
|
|
22
|
|||
|
|
23 /*
|
|||
|
|
24 * If we have not yet accelerated this jiffy, accelerate all
|
|||
|
|
25 * callbacks on this CPU.
|
|||
|
|
26 */
|
|||
|
|
27 if (rdp->last_accelerate == jiffies)
|
|||
|
|
28 return;
|
|||
|
|
29 rdp->last_accelerate = jiffies;
|
|||
|
|
30 if (rcu_segcblist_pend_cbs(&rdp->cblist)) {
|
|||
|
|
31 rnp = rdp->mynode;
|
|||
|
|
32 raw_spin_lock_rcu_node(rnp); /* irqs already disabled. */
|
|||
|
|
33 needwake = rcu_accelerate_cbs(rnp, rdp);
|
|||
|
|
34 raw_spin_unlock_rcu_node(rnp); /* irqs remain disabled. */
|
|||
|
|
35 if (needwake)
|
|||
|
|
36 rcu_gp_kthread_wake();
|
|||
|
|
37 }
|
|||
|
|
38 }
|
|||
|
|
|
|||
|
|
But the only part of ``rcu_prepare_for_idle()`` that really matters for
|
|||
|
|
this discussion are lines 32–34. We will therefore abbreviate this
|
|||
|
|
function as follows:
|
|||
|
|
|
|||
|
|
.. kernel-figure:: rcu_node-lock.svg
|
|||
|
|
|
|||
|
|
The box represents the ``rcu_node`` structure's ``->lock`` critical
|
|||
|
|
section, with the double line on top representing the additional
|
|||
|
|
``smp_mb__after_unlock_lock()``.
|
|||
|
|
|
|||
|
|
Tree RCU Grace Period Memory Ordering Components
|
|||
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|||
|
|
|
|||
|
|
Tree RCU's grace-period memory-ordering guarantee is provided by a
|
|||
|
|
number of RCU components:
|
|||
|
|
|
|||
|
|
#. `Callback Registry`_
|
|||
|
|
#. `Grace-Period Initialization`_
|
|||
|
|
#. `Self-Reported Quiescent States`_
|
|||
|
|
#. `Dynamic Tick Interface`_
|
|||
|
|
#. `CPU-Hotplug Interface`_
|
|||
|
|
#. `Forcing Quiescent States`_
|
|||
|
|
#. `Grace-Period Cleanup`_
|
|||
|
|
#. `Callback Invocation`_
|
|||
|
|
|
|||
|
|
Each of the following section looks at the corresponding component in
|
|||
|
|
detail.
|
|||
|
|
|
|||
|
|
Callback Registry
|
|||
|
|
^^^^^^^^^^^^^^^^^
|
|||
|
|
|
|||
|
|
If RCU's grace-period guarantee is to mean anything at all, any access
|
|||
|
|
that happens before a given invocation of ``call_rcu()`` must also
|
|||
|
|
happen before the corresponding grace period. The implementation of this
|
|||
|
|
portion of RCU's grace period guarantee is shown in the following
|
|||
|
|
figure:
|
|||
|
|
|
|||
|
|
.. kernel-figure:: TreeRCU-callback-registry.svg
|
|||
|
|
|
|||
|
|
Because ``call_rcu()`` normally acts only on CPU-local state, it
|
|||
|
|
provides no ordering guarantees, either for itself or for phase one of
|
|||
|
|
the update (which again will usually be removal of an element from an
|
|||
|
|
RCU-protected data structure). It simply enqueues the ``rcu_head``
|
|||
|
|
structure on a per-CPU list, which cannot become associated with a grace
|
|||
|
|
period until a later call to ``rcu_accelerate_cbs()``, as shown in the
|
|||
|
|
diagram above.
|
|||
|
|
|
|||
|
|
One set of code paths shown on the left invokes ``rcu_accelerate_cbs()``
|
|||
|
|
via ``note_gp_changes()``, either directly from ``call_rcu()`` (if the
|
|||
|
|
current CPU is inundated with queued ``rcu_head`` structures) or more
|
|||
|
|
likely from an ``RCU_SOFTIRQ`` handler. Another code path in the middle
|
|||
|
|
is taken only in kernels built with ``CONFIG_RCU_FAST_NO_HZ=y``, which
|
|||
|
|
invokes ``rcu_accelerate_cbs()`` via ``rcu_prepare_for_idle()``. The
|
|||
|
|
final code path on the right is taken only in kernels built with
|
|||
|
|
``CONFIG_HOTPLUG_CPU=y``, which invokes ``rcu_accelerate_cbs()`` via
|
|||
|
|
``rcu_advance_cbs()``, ``rcu_migrate_callbacks``,
|
|||
|
|
``rcutree_migrate_callbacks()``, and ``takedown_cpu()``, which in turn
|
|||
|
|
is invoked on a surviving CPU after the outgoing CPU has been completely
|
|||
|
|
offlined.
|
|||
|
|
|
|||
|
|
There are a few other code paths within grace-period processing that
|
|||
|
|
opportunistically invoke ``rcu_accelerate_cbs()``. However, either way,
|
|||
|
|
all of the CPU's recently queued ``rcu_head`` structures are associated
|
|||
|
|
with a future grace-period number under the protection of the CPU's lead
|
|||
|
|
``rcu_node`` structure's ``->lock``. In all cases, there is full
|
|||
|
|
ordering against any prior critical section for that same ``rcu_node``
|
|||
|
|
structure's ``->lock``, and also full ordering against any of the
|
|||
|
|
current task's or CPU's prior critical sections for any ``rcu_node``
|
|||
|
|
structure's ``->lock``.
|
|||
|
|
|
|||
|
|
The next section will show how this ordering ensures that any accesses
|
|||
|
|
prior to the ``call_rcu()`` (particularly including phase one of the
|
|||
|
|
update) happen before the start of the corresponding grace period.
|
|||
|
|
|
|||
|
|
+-----------------------------------------------------------------------+
|
|||
|
|
| **Quick Quiz**: |
|
|||
|
|
+-----------------------------------------------------------------------+
|
|||
|
|
| But what about ``synchronize_rcu()``? |
|
|||
|
|
+-----------------------------------------------------------------------+
|
|||
|
|
| **Answer**: |
|
|||
|
|
+-----------------------------------------------------------------------+
|
|||
|
|
| The ``synchronize_rcu()`` passes ``call_rcu()`` to ``wait_rcu_gp()``, |
|
|||
|
|
| which invokes it. So either way, it eventually comes down to |
|
|||
|
|
| ``call_rcu()``. |
|
|||
|
|
+-----------------------------------------------------------------------+
|
|||
|
|
|
|||
|
|
Grace-Period Initialization
|
|||
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|||
|
|
|
|||
|
|
Grace-period initialization is carried out by the grace-period kernel
|
|||
|
|
thread, which makes several passes over the ``rcu_node`` tree within the
|
|||
|
|
``rcu_gp_init()`` function. This means that showing the full flow of
|
|||
|
|
ordering through the grace-period computation will require duplicating
|
|||
|
|
this tree. If you find this confusing, please note that the state of the
|
|||
|
|
``rcu_node`` changes over time, just like Heraclitus's river. However,
|
|||
|
|
to keep the ``rcu_node`` river tractable, the grace-period kernel
|
|||
|
|
thread's traversals are presented in multiple parts, starting in this
|
|||
|
|
section with the various phases of grace-period initialization.
|
|||
|
|
|
|||
|
|
The first ordering-related grace-period initialization action is to
|
|||
|
|
advance the ``rcu_state`` structure's ``->gp_seq`` grace-period-number
|
|||
|
|
counter, as shown below:
|
|||
|
|
|
|||
|
|
.. kernel-figure:: TreeRCU-gp-init-1.svg
|
|||
|
|
|
|||
|
|
The actual increment is carried out using ``smp_store_release()``, which
|
|||
|
|
helps reject false-positive RCU CPU stall detection. Note that only the
|
|||
|
|
root ``rcu_node`` structure is touched.
|
|||
|
|
|
|||
|
|
The first pass through the ``rcu_node`` tree updates bitmasks based on
|
|||
|
|
CPUs having come online or gone offline since the start of the previous
|
|||
|
|
grace period. In the common case where the number of online CPUs for
|
|||
|
|
this ``rcu_node`` structure has not transitioned to or from zero, this
|
|||
|
|
pass will scan only the leaf ``rcu_node`` structures. However, if the
|
|||
|
|
number of online CPUs for a given leaf ``rcu_node`` structure has
|
|||
|
|
transitioned from zero, ``rcu_init_new_rnp()`` will be invoked for the
|
|||
|
|
first incoming CPU. Similarly, if the number of online CPUs for a given
|
|||
|
|
leaf ``rcu_node`` structure has transitioned to zero,
|
|||
|
|
``rcu_cleanup_dead_rnp()`` will be invoked for the last outgoing CPU.
|
|||
|
|
The diagram below shows the path of ordering if the leftmost
|
|||
|
|
``rcu_node`` structure onlines its first CPU and if the next
|
|||
|
|
``rcu_node`` structure has no online CPUs (or, alternatively if the
|
|||
|
|
leftmost ``rcu_node`` structure offlines its last CPU and if the next
|
|||
|
|
``rcu_node`` structure has no online CPUs).
|
|||
|
|
|
|||
|
|
.. kernel-figure:: TreeRCU-gp-init-2.svg
|
|||
|
|
|
|||
|
|
The final ``rcu_gp_init()`` pass through the ``rcu_node`` tree traverses
|
|||
|
|
breadth-first, setting each ``rcu_node`` structure's ``->gp_seq`` field
|
|||
|
|
to the newly advanced value from the ``rcu_state`` structure, as shown
|
|||
|
|
in the following diagram.
|
|||
|
|
|
|||
|
|
.. kernel-figure:: TreeRCU-gp-init-3.svg
|
|||
|
|
|
|||
|
|
This change will also cause each CPU's next call to
|
|||
|
|
``__note_gp_changes()`` to notice that a new grace period has started,
|
|||
|
|
as described in the next section. But because the grace-period kthread
|
|||
|
|
started the grace period at the root (with the advancing of the
|
|||
|
|
``rcu_state`` structure's ``->gp_seq`` field) before setting each leaf
|
|||
|
|
``rcu_node`` structure's ``->gp_seq`` field, each CPU's observation of
|
|||
|
|
the start of the grace period will happen after the actual start of the
|
|||
|
|
grace period.
|
|||
|
|
|
|||
|
|
+-----------------------------------------------------------------------+
|
|||
|
|
| **Quick Quiz**: |
|
|||
|
|
+-----------------------------------------------------------------------+
|
|||
|
|
| But what about the CPU that started the grace period? Why wouldn't it |
|
|||
|
|
| see the start of the grace period right when it started that grace |
|
|||
|
|
| period? |
|
|||
|
|
+-----------------------------------------------------------------------+
|
|||
|
|
| **Answer**: |
|
|||
|
|
+-----------------------------------------------------------------------+
|
|||
|
|
| In some deep philosophical and overly anthromorphized sense, yes, the |
|
|||
|
|
| CPU starting the grace period is immediately aware of having done so. |
|
|||
|
|
| However, if we instead assume that RCU is not self-aware, then even |
|
|||
|
|
| the CPU starting the grace period does not really become aware of the |
|
|||
|
|
| start of this grace period until its first call to |
|
|||
|
|
| ``__note_gp_changes()``. On the other hand, this CPU potentially gets |
|
|||
|
|
| early notification because it invokes ``__note_gp_changes()`` during |
|
|||
|
|
| its last ``rcu_gp_init()`` pass through its leaf ``rcu_node`` |
|
|||
|
|
| structure. |
|
|||
|
|
+-----------------------------------------------------------------------+
|
|||
|
|
|
|||
|
|
Self-Reported Quiescent States
|
|||
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|||
|
|
|
|||
|
|
When all entities that might block the grace period have reported
|
|||
|
|
quiescent states (or as described in a later section, had quiescent
|
|||
|
|
states reported on their behalf), the grace period can end. Online
|
|||
|
|
non-idle CPUs report their own quiescent states, as shown in the
|
|||
|
|
following diagram:
|
|||
|
|
|
|||
|
|
.. kernel-figure:: TreeRCU-qs.svg
|
|||
|
|
|
|||
|
|
This is for the last CPU to report a quiescent state, which signals the
|
|||
|
|
end of the grace period. Earlier quiescent states would push up the
|
|||
|
|
``rcu_node`` tree only until they encountered an ``rcu_node`` structure
|
|||
|
|
that is waiting for additional quiescent states. However, ordering is
|
|||
|
|
nevertheless preserved because some later quiescent state will acquire
|
|||
|
|
that ``rcu_node`` structure's ``->lock``.
|
|||
|
|
|
|||
|
|
Any number of events can lead up to a CPU invoking ``note_gp_changes``
|
|||
|
|
(or alternatively, directly invoking ``__note_gp_changes()``), at which
|
|||
|
|
point that CPU will notice the start of a new grace period while holding
|
|||
|
|
its leaf ``rcu_node`` lock. Therefore, all execution shown in this
|
|||
|
|
diagram happens after the start of the grace period. In addition, this
|
|||
|
|
CPU will consider any RCU read-side critical section that started before
|
|||
|
|
the invocation of ``__note_gp_changes()`` to have started before the
|
|||
|
|
grace period, and thus a critical section that the grace period must
|
|||
|
|
wait on.
|
|||
|
|
|
|||
|
|
+-----------------------------------------------------------------------+
|
|||
|
|
| **Quick Quiz**: |
|
|||
|
|
+-----------------------------------------------------------------------+
|
|||
|
|
| But a RCU read-side critical section might have started after the |
|
|||
|
|
| beginning of the grace period (the advancing of ``->gp_seq`` from |
|
|||
|
|
| earlier), so why should the grace period wait on such a critical |
|
|||
|
|
| section? |
|
|||
|
|
+-----------------------------------------------------------------------+
|
|||
|
|
| **Answer**: |
|
|||
|
|
+-----------------------------------------------------------------------+
|
|||
|
|
| It is indeed not necessary for the grace period to wait on such a |
|
|||
|
|
| critical section. However, it is permissible to wait on it. And it is |
|
|||
|
|
| furthermore important to wait on it, as this lazy approach is far |
|
|||
|
|
| more scalable than a “big bang” all-at-once grace-period start could |
|
|||
|
|
| possibly be. |
|
|||
|
|
+-----------------------------------------------------------------------+
|
|||
|
|
|
|||
|
|
If the CPU does a context switch, a quiescent state will be noted by
|
|||
|
|
``rcu_note_context_switch()`` on the left. On the other hand, if the CPU
|
|||
|
|
takes a scheduler-clock interrupt while executing in usermode, a
|
|||
|
|
quiescent state will be noted by ``rcu_sched_clock_irq()`` on the right.
|
|||
|
|
Either way, the passage through a quiescent state will be noted in a
|
|||
|
|
per-CPU variable.
|
|||
|
|
|
|||
|
|
The next time an ``RCU_SOFTIRQ`` handler executes on this CPU (for
|
|||
|
|
example, after the next scheduler-clock interrupt), ``rcu_core()`` will
|
|||
|
|
invoke ``rcu_check_quiescent_state()``, which will notice the recorded
|
|||
|
|
quiescent state, and invoke ``rcu_report_qs_rdp()``. If
|
|||
|
|
``rcu_report_qs_rdp()`` verifies that the quiescent state really does
|
|||
|
|
apply to the current grace period, it invokes ``rcu_report_rnp()`` which
|
|||
|
|
traverses up the ``rcu_node`` tree as shown at the bottom of the
|
|||
|
|
diagram, clearing bits from each ``rcu_node`` structure's ``->qsmask``
|
|||
|
|
field, and propagating up the tree when the result is zero.
|
|||
|
|
|
|||
|
|
Note that traversal passes upwards out of a given ``rcu_node`` structure
|
|||
|
|
only if the current CPU is reporting the last quiescent state for the
|
|||
|
|
subtree headed by that ``rcu_node`` structure. A key point is that if a
|
|||
|
|
CPU's traversal stops at a given ``rcu_node`` structure, then there will
|
|||
|
|
be a later traversal by another CPU (or perhaps the same one) that
|
|||
|
|
proceeds upwards from that point, and the ``rcu_node`` ``->lock``
|
|||
|
|
guarantees that the first CPU's quiescent state happens before the
|
|||
|
|
remainder of the second CPU's traversal. Applying this line of thought
|
|||
|
|
repeatedly shows that all CPUs' quiescent states happen before the last
|
|||
|
|
CPU traverses through the root ``rcu_node`` structure, the “last CPU”
|
|||
|
|
being the one that clears the last bit in the root ``rcu_node``
|
|||
|
|
structure's ``->qsmask`` field.
|
|||
|
|
|
|||
|
|
Dynamic Tick Interface
|
|||
|
|
^^^^^^^^^^^^^^^^^^^^^^
|
|||
|
|
|
|||
|
|
Due to energy-efficiency considerations, RCU is forbidden from
|
|||
|
|
disturbing idle CPUs. CPUs are therefore required to notify RCU when
|
|||
|
|
entering or leaving idle state, which they do via fully ordered
|
|||
|
|
value-returning atomic operations on a per-CPU variable. The ordering
|
|||
|
|
effects are as shown below:
|
|||
|
|
|
|||
|
|
.. kernel-figure:: TreeRCU-dyntick.svg
|
|||
|
|
|
|||
|
|
The RCU grace-period kernel thread samples the per-CPU idleness variable
|
|||
|
|
while holding the corresponding CPU's leaf ``rcu_node`` structure's
|
|||
|
|
``->lock``. This means that any RCU read-side critical sections that
|
|||
|
|
precede the idle period (the oval near the top of the diagram above)
|
|||
|
|
will happen before the end of the current grace period. Similarly, the
|
|||
|
|
beginning of the current grace period will happen before any RCU
|
|||
|
|
read-side critical sections that follow the idle period (the oval near
|
|||
|
|
the bottom of the diagram above).
|
|||
|
|
|
|||
|
|
Plumbing this into the full grace-period execution is described
|
|||
|
|
`below <Forcing Quiescent States_>`__.
|
|||
|
|
|
|||
|
|
CPU-Hotplug Interface
|
|||
|
|
^^^^^^^^^^^^^^^^^^^^^
|
|||
|
|
|
|||
|
|
RCU is also forbidden from disturbing offline CPUs, which might well be
|
|||
|
|
powered off and removed from the system completely. CPUs are therefore
|
|||
|
|
required to notify RCU of their comings and goings as part of the
|
|||
|
|
corresponding CPU hotplug operations. The ordering effects are shown
|
|||
|
|
below:
|
|||
|
|
|
|||
|
|
.. kernel-figure:: TreeRCU-hotplug.svg
|
|||
|
|
|
|||
|
|
Because CPU hotplug operations are much less frequent than idle
|
|||
|
|
transitions, they are heavier weight, and thus acquire the CPU's leaf
|
|||
|
|
``rcu_node`` structure's ``->lock`` and update this structure's
|
|||
|
|
``->qsmaskinitnext``. The RCU grace-period kernel thread samples this
|
|||
|
|
mask to detect CPUs having gone offline since the beginning of this
|
|||
|
|
grace period.
|
|||
|
|
|
|||
|
|
Plumbing this into the full grace-period execution is described
|
|||
|
|
`below <Forcing Quiescent States_>`__.
|
|||
|
|
|
|||
|
|
Forcing Quiescent States
|
|||
|
|
^^^^^^^^^^^^^^^^^^^^^^^^
|
|||
|
|
|
|||
|
|
As noted above, idle and offline CPUs cannot report their own quiescent
|
|||
|
|
states, and therefore the grace-period kernel thread must do the
|
|||
|
|
reporting on their behalf. This process is called “forcing quiescent
|
|||
|
|
states”, it is repeated every few jiffies, and its ordering effects are
|
|||
|
|
shown below:
|
|||
|
|
|
|||
|
|
.. kernel-figure:: TreeRCU-gp-fqs.svg
|
|||
|
|
|
|||
|
|
Each pass of quiescent state forcing is guaranteed to traverse the leaf
|
|||
|
|
``rcu_node`` structures, and if there are no new quiescent states due to
|
|||
|
|
recently idled and/or offlined CPUs, then only the leaves are traversed.
|
|||
|
|
However, if there is a newly offlined CPU as illustrated on the left or
|
|||
|
|
a newly idled CPU as illustrated on the right, the corresponding
|
|||
|
|
quiescent state will be driven up towards the root. As with
|
|||
|
|
self-reported quiescent states, the upwards driving stops once it
|
|||
|
|
reaches an ``rcu_node`` structure that has quiescent states outstanding
|
|||
|
|
from other CPUs.
|
|||
|
|
|
|||
|
|
+-----------------------------------------------------------------------+
|
|||
|
|
| **Quick Quiz**: |
|
|||
|
|
+-----------------------------------------------------------------------+
|
|||
|
|
| The leftmost drive to root stopped before it reached the root |
|
|||
|
|
| ``rcu_node`` structure, which means that there are still CPUs |
|
|||
|
|
| subordinate to that structure on which the current grace period is |
|
|||
|
|
| waiting. Given that, how is it possible that the rightmost drive to |
|
|||
|
|
| root ended the grace period? |
|
|||
|
|
+-----------------------------------------------------------------------+
|
|||
|
|
| **Answer**: |
|
|||
|
|
+-----------------------------------------------------------------------+
|
|||
|
|
| Good analysis! It is in fact impossible in the absence of bugs in |
|
|||
|
|
| RCU. But this diagram is complex enough as it is, so simplicity |
|
|||
|
|
| overrode accuracy. You can think of it as poetic license, or you can |
|
|||
|
|
| think of it as misdirection that is resolved in the |
|
|||
|
|
| `stitched-together diagram <Putting It All Together_>`__. |
|
|||
|
|
+-----------------------------------------------------------------------+
|
|||
|
|
|
|||
|
|
Grace-Period Cleanup
|
|||
|
|
^^^^^^^^^^^^^^^^^^^^
|
|||
|
|
|
|||
|
|
Grace-period cleanup first scans the ``rcu_node`` tree breadth-first
|
|||
|
|
advancing all the ``->gp_seq`` fields, then it advances the
|
|||
|
|
``rcu_state`` structure's ``->gp_seq`` field. The ordering effects are
|
|||
|
|
shown below:
|
|||
|
|
|
|||
|
|
.. kernel-figure:: TreeRCU-gp-cleanup.svg
|
|||
|
|
|
|||
|
|
As indicated by the oval at the bottom of the diagram, once grace-period
|
|||
|
|
cleanup is complete, the next grace period can begin.
|
|||
|
|
|
|||
|
|
+-----------------------------------------------------------------------+
|
|||
|
|
| **Quick Quiz**: |
|
|||
|
|
+-----------------------------------------------------------------------+
|
|||
|
|
| But when precisely does the grace period end? |
|
|||
|
|
+-----------------------------------------------------------------------+
|
|||
|
|
| **Answer**: |
|
|||
|
|
+-----------------------------------------------------------------------+
|
|||
|
|
| There is no useful single point at which the grace period can be said |
|
|||
|
|
| to end. The earliest reasonable candidate is as soon as the last CPU |
|
|||
|
|
| has reported its quiescent state, but it may be some milliseconds |
|
|||
|
|
| before RCU becomes aware of this. The latest reasonable candidate is |
|
|||
|
|
| once the ``rcu_state`` structure's ``->gp_seq`` field has been |
|
|||
|
|
| updated, but it is quite possible that some CPUs have already |
|
|||
|
|
| completed phase two of their updates by that time. In short, if you |
|
|||
|
|
| are going to work with RCU, you need to learn to embrace uncertainty. |
|
|||
|
|
+-----------------------------------------------------------------------+
|
|||
|
|
|
|||
|
|
Callback Invocation
|
|||
|
|
^^^^^^^^^^^^^^^^^^^
|
|||
|
|
|
|||
|
|
Once a given CPU's leaf ``rcu_node`` structure's ``->gp_seq`` field has
|
|||
|
|
been updated, that CPU can begin invoking its RCU callbacks that were
|
|||
|
|
waiting for this grace period to end. These callbacks are identified by
|
|||
|
|
``rcu_advance_cbs()``, which is usually invoked by
|
|||
|
|
``__note_gp_changes()``. As shown in the diagram below, this invocation
|
|||
|
|
can be triggered by the scheduling-clock interrupt
|
|||
|
|
(``rcu_sched_clock_irq()`` on the left) or by idle entry
|
|||
|
|
(``rcu_cleanup_after_idle()`` on the right, but only for kernels build
|
|||
|
|
with ``CONFIG_RCU_FAST_NO_HZ=y``). Either way, ``RCU_SOFTIRQ`` is
|
|||
|
|
raised, which results in ``rcu_do_batch()`` invoking the callbacks,
|
|||
|
|
which in turn allows those callbacks to carry out (either directly or
|
|||
|
|
indirectly via wakeup) the needed phase-two processing for each update.
|
|||
|
|
|
|||
|
|
.. kernel-figure:: TreeRCU-callback-invocation.svg
|
|||
|
|
|
|||
|
|
Please note that callback invocation can also be prompted by any number
|
|||
|
|
of corner-case code paths, for example, when a CPU notes that it has
|
|||
|
|
excessive numbers of callbacks queued. In all cases, the CPU acquires
|
|||
|
|
its leaf ``rcu_node`` structure's ``->lock`` before invoking callbacks,
|
|||
|
|
which preserves the required ordering against the newly completed grace
|
|||
|
|
period.
|
|||
|
|
|
|||
|
|
However, if the callback function communicates to other CPUs, for
|
|||
|
|
example, doing a wakeup, then it is that function's responsibility to
|
|||
|
|
maintain ordering. For example, if the callback function wakes up a task
|
|||
|
|
that runs on some other CPU, proper ordering must in place in both the
|
|||
|
|
callback function and the task being awakened. To see why this is
|
|||
|
|
important, consider the top half of the `grace-period
|
|||
|
|
cleanup`_ diagram. The callback might be
|
|||
|
|
running on a CPU corresponding to the leftmost leaf ``rcu_node``
|
|||
|
|
structure, and awaken a task that is to run on a CPU corresponding to
|
|||
|
|
the rightmost leaf ``rcu_node`` structure, and the grace-period kernel
|
|||
|
|
thread might not yet have reached the rightmost leaf. In this case, the
|
|||
|
|
grace period's memory ordering might not yet have reached that CPU, so
|
|||
|
|
again the callback function and the awakened task must supply proper
|
|||
|
|
ordering.
|
|||
|
|
|
|||
|
|
Putting It All Together
|
|||
|
|
~~~~~~~~~~~~~~~~~~~~~~~
|
|||
|
|
|
|||
|
|
A stitched-together diagram is here:
|
|||
|
|
|
|||
|
|
.. kernel-figure:: TreeRCU-gp.svg
|
|||
|
|
|
|||
|
|
Legal Statement
|
|||
|
|
~~~~~~~~~~~~~~~
|
|||
|
|
|
|||
|
|
This work represents the view of the author and does not necessarily
|
|||
|
|
represent the view of IBM.
|
|||
|
|
|
|||
|
|
Linux is a registered trademark of Linus Torvalds.
|
|||
|
|
|
|||
|
|
Other company, product, and service names may be trademarks or service
|
|||
|
|
marks of others.
|