K 10
svn:author
V 5
markj
K 8
svn:date
V 27
2018-03-02T21:50:02.175472Z
K 7
svn:log
V 3331
Add queues for batching page queue operations and page frees.

As an alternative to the approach taken in r328860, relax the locking
protocol for the queue field of struct vm_page and introduce per-CPU
batch queues for each page queue and for each free queue. This approach
reduces lock contention for enqueue, dequeue, and requeue operations
by separating logical and physical queue state. In general, logical
queue state is protected by the page lock, while physical queue state
is protected by a page queue lock. Queue state is encoded in the
queue and aflags fields. When performing a queue operation on a page,
the logical operation is performed first, with the page lock held,
and the physical operation is deferred using a batch queue. Physical
operations may be deferred indefinitely (in particular, until after
the page has been freed), but the number of pages whose logical and
physical queue states do not match is bounded by a small number.

The queue state of pages is also now decoupled from the allocation
state: pages may be freed without having been physically dequeued
(though they must be logically dequeued). The page allocators ensure
that pages have been physically dequeued before a page is reused. One
consequence of this is that page queue locks must now be leaf locks.
As a result, active queue scanning is modified to work the same as
inactive and laundry queue scanning do, so the active queue lock is not
held when calling into the pmap layer during a scan.

The queue field now encodes the logical queue state of the page, and
the new PGA_ENQUEUED flag indicates whether the page is physically
enqueued. To update the queue field of a page, the queue lock for its
old value must be held: the page queue lock if the value is not PQ_NONE,
and the page lock otherwise. When performing such an update, one of the
new or old values must be PQ_NONE. To enqueue a page, the queue field
is updated to the index of the queue; later, the page is physically
enqueued while the page queue lock is held, and PGA_ENQUEUED is set.
The PGA_ENQUEUED flag may only be set or cleared with the corresponding
page queue lock held.

Logical dequeues and requeues are requested using the PGA_DEQUEUE and
PGA_REQUEUE flags, respectively. Both must be set with the page lock
held, and can only be cleared once the corresponding physical operation
has been performed, with the page queue lock held. As mentioned above,
pages must be at least logically dequeued before being freed.

The inactive queue scanning algorithm is changed to exploit the relaxed
locking protocol. Rather than acquire the inactive queue lock once
per page during a scan, we collect and physically dequeue a batch of
pages, which is then processed using only the page and object locks.
The fact that logical queue state is encoded in the page's atomic flags
allows the page daemon to synchronize with a thread which is
simultaneously freeing pages from the object.

While this approach brings with it considerable complexity, it also
allows some simplification of existing code. For instance, the page
queue lock dance in vm_object_terminate() goes away since the dequeues
are automatically batched. We also no longer need to use
vm_pageout_fallback_object_lock() in the inactive queue scan. The
laundry queue scan still uses it, but it is not required.

END