diff options
author | Alexander Lobakin <aleksander.lobakin@intel.com> | 2023-08-04 21:05:26 +0300 |
---|---|---|
committer | Jakub Kicinski <kuba@kernel.org> | 2023-08-07 23:05:53 +0300 |
commit | 06d0fbdad612cb8def19065cf1fa14fc34dba9f8 (patch) | |
tree | e902a88ce2ea04869be76384380ce95718e5c686 /include/net/page_pool | |
parent | 75eaf63ea7afeafd026ffef03bdc69e31f10829b (diff) | |
download | linux-06d0fbdad612cb8def19065cf1fa14fc34dba9f8.tar.xz |
page_pool: place frag_* fields in one cacheline
On x86_64, frag_* fields of struct page_pool are scattered across two
cachelines despite the summary size of 24 bytes. All three fields are
used in pretty much the same places, but the last field, ::frag_users,
is pushed out to the next CL, provoking unwanted false-sharing on
hotpath (frags allocation code).
There are some holes and cold members to move around. Move frag_* one
block up, placing them right after &page_pool_params perfectly at the
beginning of CL2. This doesn't do any meaningful to the second block, as
those are some destroy-path cold structures, and doesn't do anything to
::alloc_stats, which still starts at 200-byte offset, 8 bytes after CL3
(still fitting into 1 cacheline).
On my setup, this yields 1-2% of Mpps when using PP frags actively.
When it comes to 32-bit architectures with 32-byte CL: &page_pool_params
plus ::pad is 44 bytes, the block taken care of is 16 bytes within one
CL, so there should be at least no regressions from the actual change.
::pages_state_hold_cnt is not related directly to that triple, but is
paired currently with ::frags_offset and decoupling them would mean
either two 4-byte holes or more invasive layout changes.
Signed-off-by: Alexander Lobakin <aleksander.lobakin@intel.com>
Reviewed-by: Alexander Duyck <alexanderduyck@fb.com>
Link: https://lore.kernel.org/r/20230804180529.2483231-4-aleksander.lobakin@intel.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Diffstat (limited to 'include/net/page_pool')
-rw-r--r-- | include/net/page_pool/types.h | 10 |
1 files changed, 5 insertions, 5 deletions
diff --git a/include/net/page_pool/types.h b/include/net/page_pool/types.h index fcb846523398..887e7946a597 100644 --- a/include/net/page_pool/types.h +++ b/include/net/page_pool/types.h @@ -123,16 +123,16 @@ struct page_pool_stats { struct page_pool { struct page_pool_params p; + long frag_users; + struct page *frag_page; + unsigned int frag_offset; + u32 pages_state_hold_cnt; + struct delayed_work release_dw; void (*disconnect)(void *pool); unsigned long defer_start; unsigned long defer_warn; - u32 pages_state_hold_cnt; - unsigned int frag_offset; - struct page *frag_page; - long frag_users; - #ifdef CONFIG_PAGE_POOL_STATS /* these stats are incremented while in softirq context */ struct page_pool_alloc_stats alloc_stats; |