bpf: Introduce global percpu data#12797
Open
kernel-patches-daemon-bpf[bot] wants to merge 9 commits into
Open
Conversation
There are many adjacent blank lines in the verifier that have accumulated over time. Drop them for cleanup. No functional changes intended. Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Introduce global percpu data, inspired by the commit 6316f78 ("Merge branch 'support-global-data'"). It enables the definition of global percpu variables in BPF, similar to the include/linux/percpu-defs.h::DEFINE_PER_CPU() macro. For example, in BPF, it is able to define a global percpu variable like: int data SEC(".percpu"); With this patch, tools like retsnoop [1] and bpfsnoop [2] can simplify their BPF code for handling LBRs. The code can be updated from static struct perf_branch_entry lbrs[1][MAX_LBR_ENTRIES] SEC(".data.lbrs"); to static struct perf_branch_entry lbrs[MAX_LBR_ENTRIES] SEC(".percpu.lbrs"); This eliminates the need to retrieve the CPU ID using the bpf_get_smp_processor_id() helper. Additionally, by reusing global percpu data map, sharing information between tail callers and callees or freplace callers and callees becomes simpler compared to reusing percpu_array maps. Links: [1] https://github.com/anakryiko/retsnoop [2] https://github.com/bpfsnoop/bpfsnoop Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
libbpf needs a reliable way to distinguish kernels that can support global percpu data from those that cannot. Add a dedicated feature probe, so libbpf can make capability decisions early and fail predictably when global percpu data is unavailable. Acked-by: Andrii Nakryiko <andrii@kernel.org> Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Add support for global percpu data in libbpf by adding a new ".percpu" section, similar to ".data". It enables efficient handling of percpu global variables in bpf programs. When generating loader for lightweight skeleton, update the percpu_array map used for global percpu data using BPF_F_ALL_CPUS, in order to update values across all CPUs using one value slot. Unlike global data, the mmaped data for global percpu data will be marked as read-only after populating the percpu_array map. Thereafter, users can read those initialized percpu data after loading prog. If they want to update the percpu data after loading prog, they have to update the percpu_array map using key=0 instead. Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Enhance bpftool to generate skeletons that properly handle global percpu
variables. The generated skeleton now includes a dedicated structure for
percpu data, allowing users to initialize and access percpu variables more
efficiently.
For global percpu variables, the skeleton now includes a nested
structure, e.g.:
struct test_global_percpu_data {
struct bpf_object_skeleton *skeleton;
struct bpf_object *obj;
struct {
struct bpf_map *percpu;
} maps;
// ...
struct test_global_percpu_data__percpu {
int data;
char run;
struct {
char set;
int i;
int nums[7];
} struct_data;
int nums[7];
} *percpu;
// ...
};
* The "struct test_global_percpu_data__percpu *percpu" points to
initialized data, which is actually "maps.percpu->mmaped".
* Before loading the skeleton, updating the
"struct test_global_percpu_data__percpu *percpu" modifies the initial
value of the corresponding global percpu variables.
* After loading the skeleton, "maps.percpu->mmaped" has been marked as
read-only in libbpf. If users want to update the global percpu
variables, they have to update the "maps.percpu" map instead.
* For lightweight skeleton, "lskel->percpu" will be protected by
"mprotect(p, sz, PROT_READ)".
* For subskeleton, those variables of global percpu data will be
skipped.
Acked-by: Quentin Monnet <qmo@kernel.org>
Assisted-by: Codex:gpt-5.5-xhigh
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
If the arch, like s390x, does not support percpu insn, these cases won't test global percpu data by checking FEAT_PERCPU_DATA support. The following APIs have been tested for global percpu data: 1. bpf_map__set_initial_value() 2. bpf_map__initial_value() 3. generated percpu struct pointer pointing to internal map's mmaped data 4. bpf_map__lookup_elem() for global percpu data map At the same time, the case is also tested with 'bpftool gen skeleton -L'. Add a test to verify that the live vars of subskel won't include the vars for global percpu data. Assisted-by: Codex:gpt-5.5-xhigh Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Verify these two cases: 1. Direct reading the data of read-only percpu data's percpu_array map is allowed. 2. Direct writing the data of read-only percpu data's percpu_array map is disallowed. Assisted-by: Codex:gpt-5.5-xhigh Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Add two tests to verify the verifier log "R%d points to percpu_array map which cannot be used as const string\n". Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Add a test to verify that it is OK to iter the percpu_array map used for global percpu data. Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Author
|
Upstream branch: 9a3a07d |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull request for series with
subject: bpf: Introduce global percpu data
version: 9
url: https://patchwork.kernel.org/project/netdevbpf/list/?series=1126781