Add scripts to generate fresh vmlinux.h #6
Conversation
theihor
force-pushed
the
gen-script
branch
6 times, most recently
from
5ea6b67 to
5955a7c
Compare
theihor
changed the title
|
I'm going to make arch-specific kconfigs and clean up commits a bit. Will update the PR soon-ish. |
|
@anakryiko ok, I think this is ready for review Output is here: https://github.com/libbpf/vmlinux.h/actions/runs/14272784454 |
|
@theihor this is great, let's make it even more great :) I noticed that for 32-bit arm we barely get any kfunc definitions (only two, while other arches have tons of them). Let's try to debug why this is happening, those kfunc definitions in vmlinux.h is often a reason I locally substitute my custom vmlinux.h if I'm playing with libbpf-bootstrap, so it's important to get that right. Also, I'm not a big fan of per-architecture configs, at least not in the part where they all copy-paste a bunch of the same values. If this is all just due to CONFIG_64BIT=y, maybe we can just drop it and keep one small common config, have you tried that? |
Ok
That's what I started with: one small common config. But then when I realized CONFIG_64BIT needs to be removed for arm (I assumed it needs to be preserved for others), I thought there might be other divergencies between arches, and went with copy-pasting per arch. We could also use symlinks or config.common etc. Let me try to figure out a common config that works. |
It's all just lines of text, we can cat together common and per-arch configs |
Would you say having 6 files with |
|
Looks like I agree that copy-pasted configs is not great. I'll use config.common and config.$ARCH for arch-specific lines. |
yes, because arch-agnostic settings are in one place |
|
@anakryiko ready for review |
scripts/install-dependencies.sh
Outdated
| DEBIAN_FRONTEND=noninteractive sudo -E apt install -y tzdata | ||
| sudo apt install -y bc bison build-essential elfutils flex libelf-dev libdw-dev make ncurses-dev python3-docutils zstd |
There was a problem hiding this comment.
How about making these dependencies one per line and keep them sorted ?
There was a problem hiding this comment.
I fixed the order. Putting each package on a newline is not necessary, I think. This list will not change often.
| exit 1 | ||
| fi | ||
|
|
||
| sudo apt install -y llvm libcap-dev libbfd-dev zlib1g-dev |
There was a problem hiding this comment.
If we install all dependencies in scripts/install-dependencies.sh, this won't be necessary. The same for pahole.
If we install dependencies here, DEBIAN_FRONTEND=noninteractive is missing.
There was a problem hiding this comment.
The idea is to make install-bpftool.sh and install-pahole.sh independently runnable, that's why I put their specific dependencies here.
DEBIAN_FRONTEND=noninteractive is really needed only for tzdata. Why do you think it is necessary here?
There was a problem hiding this comment.
I don't know whether it is necessary for tzdata only.
Add script/gen-vmlinux-headers.sh (adapted from retsnoop [1]), that
can build a Linux Kernel for various architectures and generate
corresponding vmlinux.h header.
Add kconfigs directory with common and per-arch configurations that
can be used to produce vmlinux.h
In gen-vmlinux-header.sh if an explicit config is not specified,
append kconfigs/config.common and kconfigs/config.${ARCH} to build the
Linux Kernel.
[1] https://github.com/anakryiko/retsnoop/blob/master/scripts/gen-vmlinux-headers.sh
Signed-off-by: Ihor Solodrai <ihor.solodrai@linux.dev>
* download-latest-linux-release.sh finds and fetches latest linux release from kernel.org * install-bpftool.sh builds and installs bpftool from source in the provided Linux tree * install-pahole.sh downloads and builds pahole from source * install-dependencies.sh assumes a clean ubuntu/debian environment and set ups packages necessary to build the kernel All these can be used to run vmlinux.h gen on Github Actions. Signed-off-by: Ihor Solodrai <ihor.solodrai@linux.dev>
Define a workflow job that uses ./scripts to generate vmlinux.h from the latest Linux mainline release for various architectures. Kernel build and vmlinux.h generation execute in a separate step for each target architecture, while common dependencies are installed at the beginning. Signed-off-by: Ihor Solodrai <ihor.solodrai@linux.dev>
|
|
||
| source $(dirname "$0")/helpers.sh | ||
|
|
||
| WORKSPACE=$(pwd) |
There was a problem hiding this comment.
This assumes that the script can be run in project root only ?
There was a problem hiding this comment.
This assumes that the script can be run in project root only ?
Yeah, well realisticly this is the way it's going to run almost always. There are other assumptions, like the location of kconfigs relative to workspace.
I don't think it's worth it right now to rewrite the script to be location agnostic.
anakryiko
left a comment
anakryiko
left a comment
There was a problem hiding this comment.
This is a great unification and automation, so going to land it now. We can always improve and extend scripts as the need arises. Thanks everyone!
3 participants
As @anakryiko suggested, I took retsnoop/scripts/gen-vmlinux-headers.sh as a starting point and wrote bash scripts to be able to generate fresh vmlinux.h on demand.
I thought it's a good idea to make this automatable, so I also added scripts to install all necessary dependencies and download latest Linux mainline release.
With that I added a Github Actions job. It does the following:
gen-vmlinux-header.shfor each archI thought maybe instread of artifacts the CI job may open a PR, but that requires a bit more work, and I'm not sure if that's useful. If it's not important to update vmlinux.h quickly for each new Linux release, then we can push it manually when there is a need.
I used a slightly modified kconfig from retsnoop script. It's pretty minimal, but maybe there are other things we need here?