kernel 5.14 spec file problems, one solved, one unresolved, help

[stan]

Hi,
I'm building the 5.14 kernel from 
kernel-5.14.0-0.rc0.20210706git79160a603bdb.11.fc35.src.rpm
I've run into two issues.

The first is to do with bpftool.  I see in the comments that it is
supposed to be disabled in Fedora.  But, it still has a buildrequires:,
and there is a segment of code that is not protected even if bpftool is
turned off.

%ifnarch armv7hl
    # Generate vmlinux.h and put it to kernel-devel path
    bpftool btf dump file vmlinux format c > $RPM_BUILD_ROOT/$DevelDir/vmlinux.h
%endif

Because bpftool is turned off, this chokes.

The second is to do with a symbolic link.

# Move the devel headers out of the root file system
    mkdir -p $RPM_BUILD_ROOT/usr/src/kernels
    mv $RPM_BUILD_ROOT/lib/modules/$KernelVer/build $RPM_BUILD_ROOT/$DevelDir

# This is going to create a broken link during the build, but we don't use
    # it after this point.  We need the link to actually point to something
    # when kernel-devel is installed, and a relative link doesn't work across
    # the F17 UsrMove feature.
    ln -sf $DevelDir $RPM_BUILD_ROOT/lib/modules/$KernelVer/build

The comment says that symbolic link should not matter, but rpmbuild
complains that files are being packaged that are not registered (?,
from memory).  Building something else right now, but if you need the
exact error message, I can regenerate it later.  This code is in
prior kernel spec files without problems.  It seems that something
is being more assiduous now.  I've tried various things to work around
this, but they haven't worked.  I guess that is because I don't really
understand why this symbolic link is being created.  What is your
suggestion for how to fix this?

Finally, I've built the kernel successfully several times while testing
my fixes.  I've noticed that ccache is not being used; it is rebuilding
everything every time.  Since I'm building a kernel customized to my
hardware, that isn't so onerous.  But, how would I enable ccache so
that rebuilds are basically copy operations?

Thanks for any help.

[Justin Forbes]

On Wed, Jul 7, 2021 at 9:41 AM stan via kernel
 wrote:
> Hi,
> I'm building the 5.14 kernel from
> kernel-5.14.0-0.rc0.20210706git79160a603bdb.11.fc35.src.rpm
> I've run into two issues.
>
> The first is to do with bpftool.  I see in the comments that it is
> supposed to be disabled in Fedora.  But, it still has a buildrequires:,
> and there is a segment of code that is not protected even if bpftool is
> turned off.
>
> %ifnarch armv7hl
>     # Generate vmlinux.h and put it to kernel-devel path
>     bpftool btf dump file vmlinux format c > $RPM_BUILD_ROOT/$DevelDir/vmlinux.h
> %endif
>
> Because bpftool is turned off, this chokes.

bpftool is a builreq, the failure there has to do with where you are
building it. You need bpftool from 5.13 to build a 5.14 snapshot.  The
error is unclear, but that is the solution.  This is actually a very
useful bit in that we will use that vmlinux.h to build kernel-tools
for Fedora. It is built in kernel for ELN.  As I rebase stable Fedora
to 5.13, it will require a prebuild of kernel-tools for 5.13 followed
by the kernel build.
> The second is to do with a symbolic link.
> # Move the devel headers out of the root file system
> mkdir -p $RPM_BUILD_ROOT/usr/src/kernels
> mv $RPM_BUILD_ROOT/lib/modules/$KernelVer/build $RPM_BUILD_ROOT/$DevelDir
>
> # This is going to create a broken link during the build, but we don't use
> # it after this point.  We need the link to actually point to something
> # when kernel-devel is installed, and a relative link doesn't work across
> # the F17 UsrMove feature.
> ln -sf $DevelDir $RPM_BUILD_ROOT/lib/modules/$KernelVer/build
>
> The comment says that symbolic link should not matter, but rpmbuild
> complains that files are being packaged that are not registered (?,
> from memory).  Building something else right now, but if you need the
> exact error message, I can regenerate it later.  This code is in
> prior kernel spec files without problems.  It seems that something
> is being more assiduous now.  I've tried various things to work around
> this, but they haven't worked.  I guess that is because I don't really
> understand why this symbolic link is being created.  What is your
> suggestion for how to fix this?

Would need to know what link this is. I know perf messed up the
makefile and created a link in the wrong place. We don't package this
link, but our command to remove it wasn't looking in the right place.
Rather than fix this in spec, I sent a fix upstream which is applied
and waiting for the next Linus perf pull.
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1237 but
you shouldn't be hitting that at all if you are doing a fedora build
which doesn't build perf.  If it isn't that, let me know what it
actually is, since I am not seeing it in rawhide builds.
> Finally, I've built the kernel successfully several times while testing
> my fixes.  I've noticed that ccache is not being used; it is rebuilding
> everything every time.  Since I'm building a kernel customized to my
> hardware, that isn't so onerous.  But, how would I enable ccache so
> that rebuilds are basically copy operations?

For building rpms, I am not sure here.  When I am debugging and
testing non packaging related fixes, I tend to just build manually in
the kernel tree.

Justin

[stan]

On Wed, 7 Jul 2021 10:21:01 -0500
Justin Forbes  wrote:
> bpftool is a builreq, the failure there has to do with where you are
> building it. You need bpftool from 5.13 to build a 5.14 snapshot.  The
> error is unclear, but that is the solution.  This is actually a very
> useful bit in that we will use that vmlinux.h to build kernel-tools
> for Fedora. It is built in kernel for ELN.  As I rebase stable Fedora
> to 5.13, it will require a prebuild of kernel-tools for 5.13 followed
> by the kernel build.

I have bpftool-5.13.0-1.fc35.x86_64 installed, and that seems to have
satisfied the buildrequire.  I am also building perf, tools,
headers, and devel, so that everything is in sync with the custom
kernel.  That used to work fine as late as 5.10 kernels.  So, my
takeaway is that this has changed, and I now need to enable the build
of vmlinux.h.

The error message I received was that bpftool could not find BFT.  I
infer from your response that turning on the build of bpf will remedy
that.
> Would need to know what link this is. I know perf messed up the
> makefile and created a link in the wrong place. We don't package this
> link, but our command to remove it wasn't looking in the right place.
> Rather than fix this in spec, I sent a fix upstream which is applied
> and waiting for the next Linus perf pull.
> https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1237 but
> you shouldn't be hitting that at all if you are doing a fedora build
> which doesn't build perf.  If it isn't that, let me know what it
> actually is, since I am not seeing it in rawhide builds.

But as I say above, I *am* building perf.  So, this solves the second
problem.  I can wait for a fedora kernel that has the fix, or I can try
to apply that patch manually.

The main reason I'm doing this is that the Fedora kernel has the
configuration option for the rtl2832 module turned on.  That is to use
it for its intended purpose, as a tuner, and that is reasonable.  But, I
use an rtl2832 as a source of atmospheric entropy.  My daemon can't
acquire it unless the rtl2832 is free, which it isn't with a stock
Fedora kernel.  The slimmed down size and reduced attack surface are
just bonuses of building the custom kernel.

Thanks for your help.

[stan]

On Wed, 7 Jul 2021 10:04:07 -0700
stan via kernel  wrote:
> On Wed, 7 Jul 2021 10:21:01 -0500
> Justin Forbes  wrote:
> > bpftool is a builreq, the failure there has to do with where you are
> > building it. You need bpftool from 5.13 to build a 5.14 snapshot.
> > The error is unclear, but that is the solution.  This is actually a
> > very useful bit in that we will use that vmlinux.h to build
> > kernel-tools for Fedora. It is built in kernel for ELN.  As I
> > rebase stable Fedora to 5.13, it will require a prebuild of
> > kernel-tools for 5.13 followed by the kernel build.
>
> I have bpftool-5.13.0-1.fc35.x86_64 installed, and that seems to have
> satisfied the buildrequire.  I am also building perf, tools,
> headers, and devel, so that everything is in sync with the custom
> kernel.  That used to work fine as late as 5.10 kernels.  So, my
> takeaway is that this has changed, and I now need to enable the build
> of vmlinux.h.
>
> The error message I received was that bpftool could not find BFT.  I
> infer from your response that turning on the build of bpf will remedy
> that.

It doesn't.  Here is the output of a compile with bpf turned on
everywhere.  It shows the creation of the link that causes the problem
of an upackaged file being packaged, and then the failure to build
vmlinux.h using bpftool.

+ cp certs/signing_key.pem certs/signing_key.pem.sign
+ cp certs/signing_key.x509 certs/signing_key.x509.sign
+ mkdir -p /home/stan/rpmbuild/BUILDROOT/kernel-5.14.0-0.rc0.20210706git79160a603bdb.11.20210706.fc35.x86_64/usr/src/kernels
+ mv /home/stan/rpmbuild/BUILDROOT/kernel-5.14.0-0.rc0.20210706git79160a603bdb.11.20210706.fc35.x86_64/lib/modules/5.14.0-0.rc0.20210706git79160a603bdb.11.20210706.fc35.x86_64/build /home/stan/rpmbuild/BUILDROOT/kernel-5.14.0-0.rc0.20210706git79160a603bdb.11.20210706.fc35.x86_64//usr/src/kernels/5.14.0-0.rc0.20210706git79160a603bdb.11.20210706.fc35.x86_64
+ ln -sf /usr/src/kernels/5.14.0-0.rc0.20210706git79160a603bdb.11.20210706.fc35.x86_64 /home/stan/rpmbuild/BUILDROOT/kernel-5.14.0-0.rc0.20210706git79160a603bdb.11.20210706.fc35.x86_64/lib/modules/5.14.0-0.rc0.20210706git79160a603bdb.11.20210706.fc35.x86_64/build
+ bpftool btf dump file vmlinux format c
Error: failed to load BTF from vmlinux: No such file or directory
error: Bad exit status from /var/tmp/rpm-tmp.Zt6DVC (%build)
    Bad exit status from /var/tmp/rpm-tmp.Zt6DVC (%build)

I have debug builds turned off.  Are they required for bpftool to work?

[Justin Forbes]

On Wed, Jul 7, 2021 at 2:40 PM stan via kernel
 wrote:
> On Wed, 7 Jul 2021 10:04:07 -0700
> stan via kernel  wrote:
> > On Wed, 7 Jul 2021 10:21:01 -0500
> > Justin Forbes  wrote:
> > > bpftool is a builreq, the failure there has to do with where you are
> > > building it. You need bpftool from 5.13 to build a 5.14 snapshot.
> > > The error is unclear, but that is the solution.  This is actually a
> > > very useful bit in that we will use that vmlinux.h to build
> > > kernel-tools for Fedora. It is built in kernel for ELN.  As I
> > > rebase stable Fedora to 5.13, it will require a prebuild of
> > > kernel-tools for 5.13 followed by the kernel build.
> >
> > I have bpftool-5.13.0-1.fc35.x86_64 installed, and that seems to have
> > satisfied the buildrequire.  I am also building perf, tools,
> > headers, and devel, so that everything is in sync with the custom
> > kernel.  That used to work fine as late as 5.10 kernels.  So, my
> > takeaway is that this has changed, and I now need to enable the build
> > of vmlinux.h.
> >
> > The error message I received was that bpftool could not find BFT.  I
> > infer from your response that turning on the build of bpf will remedy
> > that.
>
> It doesn't.  Here is the output of a compile with bpf turned on
> everywhere.  It shows the creation of the link that causes the problem
> of an upackaged file being packaged, and then the failure to build
> vmlinux.h using bpftool.
> cp certs/signing_key.pem certs/signing_key.pem.sign
> cp certs/signing_key.x509 certs/signing_key.x509.sign
> mkdir -p /home/stan/rpmbuild/BUILDROOT/kernel-5.14.0-0.rc0.20210706git79160a603bdb.11.20210706.fc35.x86_64/usr/src/kernels
> mv /home/stan/rpmbuild/BUILDROOT/kernel-5.14.0-0.rc0.20210706git79160a603bdb.11.20210706.fc35.x86_64/lib/modules/5.14.0-0.rc0.20210706git79160a603bdb.11.20210706.fc35.x86_64/build /home/stan/rpmbuild/BUILDROOT/kernel-5.14.0-0.rc0.20210706git79160a603bdb.11.20210706.fc35.x86_64//usr/src/kernels/5.14.0-0.rc0.20210706git79160a603bdb.11.20210706.fc35.x86_64
> ln -sf /usr/src/kernels/5.14.0-0.rc0.20210706git79160a603bdb.11.20210706.fc35.x86_64 /home/stan/rpmbuild/BUILDROOT/kernel-5.14.0-0.rc0.20210706git79160a603bdb.11.20210706.fc35.x86_64/lib/modules/5.14.0-0.rc0.20210706git79160a603bdb.11.20210706.fc35.x86_64/build
> bpftool btf dump file vmlinux format c
>
> Error: failed to load BTF from vmlinux: No such file or directory
> error: Bad exit status from /var/tmp/rpm-tmp.Zt6DVC (%build)
>     Bad exit status from /var/tmp/rpm-tmp.Zt6DVC (%build)
>
> I have debug builds turned off.  Are they required for bpftool to work?

No, BTF debuginfo is set on regular kernels as well. Are you building
this on a rawhide system? My build/test system here is F34, but with
bpftool-5.13.0-1.fc35.x86_64 (rebuilt against F34) it seems to pass
fine on this.  I got the same error that you did when I was running an
older version of kernel-tools.  In your particular user case though,
you can comment out that line. The vmlinux.h created there is only
used in 2 instances, where kernel-tools is built separately from the
kernel, and when cross compiling the kernel (because tools have to be
built native).

Justin

[stan]

On Wed, 7 Jul 2021 15:55:15 -0500
Justin Forbes  wrote:
> No, BTF debuginfo is set on regular kernels as well. Are you building
> this on a rawhide system? My build/test system here is F34, but with
> bpftool-5.13.0-1.fc35.x86_64 (rebuilt against F34) it seems to pass
> fine on this.  I got the same error that you did when I was running an
> older version of kernel-tools.  In your particular user case though,
> you can comment out that line. The vmlinux.h created there is only
> used in 2 instances, where kernel-tools is built separately from the
> kernel, and when cross compiling the kernel (because tools have to be
> built native).

Yes, on rawhide, running under the Fedora 5.14 kernel I am trying to
custom build. Great, that is simple and straightforward.