Running rpmlint within mock

[Jason L Tibbitts III]

Christian Iseli and I were discussing the possibility of automatically
running rpmlint somehow.  It seems that the end of the mock build
process is the ideal place for this.  It has a chroot already set up
with the package's build requirements already installed.  (Obviously
this doesn't include the runtime requirements, but generally there's
quite some overlap.)  It also has easy access to the freshly built
binary and source RPMs.

How difficult would it be to, at the end of the build process, install
the freshly built package, install rpmlint, and run rpmlint on the
source and any binary RPMs that were built?

My python is poor but I'm willing to take a stab at it if someone
could give me a few pointers.

 - J<

[Clark Williams]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jason L Tibbitts III wrote:
>  Christian Iseli and I were discussing the possibility of
> automatically
>  running rpmlint somehow.  It seems that the end of the mock build
>  process is the ideal place for this.  It has a chroot already set up
>  with the package's build requirements already installed.  (Obviously
>  this doesn't include the runtime requirements, but generally there's
>  quite some overlap.)  It also has easy access to the freshly built
>  binary and source RPMs.
>
>  How difficult would it be to, at the end of the build process, install
>  the freshly built package, install rpmlint, and run rpmlint on the
>  source and any binary RPMs that were built? 
Sounds like a good idea.

Why do we need to install the just-generated binary RPMs in the
chroot? Can't we get the same info by just running rpmlint against the
binary and source RPMs?

> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.4 (GNU/Linux)
> Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
>
> iD8DBQFEt+abHyuj/+TTEp0RAvtPAJ4iHlPJFP2ofetHfwFGrSSKuv1wogCcDn2u
> 8Rv7tZiv3JZkXadh9+mK/Es=
> =/0dA
> -----END PGP SIGNATURE-----

[Christian Iseli]

williams(a)redhat.com said:
>  Why do we need to install the just-generated binary RPMs in the
> chroot? 
AFAIU, rpmlint can catch more/other things when a package is installed.
So the full thing seems to be:
 1 rpmlint the src.rpm
 2 rpmlint the <binary/noarch>.rpm
 3 install the rpm and then run rpmlint again

Since mock builds a chroot anyway, Jason and I thought it might not cost much
more just to install the built packages and do an rpmlint then

					Christian

[Jesse Keating]

On Friday 14 July 2006 16:11, Christian.Iseli(a)licr.org wrote:
>  AFAIU, rpmlint can catch more/other things when a package is
> installed.
>  So the full thing seems to be:
>   1 rpmlint the src.rpm
>   2 rpmlint the <binary/noarch>.rpm
>   3 install the rpm and then run rpmlint again
>
>  Since mock builds a chroot anyway, Jason and I thought it might not cost
>  much more just to install the built packages and do an rpmlint then 
This sounds like much more of a job of the Fedora Test Project software that 
Will Woods is talking about. Then you could do more tests like testing for 
conflicts, testing for scriptlet errors, testing for upgrade path / removal, 
etc..  A much more appropriate place for it.

-- 
Jesse Keating RHCE      (geek.j2solutions.net)
Fedora Legacy Team      (www.fedoralegacy.org)
GPG Public Key          (geek.j2solutions.net/jkeating.j2solutions.pub

[Jason L Tibbitts III]

> >>>> "JK" == Jesse Keating
> <jkeating(a)j2solutions.net&gt; writes: 
JK> This sounds like much more of a job of the Fedora Test Project
JK> software that Will Woods is talking about.

I'm not sure how this would help with reviews of packages which
haven't yet been accepted into Fedora.  The idea is to lower the
barrier to providing a good package review.

 - J<

[Jesse Keating]

On Friday 14 July 2006 19:01, Jason L Tibbitts III wrote:
>  I'm not sure how this would help with reviews of packages which
>  haven't yet been accepted into Fedora.  The idea is to lower the
>  barrier to providing a good package review. 
Um, if the package isn't accepted into Fedora, it wouldn't be ran through the 
extras build system....

I guess thats part of where my argument is coming from.  I incorrectly assumed 
you wanted this put into the Extras build system, not mock itself.  Sorry 
about that (;


mock chroot  will allow you to run a comand in the chroot.

mockhelper yum --installroot /root localinstall /rpm
mock --noclean (needed?) -r config chroot rpmlint <PACKAGE_NAME>

-- 
Jesse Keating RHCE      (geek.j2solutions.net)
Fedora Legacy Team      (www.fedoralegacy.org)
GPG Public Key          (geek.j2solutions.net/jkeating.j2solutions.pub)

[Jason L Tibbitts III]

> >>>> "JK" == Jesse Keating
> <jkeating(a)j2solutions.net&gt; writes: 
JK> mock chroot will allow you to run a comand in the chroot.

Oh, awesome.  Honestly, very many thanks.  Might I be so bold as to
suggest something like the following patch?

Index: mock.py
===================================================================
RCS file: /cvs/fedora/mock/mock.py,v
retrieving revision 1.54
diff -u -r1.54 mock.py
--- mock.py     28 Jun 2006 15:14:45 -0000      1.54
+++ mock.py     15 Jul 2006 02:08:57 -0000
@@ -791,6 +791,7 @@
     usage = """
     usage: mock [options] /path/to/srpm
     optional commands:
+        chroot - run the specified command within the chroot
         clean - clean out the specified chroot
         init - initialize the chroot, do not build anything"""
     parser = OptionParser(usage=usage, version=__VERSION__)
cvs diff: Diffing docs
Index: docs/mock.1
===================================================================
RCS file: /cvs/fedora/mock/docs/mock.1,v
retrieving revision 1.2
diff -u -r1.2 mock.1
--- docs/mock.1 21 Aug 2005 16:29:32 -0000      1.2
+++ docs/mock.1 15 Jul 2006 02:08:57 -0000
@@ -37,6 +37,8 @@
 .TP
 \fBinit\fR \- initialize a chroot (install packages, setup devices, etc.)
 .TP
+\fBchroot\fR \- run the specified command within the chroot (which must already be
initialized)
+.TP
 \fBclean\fR \- purge the chroot tree
 .TP
 \fBrebuild\fR \- If no command is specified, rebuild is assumed. Rebuilds the specified
SRPM

 - J<

[Jesse Keating]

On Friday 14 July 2006 22:10, Jason L Tibbitts III wrote:
>  Oh, awesome.  Honestly, very many thanks.  Might I be so bold as to
>  suggest something like the following patch? 
Looks reasonable to me.  Alas, I have no commit access (:  (maybe that's a 
good thing...)

-- 
Jesse Keating RHCE      (geek.j2solutions.net)
Fedora Legacy Team      (www.fedoralegacy.org)
GPG Public Key          (geek.j2solutions.net/jkeating.j2solutions.pub)

[Clark Williams]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jesse Keating wrote:
>  On Friday 14 July 2006 22:10, Jason L Tibbitts III wrote:
> > Oh, awesome.  Honestly, very many thanks.  Might I be so bold as to
> > suggest something like the following patch?
>
>  Looks reasonable to me.  Alas, I have no commit access (:  (maybe that's a
>  good thing...)
>  
Well, I have commit access and as soon as I get back to the house,
this will go in.

Clark

> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.4 (GNU/Linux)
> Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
>
> iD8DBQFEukrDHyuj/+TTEp0RAh+cAKCLefB3+jvXIilFfMUALRV3keB3nQCeJ0AB
> ZYWrZiOmErdIYu66MGqKvks=
> =D+nL
> -----END PGP SIGNATURE-----

[Jason L Tibbitts III]

OK, here's a first pass hack.  This could be better written in Python
and could extract some bits from the mock configuration, but this does
at least work.  It builds the package, installs rpmlint and all of the
newly-built binary RPMs and (hopefully) their dependencies into the
chroot.  It then runs rpmlint on the SRPM, the binary RPMs and the
installed package names.  I have tested this against packages which
show different warnings for each and it seems to actually work.

You'll have to edit to taste, and try not to hurl as some of the
quoting is nasty.

 - J<

#!/bin/sh
MOCKDIR=/var/lib/mock/fedora-development-x86_64-core
MOCKCFG=fedora-6-x86_64-core.cfg
time mock --autocache --debug -r $MOCKCFG $*
echo =======
mock-helper yum --installroot $MOCKDIR/root install rpmlint
echo =======
mock-helper yum --installroot $MOCKDIR/root localinstall
$MOCKDIR/result/*{i386,x86_64,noarch}.rpm
echo =======
echo rpmlint of SRPM:
mock -r $MOCKCFG --debug -- chroot ls -l /builddir/build/SRPMS/\*.rpm \
  | grep -E -v '^(init$|ending$|done$|DEBUG:)' \
  | tee $MOCKDIR/result/rpmlint
mock -r $MOCKCFG --debug -- chroot rpmlint /builddir/build/SRPMS/\*.rpm \
  | grep -E -v '^(init$|ending$|done$|DEBUG:)' \
  | tee -a $MOCKDIR/result/rpmlint
echo ======
echo rpmlint of RPMs:
mock -r $MOCKCFG --debug -- chroot ls -l /builddir/build/RPMS/\*.rpm \
  | grep -E -v '^(init$|ending$|done$|DEBUG:)' \
  | tee -a $MOCKDIR/result/rpmlint
mock -r $MOCKCFG --debug -- chroot rpmlint /builddir/build/RPMS/\*.rpm \
  | grep -E -v '^(init$|ending$|done$|DEBUG:)' \
  | tee -a $MOCKDIR/result/rpmlint
echo ======
echo rpmlint of installed RPMs:
mock -r $MOCKCFG --debug -- chroot rpmlint \\\`rpm -qp /builddir/build/RPMS/\*.rpm --qf
\\\"%{NAME} \\\"\\\` \
  | grep -E -v '^(init$|ending$|done$|DEBUG:)' \
  | tee -a $MOCKDIR/result/rpmlint
echo ======
echo Results in $MOCKDIR/result

[Clark Williams]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jason L Tibbitts III wrote:
>  OK, here's a first pass hack.  This could be better written in
> Python
>  and could extract some bits from the mock configuration, but this does
>  at least work.  It builds the package, installs rpmlint and all of the
>  newly-built binary RPMs and (hopefully) their dependencies into the
>  chroot.  It then runs rpmlint on the SRPM, the binary RPMs and the
>  installed package names.  I have tested this against packages which
>  show different warnings for each and it seems to actually work.
>
>  You'll have to edit to taste, and try not to hurl as some of the
>  quoting is nasty.
>
>   - J<
>  
Note that using mockhelper is deprecated, since we've re-architected
so that mock becomes a chroot launcher for mock.py. Mockhelper will go
away in the next release (0.7).

Just so you know, I have a version of mock that takes a --rpmlint
command line option which will add rpmlint to the chroot transaction
set, install it in the chroot and then run rpmlint on the generated
SRPM and RPMs, putting the output in an rpmlint.log results file.  It
wasn't very intrusive and won't affect build systems unless they
specifically invoke it.

As an aside, I wouldn't mind looking at mock as a chroot builder for
things like the test project. It's mostly there, since it's main job
is to build a chroot and run rpmbuild within that chroot.  We should
discuss.

Clark
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.4 (GNU/Linux)
> Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
>
> iD8DBQFEukxNHyuj/+TTEp0RAgR2AKCOviuiwMw1apU/YXSXig7bzvmzggCeI+fB
> qJ6PTlrYp5XRxGhMbrGWiLA=
> =rQrE
> -----END PGP SIGNATURE-----

[Jason L Tibbitts III]

> >>>> "CW" == Clark Williams
> <williams(a)redhat.com&gt; writes: 
CW> Just so you know, I have a version of mock that takes a --rpmlint
CW> command line option which will add rpmlint to the chroot
CW> transaction set, install it in the chroot and then run rpmlint on
CW> the generated SRPM and RPMs, putting the output in an rpmlint.log
CW> results file.

Ah, very very cool.  One important thing, and the whole reason for
actually doing this, is that in addition to the SRPM and RPMs, you
need to run rpmlint in the chroot on the package name itself.  For
example, build the aplus-fsf package under review.  rpmlint will find
nothing real to complain about with the SRPM or binary RPMs, but if
you run "rpmlint aplus-fsf" in the chroot with that package installed
you get 3800 warnings.

 - J<

[Clark Williams]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jason L Tibbitts III wrote:
> >>>>> "CW" == Clark Williams
> <williams(a)redhat.com&gt; writes:
>
>  CW> Just so you know, I have a version of mock that takes a
>  --rpmlint CW> command line option which will add rpmlint to the
>  chroot CW> transaction set, install it in the chroot and then run
>  rpmlint on CW> the generated SRPM and RPMs, putting the output in
>  an rpmlint.log CW> results file.
>
>  Ah, very very cool.  One important thing, and the whole reason for
>  actually doing this, is that in addition to the SRPM and RPMs, you
>  need to run rpmlint in the chroot on the package name itself.  For
>  example, build the aplus-fsf package under review.  rpmlint will
>  find nothing real to complain about with the SRPM or binary RPMs,
>  but if you run "rpmlint aplus-fsf" in the chroot with that package
>  installed you get 3800 warnings. 
Well dang, now you're going to make me work for it :).

What I've got now just runs rpmlint inside the chroot on the source
and binary packages. How about we go with that for a bit and let me
think on how we could install into the chroot? I'm not really against
it, since I think it goes with the "chroot manager" task of mock, but
I'd like to look at how to do it without creating more problems than
we solve.

The way I see it, mock's primary mission is to be the tool that
creates a pristine chroot for the purpose of buiding a package (or set
of packages); everything we do here needs to keep that in mind. A
related secondary mission though could be management of a longer lived
chroot used for various purposes (testing, cross-building, etc.).

Clark

> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.4 (GNU/Linux)
> Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
>
> iD8DBQFEumcfHyuj/+TTEp0RAuuLAJ4x0mweM+j+yGwNBJ+mcKb2DA+XeQCeKrcs
> mjsmeV5HcgqdONQgUE151+k=
> =zNi5
> -----END PGP SIGNATURE-----

[Jesse Keating]

On Sunday 16 July 2006 12:19, Clark Williams wrote:
>  What I've got now just runs rpmlint inside the chroot on the
> source
>  and binary packages. How about we go with that for a bit and let me
>  think on how we could install into the chroot? I'm not really against
>  it, since I think it goes with the "chroot manager" task of mock, but
>  I'd like to look at how to do it without creating more problems than
>  we solve.
>
>  The way I see it, mock's primary mission is to be the tool that
>  creates a pristine chroot for the purpose of buiding a package (or set
>  of packages); everything we do here needs to keep that in mind. A
>  related secondary mission though could be management of a longer lived
>  chroot used for various purposes (testing, cross-building, etc.). 
How much of this usage falls under what Mach does?  Mock has a pretty narrow 
purpose, and trying to add anything to it might be better through the mach 
project.

-- 
Jesse Keating RHCE      (geek.j2solutions.net)
Fedora Legacy Team      (www.fedoralegacy.org)
GPG Public Key          (geek.j2solutions.net/jkeating.j2solutions.pub)

[Jason L Tibbitts III]

> >>>> "CW" == Clark Williams
> <williams(a)redhat.com&gt; writes: 
CW> Well dang, now you're going to make me work for it :).

Well, no big deal; I'm happy with what I have now, although I really
hope that I'll be able to continue to make it work somehow after
mockhelper goes away.

I think some type of functionality like this is important; we (package
reviewers, that is) need to make sure that the binary packages
actually install, and then run the rpmlint checks on the installed
packages.  But it doesn't necessarily need to be built into mock; mock
just needs to provide the hooks.

 - J<

[Andreas Thienemann]

Hi tibbs,

On Fri, 14 Jul 2006, Jason L Tibbitts III wrote:

>  How difficult would it be to, at the end of the build process,
> install
>  the freshly built package, install rpmlint, and run rpmlint on the
>  source and any binary RPMs that were built? 
Installing the freshly built package is
a big no.
But it's not even necessary for running rpmlint.

You could run rpmlint on the resulting rpms easily.

regards,
 andreas

[Christian Iseli]

andreas(a)bawue.net said:
>  Installing the freshly built package is a big no. 
May I ask why ?

					Christian

[Toshio Kuratomi]

On Fri, 2006-07-14 at 22:12 +0200, Christian.Iseli(a)licr.org wrote:
>  andreas(a)bawue.net said:
>  > Installing the freshly built package is a big no.
>  
>  May I ask why ? 
I don't know what Andras had in mind but one potential problem is that
BuildRequires might not pull in all of a package's Requires.  Noarch
packages would be the primary place you'll see this.

-Toshio

[Jason L Tibbitts III]

> >>>> "TK" == Toshio Kuratomi
> <toshio(a)tiki-lounge.com&gt; writes: 
TK> I don't know what Andras had in mind but one potential problem is
TK> that BuildRequires might not pull in all of a package's Requires.
TK> Noarch packages would be the primary place you'll see this.

Obviously there's a working yum and a full set of package repositories
available, else the chroot wouldn't have been populated in the first
place.  My point was only that the majority of package dependencies
are likely to already be installed in the chroot.  It should be
trivial to get the rest with 'yum localinstall'.

 - J<

[Jesse Keating]

On Friday 14 July 2006 19:03, Jason L Tibbitts III wrote:
>  Obviously there's a working yum and a full set of package
> repositories
>  available, else the chroot wouldn't have been populated in the first
>  place.  My point was only that the majority of package dependencies
>  are likely to already be installed in the chroot.  It should be
>  trivial to get the rest with 'yum localinstall'. 
The working yum is outside the mockroot.  Mock uses mockhelper to 
yum --installroot packages.  So you'd have to leave the root, localinstall, 
and re-enter the chroot.

-- 
Jesse Keating RHCE      (geek.j2solutions.net)
Fedora Legacy Team      (www.fedoralegacy.org)
GPG Public Key          (geek.j2solutions.net/jkeating.j2solutions.pub)

[Andreas Thienemann]

On Fri, 14 Jul 2006 Christian.Iseli(a)licr.org wrote:

>  > Installing the freshly built package is a big no.
>  May I ask why ? 
sure.
mock is a chrooted buildtool.
Installing the package outside of the chroot is completely senseless.
We are building quite a large amount of packages on rhel4, installing a 
fc5 package is sure to break or host the system.

Not gonna happen.

Or even neater: I know that there are at least two people building rpms 
with mock on a debian host.


regards,
 andreas

[Christian Iseli]

andreas(a)bawue.net said:
>  Installing the package outside of the chroot is completely senseless.
>  
I completely agree, but I was thinking about installing the package *within* 
the chroot, not on the build host's root...

Is that also a big no ?

					Christian

[Christian Iseli]

tibbs(a)math.uh.edu said:
>  Christian Iseli and I were discussing the possibility of
> automatically
>  running rpmlint somehow. 
Hey, thanks for posting this...  I was just about to post such a message 
myself :-)

I've tried to give some thought and get some feedback from the f-e-l list 
already.

Basically for each built package, I'd like to
 - get rpmlint output, possibly comparing it against a reference output to see 
   if unknown problems have crept in
 - analyze the compiler warnings
 - check the Provide'd things from the packages and compare against an 
   expected set of things provided

For starters, the whole bunch of tests would just generate a warning to the 
build submitter for things to verify.

Down the road, we might like to fail the build, or put the resulting package 
in a quarantine, if bad things are detected.

I had a first look at adding the checks in the plague builder, but after 
discussing this a bit with Jason, probably a good parts of the checks would be 
better performed while still in the mock buildroot...

My python skills are pretty much nil ATM, but I'm willing to learn too...

Any and all pieces of advice and/or code gladly accepted

Cheers,
					Christian

[Clark Williams]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Christian.Iseli(a)licr.org wrote:
>  tibbs(a)math.uh.edu said:
> > Christian Iseli and I were discussing the possibility of automatically
> > running rpmlint somehow.
>
>  Hey, thanks for posting this...  I was just about to post such a message
>  myself :-)
>
>  I've tried to give some thought and get some feedback from the f-e-l list
>  already.
>
>  Basically for each built package, I'd like to
>   - get rpmlint output, possibly comparing it against a reference output 
to see
>     if unknown problems have crept in 
Getting the rpmlint output
is easy enough. Doing anything more than
that sounds like something out of the scope of mock.

>   - analyze the compiler warnings
>   - check the Provide'd things from the packages and compare against an
>     expected set of things provided
>
>  For starters, the whole bunch of tests would just generate a warning to 
the
>  build submitter for things to verify. 
Ah, we're talking automated test suite here. That's not mock.
>  Down the road, we might like to fail the build, or put the resulting 
package
>  in a quarantine, if bad things are detected.
>
>  I had a first look at adding the checks in the plague builder, but after
>  discussing this a bit with Jason, probably a good parts of the checks 
would be
>  better performed while still in the mock buildroot...
>
>  My python skills are pretty much nil ATM, but I'm willing to learn too...
>
>  Any and all pieces of advice and/or code gladly accepted 
I applaud your desire to automate this stuff, but I don't think that
mock is the right place for it. Mock is strictly a tool for managing a
chroot and building SRPMs in those chroots.  Anything else just
introduces the potential for breaking builds and then Jeremy and Dan
start sending me nasty emails :).

That being said, I think it would be fairly straightforward to run
rpmlint on the generated RPMs and on the SRPM and stash the output
somewhere that's easily accessible. At that point you're talking about
writing an analysis tool to classify the output of a mock build.

Clark




> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.4 (GNU/Linux)
> Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
>
> iD8DBQFEt+7NHyuj/+TTEp0RAuVPAKCKRg/kINpep85DA3F7OXFA+nUi6gCgnPsd
> AfZkfqe9NkWCDNzngIHLqZ0=
> =nvO5
> -----END PGP SIGNATURE-----

[Christian Iseli]

williams(a)redhat.com said:
>  Getting the rpmlint output is easy enough. Doing anything more than
> that
>  sounds like something out of the scope of mock. 
Agreed.

>  Ah, we're talking automated test suite here. 
Yup.

>  That's not mock. 
Ok.

>  I applaud your desire to automate this stuff, but I don't think
> that mock is
>  the right place for it. Mock is strictly a tool for managing a chroot and
>  building SRPMs in those chroots.  Anything else just introduces the potential
>  for breaking builds and then Jeremy and Dan start sending me nasty emails :). 
:)

>  That being said, I think it would be fairly straightforward to run
> rpmlint on
>  the generated RPMs and on the SRPM and stash the output somewhere that's
>  easily accessible. 
Yes, that'd be neat.

>  At that point you're talking about writing an analysis
>  tool to classify the output of a mock build.  
Yes.  I'd like to add that to plague, or as a standalone thing that is called 
by plague after mock succeeds.

					Christian

[Bill Nottingham]

Christian.Iseli(a)licr.org (Christian.Iseli(a)licr.org) said: 
>  My python skills are pretty much nil ATM, but I'm willing to
> learn too...
>  
>  Any and all pieces of advice and/or code gladly accepted 
Sounds like a good idea to me... I suggested some other things to check 
on this list a few months back.

Bill

[Christian Iseli]

notting(a)redhat.com said:
>  Sounds like a good idea to me... I suggested some other things to
> check  on
>  this list a few months back.  
Yes.  I hope I put them all on the wiki here:
http://fedoraproject.org/wiki/Extras/SIGs/QA

					Christian

[Jesse Keating]

On Friday 14 July 2006 13:22, Jason L Tibbitts III wrote:
>  How difficult would it be to, at the end of the build process,
> install
>  the freshly built package, install rpmlint, and run rpmlint on the
>  source and any binary RPMs that were built? 
Why do you need to install the package?  rpmlint works on the binary and srpm.

I think we really need to separate the idea of Building and Testing.  Mock is 
a build tool, not a test tool.

-- 
Jesse Keating RHCE      (geek.j2solutions.net)
Fedora Legacy Team      (www.fedoralegacy.org)
GPG Public Key          (geek.j2solutions.net/jkeating.j2solutions.pub

[Jason L Tibbitts III]

> >>>> "JK" == Jesse Keating
> <jkeating(a)j2solutions.net&gt; writes: 
JK> Why do you need to install the package?  rpmlint works on the
JK> binary and srpm.

Because I was told that is the only way rpmlint can run certain
checks.  The point is that in order to do a proper review, I should
install the package but I just cannot install everything I review on
my build host.

JK> I think we really need to separate the idea of Building and
JK> Testing.  Mock is a build tool, not a test tool.

The whole point is that mock already has everything necessary and
pregenerated.  It has the chroot just sitting there.  It has the
security infrastructure and the procedures for doing things within the
chroot already done.  It seems to me to be terribly foolish to
duplicate that.

How about this: does mock provide (or could it provide) a way for me
to install a couple of packages and run a command within the chroot?
Or is this relatively simple to do safely without using mock?  Forget
about rpmlint; maybe I need this for debugging or to figure out why
something isn't building.

 - J<

[Jesse Keating]

On Friday 14 July 2006 18:48, Jason L Tibbitts III wrote:
>  The whole point is that mock already has everything necessary and
>  pregenerated.  It has the chroot just sitting there.  It has the
>  security infrastructure and the procedures for doing things within the
>  chroot already done.  It seems to me to be terribly foolish to
>  duplicate that.
>
>  How about this: does mock provide (or could it provide) a way for me
>  to install a couple of packages and run a command within the chroot?
>  Or is this relatively simple to do safely without using mock?  Forget
>  about rpmlint; maybe I need this for debugging or to figure out why
>  something isn't building. 
The same software could be used.  Mock can do it, it just isn't designed with 
this task in mind.

More to the point, there are two distinct tasks here, one is building the 
package, the other is testing.  They have some pretty different needs, and 
trying to do both of them in the same harness isn't the best of ideas.  
rpmlint is just a test, it leads to more tests, like I described.  What about 
doing functional tests on things like php packages, where we want to spin up 
a local apache, browse to some pages, and make sure the content looks right.  
These are the types of things that can be done with dogtail in a real test 
environment that really really really shouldn't be done at build time.

-- 
Jesse Keating RHCE      (geek.j2solutions.net)
Fedora Legacy Team      (www.fedoralegacy.org)
GPG Public Key          (geek.j2solutions.net/jkeating.j2solutions.pub)

[Jason L Tibbitts III]

> >>>> "JK" == Jesse Keating
> <jkeating(a)j2solutions.net&gt; writes: 
JK> More to the point, there are two distinct tasks here, one is
JK> building the package, the other is testing.

Well, what I'm getting out of this is that you're telling me to ignore
what is pretty much a perfect time-saver in the form of existing code
and an existing populated chroot, and instead bring things up from
scratch because there's a philosophical issue with using any of the
"build" infrastructure for anything relating to testing.  I guess I
should never have mentioned rpmlint and just asked for a way to
install a package and run a command inside the chroot via the mock
infrastructure.

> From my point it's all testing, because I'm testing the
> package 
submission by seeing if it builds.  So perhaps that's why I'm just
not
seeing it the same as you.  But OK, I'll just either rip out the bits
of mock I think I can use, or just give up and write my own stuff that
uses the chroot that mock leaves me.  But it sure would have been nice
to get a quick hint and perhaps a little help from someone who knows
the insides of mock.

 - J<

[Jesse Keating]

On Friday 14 July 2006 19:14, Jason L Tibbitts III wrote:
>  >From my point it's all testing, because I'm testing the
> package
>
>  submission by seeing if it builds.  So perhaps that's why I'm just not
>  seeing it the same as you.  But OK, I'll just either rip out the bits
>  of mock I think I can use, or just give up and write my own stuff that
>  uses the chroot that mock leaves me.  But it sure would have been nice
>  to get a quick hint and perhaps a little help from someone who knows
>  the insides of mock. 
You should be testing your package in a local mock rather than abusing the 
buildsystem resources to just test a build.  We deal with this internally 
too.

I'm not saying you couldn't use the mock software as is, in a different 
harness.  Internally we split the majority of testing away from building to 
not slog down our build system with running excessive tests which really 
belong in a QA phase.

We do have some infrastructure to run some tests that are in the 'tests/' 
branch of a package module.  However this is limited and has caused problems 
in our build environment when some packages like to bind to local ports 
(doesn't work too well when you have multiple builds happening at the same 
time, whoops that part is already in use....  However these are for limited 
tests.  I also think that it is strange that we have tests here and also in 
our testing infrastructure.

-- 
Jesse Keating RHCE      (geek.j2solutions.net)
Fedora Legacy Team      (www.fedoralegacy.org)
GPG Public Key          (geek.j2solutions.net/jkeating.j2solutions.pub)

[Jason L Tibbitts III]

> >>>> "JK" == Jesse Keating
> <jkeating(a)j2solutions.net&gt; writes: 
JK> You should be testing your package in a local mock rather than
JK> abusing the buildsystem resources to just test a build.  We deal
JK> with this internally too.

I'm not sure where I gave the impression that this had anything to do
with the buildsystem, but I certainly didn't mean to.  I review lots
of Fedora Extras package submissions; they aren't yet in CVS and can't
possibly be built on the buildsystem machines.  I'm building them
myself, on my builder machines.  The whole discussion is about doing
something extra with mock itself, and has nothing to do with plague or
with the actual Extras buildsys.

It is for the purpose of doing more complete and accurate package
reviews that I'd like to get at the mock chroot, install a couple of
extra packages into it and run an arbitrary command or two.  This is
all running on my machines (or the local machines of the Extras
reviewers, whose lives I would like to make just a bit easier).  FESCo
has charged me with getting more reviews done and improving the
quality of reviews; this is one of the many things I'm looking at.

 - J<

[Christian Iseli]

tibbs(a)math.uh.edu said:
>  The whole discussion is about doing something extra with mock itself,
> and has
>  nothing to do with plague or with the actual Extras buildsys. 
It was probably me who gave that impression.  Sorry about that.

I would actually like the buildsys to run rpmlint on built packages.

But I'll try to ponder options a bit more, and start a new thread when I feel 
ready :-)

Cheers,
					Christian

[Michael Schwendt]

On Fri, 14 Jul 2006 20:52:16 -0500, Jason L Tibbitts III wrote:

>  It is for the purpose of doing more complete and accurate package
>  reviews that I'd like to get at the mock chroot, install a couple of
>  extra packages into it and run an arbitrary command or two.  This is
>  all running on my machines (or the local machines of the Extras
>  reviewers, whose lives I would like to make just a bit easier).  FESCo
>  has charged me with getting more reviews done and improving the
>  quality of reviews; this is one of the many things I'm looking at. 
Sounds more like you want "yum install mach" and continue there.

[David Malcolm]

On Fri, 2006-07-14 at 19:04 -0400, Jesse Keating wrote:
>  On Friday 14 July 2006 18:48, Jason L Tibbitts III wrote:
>  > The whole point is that mock already has everything necessary and
>  > pregenerated.  It has the chroot just sitting there.  It has the
>  > security infrastructure and the procedures for doing things within the
>  > chroot already done.  It seems to me to be terribly foolish to
>  > duplicate that.
>  >
>  > How about this: does mock provide (or could it provide) a way for me
>  > to install a couple of packages and run a command within the chroot?
>  > Or is this relatively simple to do safely without using mock?  Forget
>  > about rpmlint; maybe I need this for debugging or to figure out why
>  > something isn't building.
>  
>  The same software could be used.  Mock can do it, it just isn't designed with 
>  this task in mind.
>  
>  More to the point, there are two distinct tasks here, one is building the 
>  package, the other is testing.  They have some pretty different needs, and 
>  trying to do both of them in the same harness isn't the best of ideas.  
>  rpmlint is just a test, it leads to more tests, like I described.  What about 
>  doing functional tests on things like php packages, where we want to spin up 
>  a local apache, browse to some pages, and make sure the content looks right.  
>  These are the types of things that can be done with dogtail in a real test 
>  environment that really really really shouldn't be done at build time. 
I like
to classify testing into two types: static vs dynamic

When people refer to "static analysis" they're normally thinking of e.g.
a new compiler plugin which applies some semantic rules to the parse
tree and e.g. decides that you're mutex locking is dodgy (or whatever),
which normally involves either a build-time test, or a rebuild.

I like to generalize the term to any kind of test that doesn't require
you to actually run the code (or install it, for that matter).  This can
include the tests that rpmlint already performs, or all kinds of other
goodness (either on the metadata, or perhaps on the files in the package
payload itself).  So this can happen at some arbitrary time after the
package is built - perhaps as an automatic postprocessing step within
the build system (for both Core and Extras I hope).

So, by contrast, tests that require you to actually run the code, I call
"dynamic tests" - be it a functional test of the kind Jesse described,
or a simple "does the app start without coredumping" test; do upgrades
and downgrades work? (exercising the RPM scriptlets) etc

We're working on opening up parts of our dynamic testing framework and
our dynamic tests, as Will Woods has described in other emails.  I'd
love it if we could hook up a static testing framework as well - and I'm
not sure exactly where it would sit (mock, plague, somewhere else).  The
ideas on the wiki page for a pluggable analysis framework look great.

Hope this helps
Dave

[Jason L Tibbitts III]

> >>>> "JLT" == Jason L Tibbitts, <Jason>
> writes: 
JLT> Because I was told that is the only way rpmlint can run certain
JLT> checks.

For an example of a check that only works if you actually install the
package, see:

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=198835#c6

 - J<