Weird problem building zsh in local mock but not in koji

[Roland McGrath]

> Unfortunately that level of magic is mostly beyond me.

That's OK.  We can translate for you.

> RM> The things to look at are whether the tests are using a
> RM> temporary pty
> 
> I do not really know how to determine that.  

The TTY column in ps is a start.

> However, it simplifies things to know that if there's any difference it's
> not in the spec but in the environment in which mock is called.

IMHO it is the fault of the package's test suite that it cares the ambient
environment where 'make check' is run.  If it's going to be affected by the
tty state of the caller of "make", then it should run those tests inside a
temporary pty (i.e. use expect or script or something).

However, I also think that we should endeavor to make the multiple
recommended ways of building Fedora rpms (i.e. "use koji" and "run mock
yourself") run their builds in a consistent environment across all the
recommended methods.

> RM> and what 'ps j' says about all the processes involved
> RM> in the build/test (including mock and its layers of whatever before
> RM> the rpmbuild).
> 
> Here's the forest from px axjf (sorry for long lines):
> 
   PPID   PID  PGID   SID TTY      TPGID STAT   UID   TIME COMMAND
> 23843 10479 10479 10479 ?           -1 SNs      0   0:00  \_ sshd: tibbs [priv]
> 10479 10486 10479 10479 ?           -1 SN    7225   0:00  |   \_ sshd: tibbs at pts/3
> 10486 10487 10487 10487 pts/3    10527 SNs   7225   0:00  |       \_ -zsh
> 10487 10527 10527 10487 pts/3    10527 SN+   7225   0:00  |           \_ /bin/bash /home/tibbs/bin/dobuild /home/tibbs/work/extras-cvs/zsh/devel/zsh-4.3.10-4.fc13.src.rpm
> 10527 10530 10527 10487 pts/3    10527 SN+   7225   0:02  |               \_ /usr/bin/python -tt /usr/sbin/mock -r fedora-rawhide-x86_64 -v --rebuild /home/tibbs/work/extras-cvs/zsh/devel/zsh-4.3.10-4.fc13.src.rpm

This shows mock running on your normal tty (TTY matches your command-line
shell) in the foreground (PGID==TPGID means "foreground", and PGID!=TPGID,
means "background"), as you would expect.

> 10530 13230 13230 10487 pts/3    10527 TN    7225   0:00  |               |   \_ rpmbuild -bb --target x86_64 --nodeps builddir/build/SPECS/zsh.spec

This shows mock runs rpmbuild in the same session and in the background.
This is the same is if you had done:

	$ rpmbuild -bb --target x86_64 --nodeps builddir/build/SPECS/zsh.spec
	^Z (before the test suite)
	[1]+  Stopped                 rpmbuild -bb --target x86_64 --nodeps builddir/build/SPECS/zsh.spec
	$ bg
	[1]+ rpmbuild -bb --target x86_64 --nodeps builddir/build/SPECS/zsh.spec &
	$ 

I think if you try it that way you will see the same problem.  I suspect
that if you run it in the foreground, you won't have any problem.

   PPID   PID  PGID   SID TTY      TPGID STAT   UID   TIME COMMAND
> 13230 28410 13230 10487 pts/3    10527 TN    7225   0:00  |               |       \_ /bin/sh -e /var/tmp/rpm-tmp.5VZZeU
> 28410 28414 13230 10487 pts/3    10527 TN    7225   0:00  |               |           \_ make test
> 28414 28415 13230 10487 pts/3    10527 TN    7225   0:00  |               |               \_ /bin/sh -c cd Test ; make check
> 28415 28416 13230 10487 pts/3    10527 TN    7225   0:00  |               |                   \_ make check
> 28416 28505 13230 10487 pts/3    10527 TN    7225   0:00  |               |                       \_ /bin/sh -c if ZTST_testlist="`for f in ./*.ztst; \?           do echo $f; done`" \? ZTST_srcdir="." \? ZTST_exe=../Src/zsh \? ../Src/zsh +Z -f ./runtests.zsh; then \? sta
> 28505 28507 13230 10487 pts/3    10527 TN    7225   0:00  |               |                           \_ ../Src/zsh +Z -f ./runtests.zsh
> 28507 29888 13230 10487 pts/3    10527 TN    7225   0:00  |               |                               \_ ../Src/zsh +Z -f ./ztst.zsh ./B04read.ztst

This all shows that rpmbuild on down did no job control fiddling, so it's
all in the rpmbuild "job" (PGID == PID of rpmbuild).

> Is any of that remotely helpful?  On the buildsys, a mock build (not for
> zsh, but they should all start the same) looks sort of like this:

Yes, this is the information we needed to explain what you are seeing.

   PPID   PID  PGID   SID TTY      TPGID STAT   UID   TIME COMMAND
>     1  3056  3055  3055 ?           -1 S        0 391:44 /usr/bin/python /usr/sbin/kojid --force-lock --verbose
>  3056 17091 17091  3055 ?           -1 S        0   0:07  \_ /usr/bin/python /usr/sbin/kojid --force-lock --verbose
> 17091 17272 17091  3055 ?           -1 S      101   0:01      \_ /usr/bin/python -tt /usr/sbin/mock -r koji/dist-f13-build-653289-93821 --no-clean --target x86_64 --rebuild /mnt/koji/work/tasks/8377/1828377/kvirc-4.0.0-0.19.rc1.fc13.src.rpm
> 17272 25386 25386  3055 ?           -1 S      101   0:00          \_ rpmbuild -bb --target x86_64 --nodeps builddir/build/SPECS/kvirc.spec
> 25386 25413 25386  3055 ?           -1 S      101   0:00              \_ /bin/sh -e /var/tmp/rpm-tmp.vzHree
> 25413 25917 25386  3055 ?           -1 S      101   0:00                  \_ make -j4

This shows that kojid runs mock in an orphan session (SID == PID of
somebody dead) with no controlling terminal (TTY/TPGID shows ?/-1).  From
mock on down behaves the same, i.e. putting rpmbuild in its own process
group.  But with no tty, this behaves differently than being in a
background process group.

Off hand I think that mock should run rpmbuild either in a temporary
pty/session or with no tty at all.  The no tty option requires mock to
catch the SIGHUP when its own tty goes away and know to go kill the
rpmbuild that won't get its own SIGHUP, or else being in the middle of a
mock command when your session dies (ssh drop, modem drop, terminal window
killed, etc.) will leave the build still running.

The temporary pty option seems better to me off hand.  Then the whole
rpmbuild would be in the foreground on that pty, so the bonehead packages
like zsh won't be upset.


Thanks,
Roland