Mock users,
I have posted a new version of mock for review and comments.
RPM download: http://linux.dell.com/libsmbios/download/mock/mock-0.8.1/ Git repository: http://linux.dell.com/git/mock-v2.git
Compatibility: -- Mock 0.8.x+ has been written on Fedora 7 and is intended for use on Fedora 7 and higher. A limited amount of testing has been done on FC6, so please report any FC6-specific bugs and we will make best-effort attempts to fix them.
-- I have tried to maintain a high degree of compatibility with mock 0.7 in cmdline args and
New stuff: -- plugin system to better modularize things -- new plugins: -- Yum cache -- root cache -- ccache -- bind mount -- root cache (formerly called autocache) is now significantly smaller in size -- speed increases: mock 0.8 is now in almost every case minutes faster than 0.7, especially for multiple builds. -- uses python logging.py module for increased logging flexibility -- new command: 'mock install PACKAGE' to run a 'yum install PACKAGE' inside the buildroot -- expanded command: 'mock installdeps RPM_FILE' can now install deps for a local binary RPM. Formerlly, only SRPMS were supported. -- new option: "--cleanup-after" that can be used with "--resultdir". This will do a cleanup of the buildroot after the build. This is enabled by default such that any '--resultdir' builds will be automatically cleaned up -- All of the config files delivered with mock have been tested and should work. (The test case compiles mock using each of the config files delivered with mock.)
Changed: -- cmdline requires "rebuild" argument when rebuilding srpms. Previously you could just pass srpm name. -- output has slightly changed. Mock is now slightly more verbose. -- formerly /dev from the host was bind-mounted into the chroot. This is now not enabled by default, but can be configured easily per-chroot using the bind plugin. See the 'defaults.cfg' files for config details. -- '-r CONFIG' option can no longer accept full config filenames ("config.cfg"). Leave off the '.cfg' for mock 0.8+. Formerly, config ('-r' option) could be specified using either "config.cfg" or "config" and mock would look for /etc/mock/config.cfg, automatically adding the '.cfg', if necessary. -- logs are not overwritten or truncated for --no-clean or --resultdir builds.
-- cmdline options removed: --autocache, --rebuild-cache: use "--enable-plugin" or "--disable-plugin" instead --verbose, --debug, --quiet: the log files contain all the information. --verbose could possibly return. --statedir: statedir didnt have much point in life and has gone away
Config files: -- Old config files will, by and large, still work. There are several options in the old config files that are no longer applicable now that 'mock-helper' has went away. The next release of mock should have warnings enabled if it sees that your config file has obsolete options.
Internal Changes: -- now modularized -- mock-helper is gone. Instead there is a setuid-wrapper that calls mock.py. This vastly simplifies development.
Plugin Notes: -- the plugin system is somewhat rudimentary at this time. If this becomes something that is truly useful for out-of-tree plugins or third-parties, we can formalize the interface some more. As of now, the plugin interface should have most of the hooks to do useful stuff, but some other things like the 'conduit' interface that yum uses for plugins is not present. This would formalize the interface a bit more and make sure the plugins dont grope too badly into internals they shouldnt.
Future plans: -- Upstream mock git will be updated to this version next week after we sort out branching for the 0.7 release. -- This version of mock will be checked into Fedora 9 next week. -- yum integration: plan to try to use the yum API rather than the cmdline. this will enable several optimizations and speed ups due to not redownloading the yum metadata multiple times. -- config file format change: at some future point, we probably will switch to an .ini format config file.
SPECIAL SECURITY WARNING:
1) default /usr/bin/mock owner is root:mock with permissions 4550 (-rwsrwx---). Thus, it may only be run by the 'root' user or members of the 'mock' group. DO NOT PUT UNTRUSTED USERS IN THE 'MOCK' GROUP! The current code (as well as all previous versions) has many easily-used mechanisms that an untrusted user could use to elevate their local privileges. IT IS NOT THE GOAL OF THE MOCK DEVELOPERS TO MAKE MOCK A TOOL THAT CAN BE USED BY UNTRUSTED LOCAL USERS.
2) All 'rpmbuild' commands, as well as the 'rpm' command to install and unpack the to-be-built SRPM are run as the user running 'mock' with no elevated privileges. These 'rpm' and 'rpmbuild' commands are run in the chroot environment with these lower privileges. Thus, it should be relatively safe to use mock to compile code from untrusted sources. That being said, MOCK HAS NOT BEEN THOROUGHLY AUDITED TO GUARANTEE THE SAFETY OF THIS.
-- Michael
On Sun, 21 Oct 2007 17:22:58 -0500 Michael E Brown Michael_E_Brown@dell.com wrote:
I have posted a new version of mock for review and comments.
I'm going to wait until we get Fedora 8 out the door before I start looking at this. I'd like to put it through my own paces, which involve doing pungi spins and such inside mock. Getting some time tests with that would be interesting.
On Sun, 2007-10-21 at 17:22 -0500, Michael E Brown wrote:
Mock users,
I have posted a new version of mock for review and comments.
RPM download: http://linux.dell.com/libsmbios/download/mock/mock-0.8.1/ Git repository: http://linux.dell.com/git/mock-v2.git
These changes look like a great start. You should look at integrating the 'cost' feature from yum 3.2.6 and above a bit more, I think. It'll let you prefer a specific repo over another one for the same pkg nevra. Handy for having your just-built-but-same-version repo be pulled from instead of an older, remote repo.
Also handy for a local-cache to drag from.
-sv
On Mon, 22 Oct 2007 02:15:36 +0000 seth vidal skvidal@fedoraproject.org wrote:
These changes look like a great start. You should look at integrating the 'cost' feature from yum 3.2.6 and above a bit more, I think. It'll let you prefer a specific repo over another one for the same pkg nevra. Handy for having your just-built-but-same-version repo be pulled from instead of an older, remote repo.
Also handy for a local-cache to drag from.
THis is a big must. We absolutely need this before we enable the koji static-repos by default. Otherwise quite frequently all packages will be pulled from the static-repos over a costly data line instead of from a local mirror or on disk cache.
On Oct 21, 2007, at 3:22 PM, Michael E Brown wrote:
Mock users,
I have posted a new version of mock for review and comments.
RPM download: http://linux.dell.com/libsmbios/download/mock/ mock-0.8.1/ Git repository: http://linux.dell.com/git/mock-v2.git
Hi Michael, I think this is great -- thanks a lot for all the work you've put into the update.
I've tried it out a bit on both EL5 and FC6 and I haven't been able to get it working at all. On both systems, a mock -r <root> rebuild <srpm> gives me the following error: Error: Command(/usr/bin/yum --installroot /var/lib/mock/fedora-7-i386/ root install buildsys-build) failed. See logs for output.
The yum error that shows up in root.log is: 2007-11-12 19:49:38,210 - trace_decorator.py:21:DEBUG: ENTER: do(('/ usr/bin/yum --installroot /var/lib/mock/fedora-7-i386/root install buildsys-build', None, 0, True, 0), {}) 2007-11-12 19:49:38,211 - util.py:156:DEBUG: Run cmd: /usr/bin/yum -- installroot /var/lib/mock/fedora-7-i386/root install buildsys-build 2007-11-12 19:49:38,211 - util.py:163:DEBUG: Executing timeout(0): / usr/bin/yum --installroot /var/lib/mock/fedora-7-i386/root install buildsys-build 2007-11-12 19:49:38,569 - util.py:181:DEBUG: Loading "installonlyn" plugin 2007-11-12 19:49:38,569 - util.py:181:DEBUG: Setting up Install Process 2007-11-12 19:49:38,569 - util.py:181:DEBUG: Setting up repositories 2007-11-12 19:49:38,570 - util.py:181:DEBUG: No Repositories Available to Set Up 2007-11-12 19:49:38,570 - util.py:181:DEBUG: Reading repository metadata in from local files 2007-11-12 19:49:38,570 - util.py:181:DEBUG: Parsing package install arguments 2007-11-12 19:49:38,570 - util.py:181:DEBUG: Traceback (most recent call last): 2007-11-12 19:49:38,570 - util.py:181:DEBUG: File "/usr/bin/yum", line 29, in ? 2007-11-12 19:49:38,571 - util.py:181:DEBUG: yummain.main(sys.argv[1:]) 2007-11-12 19:49:38,571 - util.py:181:DEBUG: File "/usr/share/yum- cli/yummain.py", line 94, in main 2007-11-12 19:49:38,571 - util.py:181:DEBUG: result, resultmsgs = base.doCommands() 2007-11-12 19:49:38,571 - util.py:181:DEBUG: File "/usr/share/yum- cli/cli.py", line 381, in doCommands 2007-11-12 19:49:38,571 - util.py:181:DEBUG: return self.yum_cli_commands[self.basecmd].doCommand(self, self.basecmd, self.extcmds) 2007-11-12 19:49:38,572 - util.py:181:DEBUG: File "/usr/share/yum- cli/yumcommands.py", line 134, in doCommand 2007-11-12 19:49:38,572 - util.py:181:DEBUG: return base.installPkgs(extcmds) 2007-11-12 19:49:38,572 - util.py:181:DEBUG: File "/usr/share/yum- cli/cli.py", line 553, in installPkgs 2007-11-12 19:49:38,572 - util.py:181:DEBUG: exactmatch, matched, unmatched = \ 2007-11-12 19:49:38,572 - util.py:181:DEBUG: File "/usr/lib/ python2.4/site-packages/yum/packageSack.py", line 360, in matchPackageNames 2007-11-12 19:49:38,573 - util.py:181:DEBUG: unmatched = list(unmatched) 2007-11-12 19:49:38,573 - util.py:181:DEBUG: TypeError: iteration over non-sequence 2007-11-12 19:49:38,573 - trace_decorator.py:32:DEBUG: EXCEPTION: Command(/usr/bin/yum --installroot /var/lib/mock/fedora-7-i386/root install buildsys-build) failed. See logs for output.
I get the same error when using different mock roots. Am I just doing something completely wrong? For the fc6 box: $ rpm -q yum yum-3.0.6-1.fc6 $ rpm -q mock mock-0.8.1-1.fc6
The mock configs are unchanged from those included in the RPM. Which actually brings me to my 2nd "problem" in that the old mock rpm used to mark /etc/mock/*cfg as config(noreplace) IIRC, and now it only claims ownership of the directory. The end result seems to be that the new package completely blows away all custom configs that were in / etc/mock. Was this done deliberately due to config file changes?
Thanks, Jeff
On Mon, Nov 12, 2007 at 08:02:13PM -0800, Jeff Sheltren wrote:
On Oct 21, 2007, at 3:22 PM, Michael E Brown wrote:
Mock users,
I have posted a new version of mock for review and comments.
RPM download: http://linux.dell.com/libsmbios/download/mock/ mock-0.8.1/
Ah. I forgot that I had made a 'test' release of mock and put it up there. Can you try the latest 0.8.7 release?
Git repository: http://linux.dell.com/git/mock-v2.git
I dont always keep this git repo up-to-date unless I'm actually posting something for review on the mailing list. I think our mirror scripts got stuck and this hasnt been resynced in a week or two.
Hi Michael, I think this is great -- thanks a lot for all the work you've put into the update.
I've tried it out a bit on both EL5 and FC6 and I haven't been able to get it working at all. On both systems, a mock -r <root> rebuild <srpm> gives me the following error: Error: Command(/usr/bin/yum --installroot /var/lib/mock/fedora-7-i386/ root install buildsys-build) failed. See logs for output.
The yum error that shows up in root.log is: 2007-11-12 19:49:38,210 - trace_decorator.py:21:DEBUG: ENTER: do(('/ usr/bin/yum --installroot /var/lib/mock/fedora-7-i386/root install buildsys-build', None, 0, True, 0), {}) 2007-11-12 19:49:38,211 - util.py:156:DEBUG: Run cmd: /usr/bin/yum -- installroot /var/lib/mock/fedora-7-i386/root install buildsys-build 2007-11-12 19:49:38,211 - util.py:163:DEBUG: Executing timeout(0): / usr/bin/yum --installroot /var/lib/mock/fedora-7-i386/root install buildsys-build 2007-11-12 19:49:38,569 - util.py:181:DEBUG: Loading "installonlyn" plugin 2007-11-12 19:49:38,569 - util.py:181:DEBUG: Setting up Install Process 2007-11-12 19:49:38,569 - util.py:181:DEBUG: Setting up repositories 2007-11-12 19:49:38,570 - util.py:181:DEBUG: No Repositories Available to Set Up 2007-11-12 19:49:38,570 - util.py:181:DEBUG: Reading repository metadata in from local files
These are messages from yum...
Can you check /var/lib/mock/fedora-7-i386/root/etc/yum.conf to see that it looks 'sane'?
Do you have direct internet access?
2007-11-12 19:49:38,570 - util.py:181:DEBUG: Parsing package install arguments 2007-11-12 19:49:38,570 - util.py:181:DEBUG: Traceback (most recent call last): 2007-11-12 19:49:38,570 - util.py:181:DEBUG: File "/usr/bin/yum", line 29, in ? 2007-11-12 19:49:38,571 - util.py:181:DEBUG: yummain.main(sys.argv[1:]) 2007-11-12 19:49:38,571 - util.py:181:DEBUG: File "/usr/share/yum- cli/yummain.py", line 94, in main 2007-11-12 19:49:38,571 - util.py:181:DEBUG: result, resultmsgs = base.doCommands() 2007-11-12 19:49:38,571 - util.py:181:DEBUG: File "/usr/share/yum- cli/cli.py", line 381, in doCommands 2007-11-12 19:49:38,571 - util.py:181:DEBUG: return self.yum_cli_commands[self.basecmd].doCommand(self, self.basecmd, self.extcmds) 2007-11-12 19:49:38,572 - util.py:181:DEBUG: File "/usr/share/yum- cli/yumcommands.py", line 134, in doCommand 2007-11-12 19:49:38,572 - util.py:181:DEBUG: return base.installPkgs(extcmds) 2007-11-12 19:49:38,572 - util.py:181:DEBUG: File "/usr/share/yum- cli/cli.py", line 553, in installPkgs 2007-11-12 19:49:38,572 - util.py:181:DEBUG: exactmatch, matched, unmatched = \ 2007-11-12 19:49:38,572 - util.py:181:DEBUG: File "/usr/lib/ python2.4/site-packages/yum/packageSack.py", line 360, in matchPackageNames 2007-11-12 19:49:38,573 - util.py:181:DEBUG: unmatched = list(unmatched) 2007-11-12 19:49:38,573 - util.py:181:DEBUG: TypeError: iteration over
This is all a yum traceback. Your yum is having some problems...
non-sequence 2007-11-12 19:49:38,573 - trace_decorator.py:32:DEBUG: EXCEPTION: Command(/usr/bin/yum --installroot /var/lib/mock/fedora-7-i386/root install buildsys-build) failed. See logs for output.
I get the same error when using different mock roots. Am I just doing something completely wrong? For the fc6 box: $ rpm -q yum yum-3.0.6-1.fc6 $ rpm -q mock mock-0.8.1-1.fc6
Can you please try 0.8.7, which is the latest released version? Take the srpm and use mock on a F7/F8 machine to build an RPM for RHEL5 using the shipped config file. I'm not sure if I released 0.8.1 to fedora or not...
I have two build machines running FC6, and I have tested mock there and it works ok. There is one known 'issue' that appears to me to be cosmetic (traceback on exit) but otherwise it functions ok.
The mock configs are unchanged from those included in the RPM. Which actually brings me to my 2nd "problem" in that the old mock rpm used to mark /etc/mock/*cfg as config(noreplace) IIRC, and now it only claims ownership of the directory. The end result seems to be that the new package completely blows away all custom configs that were in / etc/mock. Was this done deliberately due to config file changes?
??
%files ... %config(noreplace) %{_sysconfdir}/%{name}/*.cfg %config(noreplace) %{_sysconfdir}/%{name}/*.ini
This hasnt changed, so they should all still be config(noreplace).
So, before I make a release, I run a release test. It is doc/releasetest.sh in the git repo. It basically rebuilds mock using each config shipped with mock. This ensures that I ship only working config files and that mock works with each config.
-- Michael
On Nov 13, 2007, at 8:06 AM, Michael E Brown wrote:
Can you please try 0.8.7, which is the latest released version? Take the srpm and use mock on a F7/F8 machine to build an RPM for RHEL5 using the shipped config file. I'm not sure if I released 0.8.1 to fedora or not...
Yep, sorry for the noise, I didn't realize I was using an older version. After rebuilding 0.8.7 for EL5, it mostly works, but I have noticed on bug so far. In my configs, I have a mock repo defined using file:///some/path which is an NFS mounted directory (handled by automount). Now, if that is not currently mounted and I run mock, then yum in mock gives an error that the repodata isn't found for that repo. At that point I can run the 'mount' command, see that automount has mounted the directory, and then if I re-run mock it will complete the setup/build successfully.
Since this appears to be yum reporting the problem and not mock I'm not sure what the old mock was doing that somehow I never had this problem before.
Doing an 'ls' on the directory to get it auto-mounted before running mock also avoids the problem.
Aside from that issue (and the logging messages you mentioned), things seem to be working fine so far...
Thanks, Jeff
On Tue, Nov 13, 2007 at 03:12:50PM -0800, Jeff Sheltren wrote:
On Nov 13, 2007, at 8:06 AM, Michael E Brown wrote:
Can you please try 0.8.7, which is the latest released version? Take the srpm and use mock on a F7/F8 machine to build an RPM for RHEL5 using the shipped config file. I'm not sure if I released 0.8.1 to fedora or not...
Yep, sorry for the noise, I didn't realize I was using an older version. After rebuilding 0.8.7 for EL5, it mostly works, but I have noticed on bug so far. In my configs, I have a mock repo defined using file:///some/path which is an NFS mounted directory (handled by automount). Now, if that is not currently mounted and I run mock, then yum in mock gives an error that the repodata isn't found for that repo. At that point I can run the 'mount' command, see that automount has mounted the directory, and then if I re-run mock it will complete the setup/build successfully.
This is because mock now clone()'s itself and creates a new namespace before exe()-ing the mock.py code. Your automounter is not sharing the new namespace that mock creates. You can fix this by running:
# mount --make-rshared /
You can put it in your /etc/rc.local.
I was not able to find out the syntax to put this into /etc/fstab, which is where I think it goes.
-- Michael
Michael E Brown wrote:
On Tue, Nov 13, 2007 at 03:12:50PM -0800, Jeff Sheltren wrote:
Yep, sorry for the noise, I didn't realize I was using an older version. After rebuilding 0.8.7 for EL5, it mostly works, but I have noticed on bug so far. In my configs, I have a mock repo defined using file:///some/path which is an NFS mounted directory (handled by automount). Now, if that is not currently mounted and I run mock, then yum in mock gives an error that the repodata isn't found for that repo. At that point I can run the 'mount' command, see that automount has mounted the directory, and then if I re-run mock it will complete the setup/build successfully.
This is because mock now clone()'s itself and creates a new namespace before exe()-ing the mock.py code. Your automounter is not sharing the new namespace that mock creates. You can fix this by running:
# mount --make-rshared /
You can put it in your /etc/rc.local.
I was not able to find out the syntax to put this into /etc/fstab, which is where I think it goes.
The problem I'm seeing is that a mock instance runs, it accesses file:///data/sw1/... which is automounted for that process, but then I cannot access /data/sw1 for any other process until the first mock quits. mount --make-rshared / doesn't seem to help.
Hi!
I've been working on fixing Plague 0.4.4.1 (and Plague in cvs) for Fedora 8 (and 7). The following test package works for me so far:
http://mschwendt.fedorapeople.org/plague/plague-0.4.4.1-5.fc8.20071114.src.r...
For those, who test the cvs version of Plague 0.5.0 (tickets in bugzilla say so), there are at least two patches necessary to make it work on F7+F8 and with mock >= 0.8:
http://mschwendt.fedorapeople.org/plague-0.5.0-fedora8.patch http://mschwendt.fedorapeople.org/plague-cvs-mock-0.8.patch
On Wed, Nov 14, 2007 at 10:42:09PM +0100, Michael Schwendt wrote:
Hi!
I've been working on fixing Plague 0.4.4.1 (and Plague in cvs) for Fedora 8 (and 7). The following test package works for me so far:
http://mschwendt.fedorapeople.org/plague/plague-0.4.4.1-5.fc8.20071114.src.r...
For those, who test the cvs version of Plague 0.5.0 (tickets in bugzilla say so), there are at least two patches necessary to make it work on F7+F8 and with mock >= 0.8:
http://mschwendt.fedorapeople.org/plague-0.5.0-fedora8.patch http://mschwendt.fedorapeople.org/plague-cvs-mock-0.8.patch
Do we need to have a more-stable interface between mock and plague?
I set up plague sometime in the distant, distant past and actually hacked on it a bit. From what I remember, it had to read the state file because the old version of mock could sometimes exit before it had dropped the init state, or that mock could exit too quickly in the build state. (I dont remember all the details...)
I think I have resolved most of those issues with mock 0.8.x. You should be able to rely on the mock return code at least to tell you if it succeeded or failed.
Using the mock plugin interface, we could probably very easily write a plague interface that is stable if you need more information about state changes. That would be much more sustainable and reliable than parsing stdout. -- Michael
On Thu, 15 Nov 2007 11:31:17 -0600, Michael E Brown wrote:
For those, who test the cvs version of Plague 0.5.0 (tickets in bugzilla say so), there are at least two patches necessary to make it work on F7+F8 and with mock >= 0.8:
http://mschwendt.fedorapeople.org/plague-0.5.0-fedora8.patch http://mschwendt.fedorapeople.org/plague-cvs-mock-0.8.patch
Do we need to have a more-stable interface between mock and plague?
Dropping command-line args and changing status-file contents certainly is something that ought to be avoided after the gold release of the distribution. Fedora 7, on the other hand, broke Plague already with Python 2.5 and SQLite 3. With a working QA team, this would have been communicated/announced somewhere prior to F7.
I set up plague sometime in the distant, distant past and actually hacked on it a bit. From what I remember, it had to read the state file because the old version of mock could sometimes exit before it had dropped the init state, or that mock could exit too quickly in the build state. (I dont remember all the details...)
Yes, sounds right. There's code in there that marks a build as failed despite a zero exit code returned by mock. Surprisingly, a few months ago (in September?) the old work-around broke the FE buildsys frequently because it read the state file not quickly/often enough and misinterpreted mock's exit. (most recent mock is not installed everywhere!)
Nowadays, reading the state file is mostly for eye-candy, to display "prepping" instead of "building" for the long time it can take to complete a build root with remote repository downloads. The mock exit code is evaluated, and deleting a few more lines of code would get rid of what is left from the old work-around.
I think I have resolved most of those issues with mock 0.8.x. You should be able to rely on the mock return code at least to tell you if it succeeded or failed.
See other reply in this thread. All that matters to me is to keep a pair of plague+mock working for e.g. CentOS4 or 5.
On Wed, 2007-11-14 at 22:42 +0100, Michael Schwendt wrote:
Hi!
I've been working on fixing Plague 0.4.4.1 (and Plague in cvs) for Fedora 8 (and 7). The following test package works for me so far:
http://mschwendt.fedorapeople.org/plague/plague-0.4.4.1-5.fc8.20071114.src.r...
Do you want to own plague? I don't really want to, and somebody with the ability to devote more attention to it probably should.
Dan
For those, who test the cvs version of Plague 0.5.0 (tickets in bugzilla say so), there are at least two patches necessary to make it work on F7+F8 and with mock >= 0.8:
http://mschwendt.fedorapeople.org/plague-0.5.0-fedora8.patch http://mschwendt.fedorapeople.org/plague-cvs-mock-0.8.patch
On Thu, 15 Nov 2007 14:12:28 -0500, Dan Williams wrote:
On Wed, 2007-11-14 at 22:42 +0100, Michael Schwendt wrote:
Hi!
I've been working on fixing Plague 0.4.4.1 (and Plague in cvs) for Fedora 8 (and 7). The following test package works for me so far:
http://mschwendt.fedorapeople.org/plague/plague-0.4.4.1-5.fc8.20071114.src.r...
Do you want to own plague? I don't really want to, and somebody with the ability to devote more attention to it probably should.
My only interest has been in keeping it working, because it's still used for EPEL and Livna/RPMFusion, and first it seemed as if python2.5 and sqlite3 break the code badly. I haven't spent enough time on examining all the changes in 0.5.0 in cvs, except for adding fixes.
Until a few days ago I didn't know about the tickets in bugzilla and the problems on F7+F8, because my only Plague instance is on F6.
Early December Fedora Extras will be EOL'ed, however, and with it my direct relationship with Plague will end, too. How long the Extras buildsys will still be used for EPEL, I don't know.
Time is better spent on tools and looking into recent changes in bodhi.
buildsys@lists.fedoraproject.org