mock improvements
by Michael E Brown
So here is the list of things that have been requested lately and I'll
be working on a few of these over the next few weeks. If anybody has any
input, I'd take it. As I start on each, I'll most likely email the
mailing list with the outline of what I'm doing.
If anybody has existing patches for these (against current mock git),
all the better... :)
1) more reliable mount/umount
several people have pointed out instances where mock exits leaving
mounts behind (specifically /dev), and the next invokation of mock
ends up 'rm -rf' the host machine's /dev. Bad....
2) caching yum downloads
several people have commented that the autocache stuff is great for
speeding up builds, others say that it can sometimes be bad for
reproducability, and that simply saving the yum cache dir would be
better.
3) ccache integration
This is a new one that I havent seen before, but should significantly
speed up builds for people who often do
rebuild-the-entire-distribution-type things. I'm told by some that
this is bad for reproducability, but good for speeding up builds when
you are just sanity checking or when that small reproducability hit
doesnt matter. I've also seen lots of empirical data that ccache
should not cause any problems. This will be have to be specifically
enabled through a commandline or config file option, so those who care
can turn it on/off.
4) distcc integration
Pretty much the same case as ccache. Has more things that need thought
than the ccache case, above, though.
--
Michael
16 years, 2 months
RE: problems with orphanskill feature
by Michael E Brown
Do you want to do the patch or me? It will be tomorrow morning before I can get to it. (or very late tonight.)
--
Michael
-----Original Message-----
From: Clark Williams [mailto:clark.williams@gmail.com]
Sent: Thu 9/27/2007 4:11 PM
To: Brown, Michael E
Cc: Jesse Keating; fedora-buildsys-list(a)redhat.com
Subject: Re: problems with orphanskill feature
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Michael E Brown wrote:
> On Thu, Sep 27, 2007 at 01:30:15PM -0500, Clark Williams wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Michael/Jesse,
>>
>> We're seeing odd failures with mock and the orphanskill feature. What seems to happen
>> is that the '; mock-helper orphanskill <rootdir>' string is tacked onto a command
>> which is passed to do_chroot() and after the main command is run, an attempt is made
>> to run mock-helper (which is not installed in the chroot). So people are seeing a
>> "File not found" message after a successful command.
>>
>> Now it looks like to me that part of the reason for an orphanskill was that the do()
>> routine might hang until all the child processes are done, so I'm loathe to just run
>> the orphanskill after the do_chroot() is finished (I suspect twisty lines of logic,
>> all alike). Seems like we can do a couple of things:
>>
>> 1. Copy mock-helper into each chroot, so it's available for orphanskill
>> 2. Back out the orphanskill logic and try again
>>
>> Option #1 is somewhat easy, if kinda ugly (not sure I like the idea of scattering a
>> setuid-root program into all our build roots). Option #2 requires that we look at the
>> code in all the do_* and do() routines to make sure that orphanskill runs when we
>> need it to. Ideally I'd like to insure that orphanskill runs *outside* the chroot and
>> that it's not needed to keep self.do() from hanging.
>>
>> What you guys think?
>
> How about we just run two commands in a row? I see the comment but don't
> really see why. Line 973, we dont need to run orphanskill if it isnt
> chroot. For the my.do_chroot() on line 975, it looks like we could just
> do a my.do_chroot() followed by a normal os.system().
That was my first thought, but I was concerned that we might be missing something
subtle in the timeout code (hence my email to you :)).
If you think we can just run the orphanskill stuff after running the do_chroot() then
I say that's the way to go.
>
> The problem it is trying to fix is if the rpmbuild process spawns child
> processes that fork and never exit. I believe this was seen in some code
> that was running in the rpmbuild as a unit test?
>
> We should also be cc-ing fedora-buildsys-list. (done)
wups (looks shamefaced)
>
> I also understand Jesse's sentiment to just back it out. If it is going
> to take more than a day or two to fix, we could just back it out.
Let's try running it right after the rpmbuild. If that doesn't work right, then we
can just comment it out while we take a closer look at it.
Clark
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
iD8DBQFG/BxuqA4JVb61b9cRAkAIAJ0cusVhhetlhnNJA5RWTkcD4cHG5wCcD5Rb
aQo57BZBD1ME0FLDj6nARq4=
=itpW
-----END PGP SIGNATURE-----
16 years, 2 months
Re: problems with orphanskill feature
by Michael E Brown
On Thu, Sep 27, 2007 at 01:30:15PM -0500, Clark Williams wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Michael/Jesse,
>
> We're seeing odd failures with mock and the orphanskill feature. What seems to happen
> is that the '; mock-helper orphanskill <rootdir>' string is tacked onto a command
> which is passed to do_chroot() and after the main command is run, an attempt is made
> to run mock-helper (which is not installed in the chroot). So people are seeing a
> "File not found" message after a successful command.
>
> Now it looks like to me that part of the reason for an orphanskill was that the do()
> routine might hang until all the child processes are done, so I'm loathe to just run
> the orphanskill after the do_chroot() is finished (I suspect twisty lines of logic,
> all alike). Seems like we can do a couple of things:
>
> 1. Copy mock-helper into each chroot, so it's available for orphanskill
> 2. Back out the orphanskill logic and try again
>
> Option #1 is somewhat easy, if kinda ugly (not sure I like the idea of scattering a
> setuid-root program into all our build roots). Option #2 requires that we look at the
> code in all the do_* and do() routines to make sure that orphanskill runs when we
> need it to. Ideally I'd like to insure that orphanskill runs *outside* the chroot and
> that it's not needed to keep self.do() from hanging.
>
> What you guys think?
How about we just run two commands in a row? I see the comment but don't
really see why. Line 973, we dont need to run orphanskill if it isnt
chroot. For the my.do_chroot() on line 975, it looks like we could just
do a my.do_chroot() followed by a normal os.system().
The problem it is trying to fix is if the rpmbuild process spawns child
processes that fork and never exit. I believe this was seen in some code
that was running in the rpmbuild as a unit test?
We should also be cc-ing fedora-buildsys-list. (done)
I also understand Jesse's sentiment to just back it out. If it is going
to take more than a day or two to fix, we could just back it out.
--
Michael
16 years, 2 months
koji initialization, docs
by aberoham＠gmail.com
I have latest trunk version of koji, kojid, kojira, etc setup and running.
But now what? Does any administration documentation exist? To figure out how
to put gas in this engine do I really have to read the source?
Thanks.
16 years, 2 months
[PATCH koji] added koji-helper setuid program
by Enrico Scholz
This patch adds a 'koji-helper' setuid program which implements the
following methods:
* koji-helper rmrf <dir>
removes everything under <dir>, inclusive <dir>. It does not cross
filesystem borders
* koji-helper rmtree <dir>
removes everything under <dir>, but not <dir> itself. It does not cross
filesystem borders
Methods above are implemented to replace the python 'safe_rmtree()' method
which was never safe, nor will work when 'kojid' is running as non-root.
Signed-off-by: Enrico Scholz <enrico.scholz(a)informatik.tu-chemnitz.de>
---
Makefile | 15 ++-
builder/kojid | 53 +++--------
koji.spec | 3 -
src/koji-helper.c | 260 +++++++++++++++++++++++++++++++++++++++++++++++++++++
4 files changed, 286 insertions(+), 45 deletions(-)
diff --git a/Makefile b/Makefile
index cd88d4b..2a239fd 100644
--- a/Makefile
+++ b/Makefile
@@ -2,6 +2,8 @@ NAME=koji
SPECFILE = $(firstword $(wildcard *.spec))
SUBDIRS = hub builder koji cli docs util www
+sbindir = /usr/sbin
+
ifdef DIST
DIST_DEFINES := --define "dist $(DIST)"
endif
@@ -52,11 +54,14 @@ ifndef TAG
TAG=$(NAME)-$(VERSION)-$(RELEASE)
endif
-_default:
- @echo "read the makefile"
+all: src/koji-helper
+
+src/koji-helper: src/koji-helper.c
+ $(CC) $(CFLAGS) $< -o $@
clean:
rm -f *.o *.so *.pyc *~ koji*.bz2 koji*.src.rpm
+ rm -f src/koji-helper
rm -rf koji-$(VERSION)
for d in $(SUBDIRS); do make -s -C $$d clean; done
@@ -100,14 +105,16 @@ force-tag::
# @$(MAKE) tag TAG_OPTS="-F $(TAG_OPTS)"
DESTDIR ?= /
-install:
+install: all
@if [ "$(DESTDIR)" = "" ]; then \
echo " "; \
echo "ERROR: A destdir is required"; \
exit 1; \
fi
- mkdir -p $(DESTDIR)
+ mkdir -p $(DESTDIR)$(sbindir)
+
+ install -p -m0710 src/koji-helper $(DESTDIR)$(sbindir)
for d in $(SUBDIRS); do make DESTDIR=`cd $(DESTDIR); pwd` \
-C $$d install; [ $$? = 0 ] || exit 1; done
diff --git a/builder/kojid b/builder/kojid
index 6e973fe..5940954 100755
--- a/builder/kojid
+++ b/builder/kojid
@@ -150,35 +150,17 @@ def log_output(path, args, outfile, uploadpath, cwd=None, logerror=0, append=0,
outfd.close()
return status[1]
-def safe_rmtree(path, unmount=False, strict=True):
+def safe_rmtree(path, strict=True, op='rmrf'):
logger = logging.getLogger("koji.build")
- #safe remove: with -xdev the find cmd will not cross filesystems
- # (though it will cross bind mounts from the same filesystem)
- if not os.path.exists(path):
- logger.debug("No such path: %s" % path)
- return
- if unmount:
- umount_all(path)
- #first rm -f non-directories
- logger.debug('Scrubbing files in %s' % path)
- rv = os.system("find '%s' -xdev \\! -type d -print0 |xargs -0 rm -f" % path)
- msg = 'file removal failed (code %r) for %s' % (rv,path)
- if rv != 0:
- logger.warn(msg)
- if strict:
- raise koji.GenericError, msg
- else:
- return rv
- #them rmdir directories
- #with -depth, we start at the bottom and work up
- logger.debug('Scrubbing directories in %s' % path)
- rv = os.system("find '%s' -xdev -depth -type d -print0 |xargs -0 rmdir" % path)
- msg = 'dir removal failed (code %r) for %s' % (rv,path)
- if rv != 0:
- logger.warn(msg)
- if strict:
- raise koji.GenericError, msg
- return rv
+ rc = os.spawnvp(os.P_WAIT, '/usr/sbin/koji-helper', ['/usr/sbin/koji-helper', op, path])
+ if rc!=0:
+ msg = "directory removal failed (code %r) for %s" % (rc,path)
+ logger.warn(msg)
+ if strict:
+ raise koji.GenericError, msg
+ else:
+ return rc
+ return rc
def umount_all(topdir):
"Unmount every mount under topdir"
@@ -635,7 +617,7 @@ class TaskManager(object):
if age > 3600*24:
#dir untouched for a day
self.logger.info("Removing buildroot: %s" % desc)
- if topdir and safe_rmtree(topdir, unmount=True, strict=False) != 0:
+ if topdir and safe_rmtree(topdir, strict=False) != 0:
continue
#also remove the config
try:
@@ -644,15 +626,7 @@ class TaskManager(object):
self.logger.warn("%s: can't remove config: %s" % (desc, e))
elif age > 120:
if rootdir:
- try:
- flist = os.listdir(rootdir)
- except OSError, e:
- self.logger.warn("%s: can't list rootdir: %s" % (desc, e))
- continue
- if flist:
- self.logger.info("%s: clearing rootdir" % desc)
- for fn in flist:
- safe_rmtree("%s/%s" % (rootdir,fn), unmount=True, strict=False)
+ safe_rmtree(rootdir, strict=False, op='rmtree')
else:
self.logger.debug("Recent buildroot: %s: %i seconds" % (desc,age))
self.logger.debug("Local buildroots: %d" % len(local_br))
@@ -1211,8 +1185,7 @@ class BaseTaskHandler(object):
def removeWorkdir(self):
if self.workdir is None:
return
- safe_rmtree(self.workdir, unmount=False, strict=True)
- #os.spawnvp(os.P_WAIT, 'rm', ['rm', '-rf', self.workdir])
+ os.spawnvp(os.P_WAIT, 'rm', ['rm', '-rf', self.workdir])
def wait(self, subtasks=None, all=False, failany=False):
"""Wait on subtasks
diff --git a/koji.spec b/koji.spec
index 13d3bf0..f14bb6e 100644
--- a/koji.spec
+++ b/koji.spec
@@ -16,7 +16,6 @@ Group: Applications/System
URL: http://hosted.fedoraproject.org/projects/koji
Source: %{name}-%{PACKAGE_VERSION}.tar.bz2
BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n)
-BuildArch: noarch
Requires: python-krbV >= 1.0.13
Requires: rpm-python
Requires: pyOpenSSL
@@ -89,6 +88,7 @@ koji-web is a web UI to the Koji system.
%setup -q
%build
+make CFLAGS="$CFLAGS" CC="%__cc" all
%install
rm -rf $RPM_BUILD_ROOT
@@ -125,6 +125,7 @@ rm -rf $RPM_BUILD_ROOT
%files builder
%defattr(-,root,root)
+%attr(4710,root,kojibuilder) %_sbindir/koji-helper
%{_sbindir}/kojid
%{_initrddir}/kojid
%config(noreplace) %{_sysconfdir}/sysconfig/kojid
diff --git a/src/koji-helper.c b/src/koji-helper.c
new file mode 100644
index 0000000..a3d0921
--- /dev/null
+++ b/src/koji-helper.c
@@ -0,0 +1,260 @@
+/* --*- c -*--
+ * Copyright (C) 2007 Enrico Scholz <enrico.scholz(a)informatik.tu-chemnitz.de>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; version 3 of the License.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#define _GNU_SOURCE
+
+#ifndef MOCK_ROOT
+# define MOCK_ROOT "/var/lib/mock"
+#endif
+
+#include <sys/stat.h>
+#include <string.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <errno.h>
+#include <unistd.h>
+#include <dirent.h>
+#include <fcntl.h>
+#include <stdbool.h>
+
+static int __attribute__((__nonnull__(1, 2)))
+safe_chdir(char const *path, struct stat const *exp_st)
+{
+ struct stat cur_st;
+
+ if (strchr(path, '/')) {
+ fprintf(stderr, "safe_chdir(): invalid char in path '%s'\n", path);
+ return -1;
+ }
+
+ if (strcmp(path, "..")==0) {
+ fprintf(stderr, "safe_chdir(): parent dir referred\n");
+ return -1;
+ }
+
+ if (chdir(path) < 0) {
+ fprintf(stderr, "chdir(%s): %s\n", path, strerror(errno));
+ return -1;
+ }
+
+ if (stat(".", &cur_st) < 0) {
+ fprintf(stderr, "stat(%s): %s\n", path, strerror(errno));
+ return -2;
+ }
+
+ if (cur_st.st_dev != exp_st->st_dev ||
+ cur_st.st_ino != exp_st->st_ino) {
+ fprintf(stderr, "RACE: path '%s' changed before chdir()\n", path);
+ return -2;
+ }
+
+ return 0;
+}
+
+static int
+rmrf_cwd(struct stat *cwd_st)
+{
+ DIR *cwd = opendir(".");
+ int rc = -1;
+
+ if (!cwd) {
+ perror("opendir()");
+ return -1;
+ }
+
+ for (;;) {
+ struct dirent *ent = readdir(cwd);
+ struct stat st;
+
+ if (!ent)
+ break;
+
+ if (ent->d_name[0] == '.' &&
+ (ent->d_name[1] == '\0'||
+ (ent->d_name[1] == '.' && ent->d_name[2] == '\0')))
+ continue; /* skip '.' and '..' */
+
+ if (lstat(ent->d_name, &st) < 0) {
+ fprintf(stderr, "rmrf_cwd: lstat(%s): %s\n",
+ ent->d_name, strerror(errno));
+ continue;
+ }
+
+ if (cwd_st && cwd_st->st_dev != st.st_dev)
+ continue; /* do not cross devices */
+ else if (S_ISDIR(st.st_mode)) {
+ switch (safe_chdir(ent->d_name, &st)) {
+ case -1: continue;
+ case -2: break;
+ default: rmrf_cwd(&st); break;
+ }
+
+ if (fchdir(dirfd(cwd)) < 0) {
+ perror("rmrf_cwd: fchdir()");
+ goto err;
+ }
+
+ if (rmdir(ent->d_name) < 0) {
+ fprintf(stderr, "rmrf_cwd: rmdir(%s): %s\n",
+ ent->d_name, strerror(errno));
+ continue;
+ }
+ } else if (unlink(ent->d_name) < 0) {
+ fprintf(stderr, "rmrf_cwd: unlink(%s): %s\n",
+ ent->d_name, strerror(errno));
+ continue;
+ }
+ }
+
+ rc = 0;
+err:
+ closedir(cwd);
+ return rc;
+}
+
+static int
+safe_chdir_subpath(char const *path_c, size_t path_len)
+{
+ char path[path_len+1];
+ char *ptr = path;
+ int rc = 0;
+
+ if (path_len == 0)
+ return 0;
+
+ strncpy(path, path_c, path_len);
+ path[path_len] = '\0';
+
+ while (ptr) {
+ char *new_ptr = strsep(&ptr, "/");
+ struct stat st;
+
+ if (*new_ptr == '\0')
+ continue; /* skip empty path components
+ * (e.g. double /) */
+
+ if (lstat(new_ptr, &st) < 0) {
+ fprintf(stderr, "stat(%s): %s\n",
+ new_ptr, strerror(errno));
+ rc = -1;
+ } else if (!S_ISDIR(st.st_mode) || S_ISLNK(st.st_mode)) {
+ fprintf(stderr, "safe_chdir_subpath(): invalid mode of '%s': %04x\n",
+ new_ptr, st.st_mode);
+ rc = -1;
+ } else
+ rc = safe_chdir(new_ptr, &st);
+
+ if (rc < 0)
+ break;
+ }
+
+ return rc;
+}
+
+/* Usage: rmrf <dir> */
+static int
+do_rmrf(int argc, char *argv[], bool remove_parent_dir)
+{
+ char const *dir;
+ size_t dir_len;
+ char const *last_path;
+ int parent_fd;
+
+ if (argc != 2) {
+ fprintf(stderr, "wrong number of parameters for 'rmrf' operation\n");
+ return EXIT_FAILURE;
+ }
+
+ dir = argv[1];
+
+ /* strip leading MOCK_ROOT; it's a little bit hacky but required to
+ * keep backward compatibility */
+ if (strncmp(dir, MOCK_ROOT, sizeof(MOCK_ROOT)-1) == 0)
+ dir += sizeof(MOCK_ROOT)-1;
+
+ while (*dir == '/')
+ ++dir; /* strip leading '/' */
+
+ dir_len = strlen(dir);
+ while (dir_len>0 && dir[dir_len-1] == '/')
+ --dir_len; /* strip trailing '/' */
+
+ last_path = dir + dir_len;
+ while (last_path > dir && last_path[-1] != '/')
+ --last_path;
+
+ if (dir_len == 0) {
+ fprintf(stderr, "do_rmrf(): empty path\n");
+ return EXIT_FAILURE;
+ }
+ if (dir_len > 255) {
+ fprintf(stderr, "pathname too long\n");
+ return EXIT_FAILURE;
+ }
+
+
+ /* real work begins here... */
+
+ if (chdir(MOCK_ROOT) < 0) {
+ perror("chdir(<MOCK_ROOT>)");
+ return EXIT_FAILURE;
+ }
+
+ if (last_path > dir && /* else, it would be a noop */
+ safe_chdir_subpath(dir, last_path - dir) < 0)
+ return EXIT_FAILURE;
+
+ parent_fd = open(".", O_RDONLY|O_DIRECTORY);
+ if (parent_fd < 0) {
+ perror("open(<MOCK_ROOT>)");
+ return EXIT_FAILURE;
+ }
+
+ if (safe_chdir_subpath(last_path, dir+dir_len - last_path + 1) < 0)
+ return EXIT_FAILURE;
+
+ /* we are now *in* the given path */
+ if (rmrf_cwd(NULL) < 0)
+ return EXIT_FAILURE;
+
+ if (fchdir(parent_fd) < 0) {
+ perror("fchdir(<parent>)");
+ return EXIT_FAILURE;
+ }
+
+ if (remove_parent_dir &&
+ rmdir(last_path) < 0)
+ return EXIT_FAILURE;
+
+ return EXIT_SUCCESS;
+}
+
+int main(int argc, char *argv[])
+{
+ if (argc < 2) {
+ fprintf(stderr, "not enough parameters\n");
+ return EXIT_FAILURE;
+ }
+
+ if (strcmp(argv[1], "rmrf") == 0)
+ return do_rmrf(argc-1, argv+1, true);
+ else if (strcmp(argv[1], "rmtree") == 0)
+ return do_rmrf(argc-1, argv+1, false);
+ else
+ fprintf(stderr, "unknown argument\n");
+
+ return EXIT_FAILURE;
+}
16 years, 2 months
[PATCH koji] Added support for building from git repositories
by Enrico Scholz
This patch adds support for building from git:// respositories. It is used
like
koji call build 'git://<REPOSITORY>?<PACKAGE>#<TAG>' <TARGET>
The repository specified above must have the following layout:
<BASE_DIR>/<REPOSITORY>
|- common/
| `- .git/
|- ...
|- <PACKAGE>
| `- .git/
|- ...
This means that every package is hold in an own git repository. There must
be a 'common' repository which contains files required for building the
srpm.
There is no explicit support for branches; the patch just checks out the
given tag and builds from it. Perhaps it can be enhanced to check whether
branch-point is some child of the tag...
TODO: the git and cvs handlers share some common, non trivial code. This
should be generalized.
TODO: use a separate user account for the 'git' and 'make srpm' operations;
currently they are executed under the uid 'kojid' is running (which is
'root). Hence, a '%(/sbin/killall5)' in the spec file can bring down
the box.
Signed-off-by: Enrico Scholz <enrico.scholz(a)informatik.tu-chemnitz.de>
---
builder/kojid | 130 +++++++++++++++++++++++++++++++++++++++++++-
cli/koji | 9 ++-
koji.spec | 2 -
koji/__init__.py | 11 ++++
www/kojiweb/taskinfo.chtml | 7 ++
www/kojiweb/tasks.chtml | 1
6 files changed, 153 insertions(+), 7 deletions(-)
diff --git a/builder/kojid b/builder/kojid
index 5940954..bbb9774 100755
--- a/builder/kojid
+++ b/builder/kojid
@@ -1396,7 +1396,7 @@ class ChainBuildTask(BaseTaskHandler):
subtasks = []
build_tasks = []
for src in build_level:
- if src.startswith('cvs://'):
+ if src.startswith('cvs://') or src.startswith('git://'):
task_id = session.host.subtask(method='build',
arglist=[src, target, opts],
parent=self.id)
@@ -1437,8 +1437,10 @@ class BuildTask(BaseTaskHandler):
raise koji.BuildError, "arch_override is only allowed for scratch builds"
task_info = session.getTaskInfo(self.id)
# only allow admins to perform non-scratch builds from srpm
- if not src.startswith('cvs://') and not opts.get('scratch') \
- and not 'admin' in session.getUserPerms(task_info['owner']):
+ if not src.startswith('cvs://') and \
+ not src.startswith('git://') and \
+ not opts.get('scratch') and \
+ not 'admin' in session.getUserPerms(task_info['owner']):
raise koji.BuildError, "only admins may peform non-scratch builds from srpm"
target_info = session.getBuildTarget(target)
if not target_info:
@@ -1496,6 +1498,8 @@ class BuildTask(BaseTaskHandler):
if isinstance(src,str):
if src.startswith('cvs://'):
return self.getSRPMFromCVS(src)
+ if src.startswith('git://'):
+ return self.getSRPMFromGit(src)
else:
#assume this is a path under uploads
return src
@@ -1514,6 +1518,17 @@ class BuildTask(BaseTaskHandler):
srpm = result['srpm']
return srpm
+ def getSRPMFromGit(self, url):
+ #TODO - allow different ways to get the srpm
+ task_id = session.host.subtask(method='buildSRPMFromGit',
+ arglist=[url],
+ label='srpm',
+ parent=self.id)
+ # wait for subtask to finish
+ result = self.wait(task_id)[task_id]
+ srpm = result['srpm']
+ return srpm
+
def readSRPMHeader(self, srpm):
#srpm arg should be a path relative to <BASEDIR>/work
global options
@@ -1767,6 +1782,115 @@ class TagBuildTask(BaseTaskHandler):
exctype, value = sys.exc_info()[:2]
session.host.tagNotification(False, tag_id, fromtag, build_id, user_id, ignore_success, "%s: %s" % (exctype, value))
raise e
+
+
+class BuildSRPMFromGitTask(BaseTaskHandler):
+
+ Methods = ['buildSRPMFromGit']
+ _taskWeight = 0.75
+
+ def spec_sanity_checks(self, filename):
+ spec = open(filename).read()
+ for tag in ("Packager", "Distribution", "Vendor"):
+ if re.match("%s:" % tag, spec, re.M):
+ raise koji.BuildError, "%s is not allowed to be set in spec file" % tag
+ for tag in ("packager", "distribution", "vendor"):
+ if re.match("%%define\s+%s\s+" % tag, spec, re.M):
+ raise koji.BuildError, "%s is not allowed to be defined in spec file" % tag
+
+ def handler(self,url):
+ if not url.startswith('git://'):
+ raise koji.BuildError("invalid git URL: %s" % url)
+
+ # Hack it because it refuses to parse it properly otherwise
+ scheme, netloc, path, params, query, fragment = urlparse.urlparse('http'+url[3:])
+ if not (netloc and path and fragment and query):
+ raise koji.BuildError("invalid git URL: %s" % url)
+
+ # Steps:
+ # 1. GIT clone into tempdir
+ # 3. Run 'make srpm'
+
+ gitdir = self.workdir + '/git'
+ koji.ensuredir(gitdir)
+ logfile = self.workdir + "/srpm.log"
+ uploadpath = self.getUploadDir()
+ sourcedir = '%s/%s' % (gitdir, query)
+
+ cmd = ['git', 'clone', 'git://%s%s/%s' % (netloc, path, query), query]
+ if log_output(cmd[0], cmd, logfile, uploadpath, cwd=gitdir, logerror=1, append=1):
+ output = "(none)"
+ try:
+ output = open(logfile).read()
+ except IOError:
+ pass
+ raise koji.BuildError, "Error with clone 'git://%s%s/%s: %s" % (netloc, path, query, output)
+
+ cmd = ['git', 'clone', 'git://%s%s/common' % (netloc, path)]
+ self.logger.debug("executing: %s" % cmd)
+ if log_output(cmd[0], cmd, logfile, uploadpath, cwd=gitdir, logerror=1, append=1):
+ output = "(none)"
+ try:
+ output = open(logfile).read()
+ except IOError:
+ pass
+ raise koji.BuildError, "Error with clone 'git://%s%s/common: %s" % (netloc, path, output)
+
+ cmd = ['git', 'ls-files', '-t', '-s']
+ log_output(cmd[0], cmd, logfile, uploadpath, cwd='%s/common' % gitdir, logerror=1, append=1)
+
+ cmd = ['git', 'checkout', '-b', 'kojibuild', fragment]
+ self.logger.debug("executing: %s" % cmd)
+ if log_output(cmd[0], cmd, logfile, uploadpath, cwd=sourcedir, logerror=1, append=1):
+ output = "(none)"
+ try:
+ output = open(logfile).read()
+ except IOError:
+ pass
+ raise koji.BuildError, "Error with checking out tag '%s' from git://%s%s/%s: %s" % (fragment, netloc, path, query, output)
+
+ cmd = ['git', 'ls-files', '-t', '-s']
+ log_output(cmd[0], cmd, logfile, uploadpath, cwd=sourcedir, logerror=1, append=1)
+
+ spec_files = glob.glob("%s/*.spec" % sourcedir)
+ if len(spec_files) == 0:
+ raise koji.BuildError("No spec file found")
+ elif len(spec_files) > 1:
+ raise koji.BuildError("Multiple spec files found: %s" % spec_files)
+ spec_file = spec_files[0]
+
+ # Run spec file sanity checks. Any failures will throw a BuildError
+ self.spec_sanity_checks(spec_file)
+
+ #build srpm
+ cmd = ['make', '-C', sourcedir, 'srpm', '_KOJI=1.2.2', '_KOJI_TAG=%s' % fragment]
+ self.logger.debug("executing: %s" % cmd)
+ if log_output(cmd[0], cmd, logfile, uploadpath, cwd=gitdir, logerror=1, append=1):
+ raise koji.BuildError, "Error building SRPM"
+
+ srpms = glob.glob('%s/*.src.rpm' % sourcedir)
+ if len(srpms) == 0:
+ raise koji.BuildError, "No srpms found in %s" % sourcedir
+ elif len(srpms) > 1:
+ raise koji.BuildError, "Multiple srpms found in %s: %s" % (sourcedir, ", ".join(srpms))
+ else:
+ srpm = srpms[0]
+
+ # check srpm name
+ h = koji.get_rpm_header(srpm)
+ name = h[rpm.RPMTAG_NAME]
+ version = h[rpm.RPMTAG_VERSION]
+ release = h[rpm.RPMTAG_RELEASE]
+ srpm_name = "%(name)s-%(version)s-%(release)s.src.rpm" % locals()
+ if srpm_name != os.path.basename(srpm):
+ raise koji.BuildError, 'srpm name mismatch: %s != %s' % (srpm_name, os.path.basename(srpm))
+
+ #upload srpm and return
+ self.uploadFile(srpm)
+ return {
+ 'srpm' : "%s/%s" % (uploadpath, srpm_name),
+ 'log' : "%s/srpm.log" % uploadpath,
+ }
class BuildSRPMFromCVSTask(BaseTaskHandler):
diff --git a/cli/koji b/cli/koji
index 718b6e5..4f05500 100755
--- a/cli/koji
+++ b/cli/koji
@@ -669,7 +669,7 @@ def handle_build(options, session, args):
if build_opts.background:
#relative to koji.PRIO_DEFAULT
priority = 5
- if not source.startswith('cvs://'):
+ if not (source.startswith('cvs://') or source.startswith('git://')):
# only allow admins to perform non-scratch builds from srpm
if not opts['scratch'] and not session.hasPerm('admin'):
parser.error(_("builds from srpm must use --scratch"))
@@ -741,7 +741,7 @@ def handle_chain_build(options, session, args):
if build_level:
src_list.append(build_level)
build_level = []
- elif src.startswith('cvs://'):
+ elif src.startswith('cvs://') or src.startswith('git://'):
build_level.append(src)
elif '/' not in src and len(src.split('-')) >= 3:
# quick check that it looks like a N-V-R
@@ -2432,8 +2432,13 @@ def _parseTaskParams(session, method, task_id):
if method == 'buildFromCVS':
lines.append("CVS URL: %s" % params[0])
lines.append("Build Target: %s" % params[1])
+ elif method == 'buildFromGit':
+ lines.append("GIT URL: %s" % params[0])
+ lines.append("Build Target: %s" % params[1])
elif method == 'buildSRPMFromCVS':
lines.append("CVS URL: %s" % params[0])
+ elif method == 'buildSRPMFromGit':
+ lines.append("GIT URL: %s" % params[0])
elif method == 'multiArchBuild':
lines.append("SRPM: %s/work/%s" % (options.topdir, params[0]))
lines.append("Build Target: %s" % params[1])
diff --git a/koji.spec b/koji.spec
index f14bb6e..7d4d643 100644
--- a/koji.spec
+++ b/koji.spec
@@ -49,7 +49,7 @@ Requires(post): /sbin/service
Requires(preun): /sbin/chkconfig
Requires(preun): /sbin/service
Requires(pre): /usr/sbin/useradd
-Requires: cvs
+Requires: cvs git-core make
Requires: rpm-build
Requires: redhat-rpm-config
Requires: createrepo >= 0.4.10
diff --git a/koji/__init__.py b/koji/__init__.py
index d808126..5789220 100644
--- a/koji/__init__.py
+++ b/koji/__init__.py
@@ -1517,6 +1517,11 @@ def taskLabel(taskInfo):
if source.startswith('cvs://'):
source = source[source.rfind('/') + 1:]
source = source.replace('#', ':')
+ elif source.startswith('git://'):
+ source = source[source.rfind('?') + 1:]
+ tmp = source[:source.find('#')]
+ source = source[source.find('#') + 1:]
+ source = '%s:%s' % (tmp, source[source.rfind('/') + 1:])
else:
source = os.path.basename(source)
extra = '%s, %s' % (target, source)
@@ -1526,6 +1531,12 @@ def taskLabel(taskInfo):
url = url[url.rfind('/') + 1:]
url = url.replace('#', ':')
extra = url
+ elif method == 'buildSRPMFromGit':
+ if taskInfo.has_key('request'):
+ url = taskInfo['request'][0]
+ url = url[url.rfind('/') + 1:]
+ url = url.replace('#', ':')
+ extra = url
elif method == 'buildArch':
if taskInfo.has_key('request'):
srpm, tagID, arch = taskInfo['request'][:3]
diff --git a/www/kojiweb/taskinfo.chtml b/www/kojiweb/taskinfo.chtml
index 76df4bc..dd5cf72 100644
--- a/www/kojiweb/taskinfo.chtml
+++ b/www/kojiweb/taskinfo.chtml
@@ -67,6 +67,11 @@
<strong>Build Target:</strong> <a href="buildtargetinfo?name=$params[1]">$params[1]</a>
#elif $task.method == 'buildSRPMFromCVS'
<strong>CVS URL:</strong> $params[0]
+ #elif $task.method == 'buildFromGit'
+ <strong>GIT URL:</strong> $params[0]<br/>
+ <strong>Build Target:</strong> <a href="buildtargetinfo?name=$params[1]">$params[1]</a>
+ #elif $task.method == 'buildSRPMFromGit'
+ <strong>GIT URL:</strong> $params[0]
#elif $task.method == 'multiArchBuild'
<strong>SRPM:</strong> $params[0]<br/>
<strong>Build Target:</strong> <a href="buildtargetinfo?name=$params[1]">$params[1]</a><br/>
@@ -276,7 +281,7 @@ $cgi.escape($result.faultString.strip())
<a href="getfile?taskID=$task.id&name=$urllib.quote($filename)">$filename</a><br/>
#end for
#if $task.state not in ($koji.TASK_STATES.CLOSED, $koji.TASK_STATES.CANCELED, $koji.TASK_STATES.FAILED) and \
- $task.method in ('buildSRPMFromCVS', 'buildArch', 'createrepo')
+ $task.method in ('buildSRPMFromCVS', 'buildSRPMFromGit', 'buildArch', 'createrepo')
<br/>
<a href="watchlogs?taskID=$task.id">Watch logs</a>
#end if
diff --git a/www/kojiweb/tasks.chtml b/www/kojiweb/tasks.chtml
index 70de06c..32ca6ad 100644
--- a/www/kojiweb/tasks.chtml
+++ b/www/kojiweb/tasks.chtml
@@ -104,6 +104,7 @@ All
<option value="all" #if $method == 'all' then 'selected="selected"' else ''#>all</option>
<option value="build" #if $method == 'build' then 'selected="selected"' else ''#>build</option>
<option value="buildSRPMFromCVS" #if $method == 'buildSRPMFromCVS' then 'selected="selected"' else ''#>buildSRPMFromCVS</option>
+ <option value="buildSRPMFromGit" #if $method == 'buildSRPMFromGit' then 'selected="selected"' else ''#>buildSRPMFromGit</option>
<option value="buildArch" #if $method == 'buildArch' then 'selected="selected"' else ''#>buildArch</option>
<option value="buildNotification" #if $method == 'buildNotification' then 'selected="selected"' else ''#>buildNotification</option>
<option value="tagBuild" #if $method == 'tagBuild' then 'selected="selected"' else ''#>tagBuild</option>
16 years, 2 months
No such file or directory: u'/var/tmp/yum-root-O3By9L/anaconda/headers/ifd-egate-0.05-17.i386.hdr'
by Sergio Monteiro Basto
Hi,
I use updated fedora 7
what I could do ?
pungi -c /etc/pungi/f7-everything.i386 stops width :
INFO:pypungi.pungi:Running /usr/lib/anaconda-runtime/pkgorder /srv/pungi/f7/7/Everything/i386/os i386 Fedora
ERROR:pypungi.pungi:Got an error from /usr/lib/anaconda-runtime/pkgorder
ERROR:pypungi.pungi:Traceback (most recent call last):
File "/usr/lib/anaconda-runtime/pkgorder", line 169, in <module>
addGroups(ds, ["core", "base", "text-internet"])
File "/usr/lib/anaconda-runtime/pkgorder", line 105, in addGroups
processTransaction(ds)
File "/usr/lib/anaconda-runtime/pkgorder", line 77, in processTransaction
ds.populateTs(keepold=0)
File "/usr/lib/python2.5/site-packages/yum/depsolve.py", line 201, in populateTs
self.downloadHeader(txmbr.po)
File "/usr/lib/python2.5/site-packages/yum/__init__.py", line 863, in downloadHeader
cache=repo.http_caching != 'none',
File "/usr/lib/python2.5/site-packages/yum/yumRepo.py", line 599, in getHeader
cache=cache,
File "/usr/lib/python2.5/site-packages/yum/yumRepo.py", line 567, in _getFile
http_headers=headers,
File "/usr/lib/python2.5/site-packages/urlgrabber/mirror.py", line 411, in urlgrab
return self._mirror_try(func, url, kw)
File "/usr/lib/python2.5/site-packages/urlgrabber/mirror.py", line 397, in _mirror_try
return func_ref( *(fullurl,), **kwargs )
File "/usr/lib/python2.5/site-packages/urlgrabber/grabber.py", line 927, in urlgrab
return self._retry(opts, retryfunc, url, filename)
File "/usr/lib/python2.5/site-packages/urlgrabber/grabber.py", line 845, in _retry
r = apply(func, (opts,) + args, {})
File "/usr/lib/python2.5/site-packages/urlgrabber/grabber.py", line 915, in retryfunc
fo._do_grab()
File "/usr/lib/python2.5/site-packages/urlgrabber/grabber.py", line 1196, in _do_grab
else: new_fo = open(self.filename, 'wb')
IOError: [Errno 2] No such file or directory: u'/var/tmp/yum-root-O3By9L/anaconda/headers/ifd-egate-0.05-17.i386.hdr'
Thanks,
--
Sérgio M. B.
16 years, 2 months
Koji
by Oscar Bacho
Hi list
koji is a great software, but
which are case use for koji?
in the college, we are trying to create a distribution based on Fedora.
Att
Oscar Bacho
Instituto Tecnologico de Acapulco
16 years, 3 months
[patch 3/3] Add a --prioritize option
by Mark McLoughlin
In the absence of kickstart support for specifying repo priorities,
perhaps the best we can do is make the order of repos imply their
priority.
This patch adds a --prioritize command line option which causes
pungi to assign priorities to each repo based on the order they
were supplied.
Signed-off-by: Mark McLoughlin <markmc(a)redhat.com>
Index: pungi/pypungi/gather.py
===================================================================
--- pungi.orig/pypungi/gather.py
+++ pungi/pypungi/gather.py
@@ -47,7 +47,7 @@ class PungiYum(yum.YumBase):
pass
class Gather(pypungi.PungiBase):
- def __init__(self, config, ksparser):
+ def __init__(self, config, ksparser, prioritize = False):
pypungi.PungiBase.__init__(self, config)
# Set our own logging name space
@@ -119,6 +119,12 @@ class Gather(pypungi.PungiBase):
thisrepo.enablegroups = True
if not repo.priority is None:
thisrepo.priority = repo.priority
+ elif prioritize:
+ if hasattr(thisrepo, "priority"):
+ thisrepo.priority = len(self.ayum.repos.repos)
+ else:
+ self.logger.warning('The yum-priorities plugin is needed to support prioritizing repos')
+ prioritize = False
self.ayum.repos.add(thisrepo)
self.ayum.repos.enableRepo(thisrepo.id)
self.ayum._getRepos(thisrepo=thisrepo.id, doSetup = True)
Index: pungi/pungi
===================================================================
--- pungi.orig/pungi
+++ pungi/pungi
@@ -62,7 +62,7 @@ def main():
# Actually do work.
if not opts.sourceisos:
if opts.do_all or opts.do_gather:
- mygather = pypungi.gather.Gather(config, ksparser)
+ mygather = pypungi.gather.Gather(config, ksparser, opts.prioritize)
mygather.getPackageObjects()
mygather.downloadPackages()
mygather.makeCompsFile()
@@ -143,6 +143,8 @@ if __name__ == '__main__':
help='disable gathering of source packages (optional)')
parser.add_option("--sourceisos", default=False, action="store_true", dest="sourceisos",
help='Create the source isos (other arch runs must be done)')
+ parser.add_option("--prioritize", default=False, action="store_true", dest="prioritize",
+ help='Prioritize yum repositories in the order they are specified')
parser.add_option("-c", "--config", dest="config",
help='Path to kickstart config file')
--
16 years, 3 months
[patch 2/3] Support repository priorities
by Mark McLoughlin
Support yum repository priorities supplied through kickstart.
Signed-off-by: Mark McLoughlin <markmc(a)redhat.com>
Index: pungi/pypungi/gather.py
===================================================================
--- pungi.orig/pypungi/gather.py
+++ pungi/pypungi/gather.py
@@ -117,6 +117,8 @@ class Gather(pypungi.PungiBase):
self.logger.info('URL for repo %s is %s' % (thisrepo.name, thisrepo.baseurl))
thisrepo.basecachedir = self.ayum.conf.cachedir
thisrepo.enablegroups = True
+ if not repo.priority is None:
+ thisrepo.priority = repo.priority
self.ayum.repos.add(thisrepo)
self.ayum.repos.enableRepo(thisrepo.id)
self.ayum._getRepos(thisrepo=thisrepo.id, doSetup = True)
--
16 years, 3 months
