Sorry for the previous email, it got sent while editing
In koji-cli in handle_chain_build: elif '://' in src: # quick check that src might be a url build_level.append(src) elif '/' not in src and not src.endswith('.rpm') and len(src.split('-')) >= 3: # quick check that it looks like a N-V-R build_level.append(src)
should it be: elif '://' in src: # quick check that src might be a url build_level.append(src) elif '/' not in src and src.endswith('.rpm') and len(src.split('-')) >= 3: # quick check that it looks like a N-V-R build_level.append(src)
I attempted to koji chain-build a couple of srpms for testing, but task is just sitting on "waitrepo". Also, I noticed it doesn't upload srpms so what is the BKM? do you import, then chain-build?
On 02/26/2010 11:27 AM, Doug Reiland wrote:
Sorry for the previous email, it got sent while editing
In koji-cli in handle_chain_build: elif '://' in src: # quick check that src might be a url build_level.append(src) elif '/' not in src and not src.endswith('.rpm') and len(src.split('-'))>= 3: # quick check that it looks like a N-V-R build_level.append(src)
should it be: elif '://' in src: # quick check that src might be a url build_level.append(src) elif '/' not in src and src.endswith('.rpm') and len(src.split('-'))>= 3: # quick check that it looks like a N-V-R build_level.append(src)
Chain build is not designed to handle building from srpms -- only from scm. For non-admins, building from srpm can only be used with --scratch (and hence is useless for chaining since scratch builds cannot be used in repos).
Note the error message under the else: '"%s" is not a SCM URL or package N-V-R' These are the only valid entries.
You can emulate chain-build with a series of build and wait-repo commands if you like.
I attempted to koji chain-build a couple of srpms for testing, but task is just sitting on "waitrepo". Also, I noticed it doesn't upload srpms so what is the BKM? do you import, then chain-build?
Suddenly, after several years of koji use, a number of folks seem to have developed deep misconceptions about the import command.
The import command is an administrative command. It is used to import externally built rpms. The idea of importing an srpm in order to build it is contradictory. Both activities create a build with the same NVR, hence they are conflicting.
The primary use of the import command is bootstrapping. You use it to get a base set of builds into your koji instance in order get builds going.
On 02/26/2010 02:15 PM, Doug Reiland wrote:
You can emulate chain-build with a series of build and wait-repo commands if you like.
I want to be able to improve build throughput.
Is the following valid approach?
koji build --skip-tag mytag srpm1 srpm2 .... after done koji tag-pkg mytag pkg1 pkg2 .... koji regen-repo ...
I think you're assuming that a repo-regen will block other builds from taking place. That's not the case. A build will use the most recently created valid repo for the tag. Other builds can go on while a repo is being regenerated. When the repo regen is complete, it will be marked as the new latest repo, and any builds submitted after that will use the new repo instead of the old. Using --skip-tag doesn't improve throughput at all over using chain-build or wait-repo in between (dependent) builds.
ok, I just seem to be waiting for createrepo alot. Making very slow progress in package builds.
On Fri, Feb 26, 2010 at 2:27 PM, Mike Bonnet mikeb@redhat.com wrote:
On 02/26/2010 02:15 PM, Doug Reiland wrote:
You can emulate chain-build with a series of build and wait-repo commands if you like.
I want to be able to improve build throughput.
Is the following valid approach?
koji build --skip-tag mytag srpm1 srpm2 .... after done koji tag-pkg mytag pkg1 pkg2 .... koji regen-repo ...
I think you're assuming that a repo-regen will block other builds from taking place. That's not the case. A build will use the most recently created valid repo for the tag. Other builds can go on while a repo is being regenerated. When the repo regen is complete, it will be marked as the new latest repo, and any builds submitted after that will use the new repo instead of the old. Using --skip-tag doesn't improve throughput at all over using chain-build or wait-repo in between (dependent) builds. -- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys
On 02/26/2010 03:14 PM, Doug Reiland wrote:
ok, I just seem to be waiting for createrepo alot. Making very slow progress in package builds.
Is that because you only have one builder, and its busy processing newRepo tasks instead of build tasks? The simple answer is to add more builders.
The only time you should be waiting for repo regens is if your package depends on a previously-built package. In that case chain-build or scripting up something using wait-repo is the best option.
On Fri, Feb 26, 2010 at 2:27 PM, Mike Bonnet mikeb@redhat.com wrote:
On 02/26/2010 02:15 PM, Doug Reiland wrote:
You can emulate chain-build with a series of build and wait-repo commands if you like.
I want to be able to improve build throughput.
Is the following valid approach?
koji build --skip-tag mytag srpm1 srpm2 .... after done koji tag-pkg mytag pkg1 pkg2 .... koji regen-repo ...
I think you're assuming that a repo-regen will block other builds from taking place. That's not the case. A build will use the most recently created valid repo for the tag. Other builds can go on while a repo is being regenerated. When the repo regen is complete, it will be marked as the new latest repo, and any builds submitted after that will use the new repo instead of the old. Using --skip-tag doesn't improve throughput at all over using chain-build or wait-repo in between (dependent) builds. -- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys
-- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys
All I have is srpms. Is at least one createrepo builder required?
If I remove my builder from createrepo channel until a bunch of builds are done, ...
I don't under how wait-repo would work here. Or how you would script this. Does/would it assume package dependency knowledge and just plan the builds in a order that reduces dependencies?
I know, I am clueless.
On Fri, Feb 26, 2010 at 3:28 PM, Mike Bonnet mikeb@redhat.com wrote:
On 02/26/2010 03:14 PM, Doug Reiland wrote:
ok, I just seem to be waiting for createrepo alot. Making very slow progress in package builds.
Is that because you only have one builder, and its busy processing newRepo tasks instead of build tasks? The simple answer is to add more builders.
The only time you should be waiting for repo regens is if your package depends on a previously-built package. In that case chain-build or scripting up something using wait-repo is the best option.
On Fri, Feb 26, 2010 at 2:27 PM, Mike Bonnet mikeb@redhat.com wrote:
On 02/26/2010 02:15 PM, Doug Reiland wrote:
You can emulate chain-build with a series of build and wait-repo commands if you like.
I want to be able to improve build throughput.
Is the following valid approach?
koji build --skip-tag mytag srpm1 srpm2 .... after done koji tag-pkg mytag pkg1 pkg2 .... koji regen-repo ...
I think you're assuming that a repo-regen will block other builds from taking place. That's not the case. A build will use the most recently created valid repo for the tag. Other builds can go on while a repo is being regenerated. When the repo regen is complete, it will be marked as the new latest repo, and any builds submitted after that will use the new repo instead of the old. Using --skip-tag doesn't improve throughput at all over using chain-build or wait-repo in between (dependent) builds. -- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys
-- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys
-- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys
On 02/26/2010 04:06 PM, Doug Reiland wrote:
All I have is srpms. Is at least one createrepo builder required?
If I remove my builder from createrepo channel until a bunch of builds are done, ...
Then no createrepo tasks would be processed until all your builds are done. Is all of this because of a lack of hardware? The easiest thing would be to add more builders to your koji setup.
I don't under how wait-repo would work here. Or how you would script this. Does/would it assume package dependency knowledge and just plan the builds in a order that reduces dependencies?
Yes, if package bar depends on package foo then you could do something like:
koji build my-target foo-1.0-1.src.rpm && \ koji wait-repo --target my-target --build foo-1.0-1 && \ koji build my-target bar-1.0-1.src.rpm
But you need to know the dependency order. If your packages have no interdependencies (or those dependencies are satisfied by packages already built or in the external repo) then you can just throw them all at Koji at once.
I know, I am clueless.
On Fri, Feb 26, 2010 at 3:28 PM, Mike Bonnet mikeb@redhat.com wrote:
On 02/26/2010 03:14 PM, Doug Reiland wrote:
ok, I just seem to be waiting for createrepo alot. Making very slow progress in package builds.
Is that because you only have one builder, and its busy processing newRepo tasks instead of build tasks? The simple answer is to add more builders.
The only time you should be waiting for repo regens is if your package depends on a previously-built package. In that case chain-build or scripting up something using wait-repo is the best option.
On Fri, Feb 26, 2010 at 2:27 PM, Mike Bonnet mikeb@redhat.com wrote:
On 02/26/2010 02:15 PM, Doug Reiland wrote:
You can emulate chain-build with a series of build and wait-repo commands if you like.
I want to be able to improve build throughput.
Is the following valid approach?
koji build --skip-tag mytag srpm1 srpm2 .... after done koji tag-pkg mytag pkg1 pkg2 .... koji regen-repo ...
I think you're assuming that a repo-regen will block other builds from taking place. That's not the case. A build will use the most recently created valid repo for the tag. Other builds can go on while a repo is being regenerated. When the repo regen is complete, it will be marked as the new latest repo, and any builds submitted after that will use the new repo instead of the old. Using --skip-tag doesn't improve throughput at all over using chain-build or wait-repo in between (dependent) builds. -- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys
-- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys
-- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys
-- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys
just, lack of hardware. I am trying to do proof of concept before go too far.
Maybe I cut loose on the server bootstrapping howto too fast. I have a local mirror so all source rpms and rpms. Source imports worked fine, but lots of binary rpms couldn't find a source rpm and it looked like I made a mess. If a srpm builds more than one binary rpm, is the import binary going to work.
On Fri, Feb 26, 2010 at 5:14 PM, Mike Bonnet mikeb@redhat.com wrote:
On 02/26/2010 04:06 PM, Doug Reiland wrote:
All I have is srpms. Is at least one createrepo builder required?
If I remove my builder from createrepo channel until a bunch of builds are done, ...
Then no createrepo tasks would be processed until all your builds are done. Is all of this because of a lack of hardware? The easiest thing would be to add more builders to your koji setup.
I don't under how wait-repo would work here. Or how you would script this. Does/would it assume package dependency knowledge and just plan the builds in a order that reduces dependencies?
Yes, if package bar depends on package foo then you could do something like:
koji build my-target foo-1.0-1.src.rpm && \ koji wait-repo --target my-target --build foo-1.0-1 && \ koji build my-target bar-1.0-1.src.rpm
But you need to know the dependency order. If your packages have no interdependencies (or those dependencies are satisfied by packages already built or in the external repo) then you can just throw them all at Koji at once.
I know, I am clueless.
On Fri, Feb 26, 2010 at 3:28 PM, Mike Bonnet mikeb@redhat.com wrote:
On 02/26/2010 03:14 PM, Doug Reiland wrote:
ok, I just seem to be waiting for createrepo alot. Making very slow progress in package builds.
Is that because you only have one builder, and its busy processing newRepo tasks instead of build tasks? The simple answer is to add more builders.
The only time you should be waiting for repo regens is if your package depends on a previously-built package. In that case chain-build or scripting up something using wait-repo is the best option.
On Fri, Feb 26, 2010 at 2:27 PM, Mike Bonnet mikeb@redhat.com wrote:
On 02/26/2010 02:15 PM, Doug Reiland wrote:
> > You can emulate chain-build with a series of build and wait-repo > commands if you like. >
I want to be able to improve build throughput.
Is the following valid approach?
koji build --skip-tag mytag srpm1 srpm2 .... after done koji tag-pkg mytag pkg1 pkg2 .... koji regen-repo ...
I think you're assuming that a repo-regen will block other builds from taking place. That's not the case. A build will use the most recently created valid repo for the tag. Other builds can go on while a repo is being regenerated. When the repo regen is complete, it will be marked as the new latest repo, and any builds submitted after that will use the new repo instead of the old. Using --skip-tag doesn't improve throughput at all over using chain-build or wait-repo in between (dependent) builds. -- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys
-- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys
-- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys
-- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys
-- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys
I am trying to bring another builder on line. Would I copy the certs. from the hub over to the other system? I generated 2 builders at that time, like the examples. On system2, called fedora2: would I use the same koji_ca_cert.crt file
Doug
On Fri, Feb 26, 2010 at 3:28 PM, Mike Bonnet mikeb@redhat.com wrote:
On 02/26/2010 03:14 PM, Doug Reiland wrote:
ok, I just seem to be waiting for createrepo alot. Making very slow progress in package builds.
Is that because you only have one builder, and its busy processing newRepo tasks instead of build tasks? The simple answer is to add more builders.
The only time you should be waiting for repo regens is if your package depends on a previously-built package. In that case chain-build or scripting up something using wait-repo is the best option.
On Fri, Feb 26, 2010 at 2:27 PM, Mike Bonnet mikeb@redhat.com wrote:
On 02/26/2010 02:15 PM, Doug Reiland wrote:
You can emulate chain-build with a series of build and wait-repo commands if you like.
I want to be able to improve build throughput.
Is the following valid approach?
koji build --skip-tag mytag srpm1 srpm2 .... after done koji tag-pkg mytag pkg1 pkg2 .... koji regen-repo ...
I think you're assuming that a repo-regen will block other builds from taking place. That's not the case. A build will use the most recently created valid repo for the tag. Other builds can go on while a repo is being regenerated. When the repo regen is complete, it will be marked as the new latest repo, and any builds submitted after that will use the new repo instead of the old. Using --skip-tag doesn't improve throughput at all over using chain-build or wait-repo in between (dependent) builds. -- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys
-- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys
-- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys
On 02/26/2010 11:27 AM, Doug Reiland wrote:
Sorry for the previous email, it got sent while editing
In koji-cli in handle_chain_build: elif '://' in src: # quick check that src might be a url build_level.append(src) elif '/' not in src and not src.endswith('.rpm') and len(src.split('-')) >= 3: # quick check that it looks like a N-V-R build_level.append(src)
should it be: elif '://' in src: # quick check that src might be a url build_level.append(src) elif '/' not in src and src.endswith('.rpm') and len(src.split('-')) >= 3: # quick check that it looks like a N-V-R build_level.append(src)
No, the "not src.endswith('.rpm')" was intentional. chain-build does not support building from srpms. This was a hard-coded policy decision, before Koji had a more flexible policy system for specifying whether building from srpm was ok or not on a per-tag basis. chain-build could be made to support srpms now, and leave the policy to the policy system. If this is desirable, please file a ticket at http://fedorahosted.org/koji and I'll take a look.
chain-build allows you to specify groups of SCM URLs to be built, or NVRs to wait for. The NVR support allows you to start a chain-build with " foo-1.0-1 : " which would say "wait until foo-1.0-1 is available in the current repo, and then start building the rest of the chain". This was a request from Fedora developers so they could make chain-builds depend on previous non-chained builds, removing the necessity to wait for repo regens, and allowing them to break up chain builds into multiple chunks.
I attempted to koji chain-build a couple of srpms for testing, but task is just sitting on "waitrepo". Also, I noticed it doesn't upload srpms so what is the BKM? do you import, then chain-build?
I'm surprised the chain-build command didn't return an error. What command did you run?
It did error, I hacked the code when I thought it was in error.
Man, it would be nice to be able to undo an import.
Doug
On Fri, Feb 26, 2010 at 11:46 AM, Mike Bonnet mikeb@redhat.com wrote:
On 02/26/2010 11:27 AM, Doug Reiland wrote:
Sorry for the previous email, it got sent while editing
In koji-cli in handle_chain_build: elif '://' in src: # quick check that src might be a url build_level.append(src) elif '/' not in src and not src.endswith('.rpm') and len(src.split('-')) >= 3: # quick check that it looks like a N-V-R build_level.append(src)
should it be: elif '://' in src: # quick check that src might be a url build_level.append(src) elif '/' not in src and src.endswith('.rpm') and len(src.split('-')) >= 3: # quick check that it looks like a N-V-R build_level.append(src)
No, the "not src.endswith('.rpm')" was intentional. chain-build does not support building from srpms. This was a hard-coded policy decision, before Koji had a more flexible policy system for specifying whether building from srpm was ok or not on a per-tag basis. chain-build could be made to support srpms now, and leave the policy to the policy system. If this is desirable, please file a ticket at http://fedorahosted.org/koji and I'll take a look.
chain-build allows you to specify groups of SCM URLs to be built, or NVRs to wait for. The NVR support allows you to start a chain-build with " foo-1.0-1 : " which would say "wait until foo-1.0-1 is available in the current repo, and then start building the rest of the chain". This was a request from Fedora developers so they could make chain-builds depend on previous non-chained builds, removing the necessity to wait for repo regens, and allowing them to break up chain builds into multiple chunks.
I attempted to koji chain-build a couple of srpms for testing, but task is just sitting on "waitrepo". Also, I noticed it doesn't upload srpms so what is the BKM? do you import, then chain-build?
I'm surprised the chain-build command didn't return an error. What command did you run? -- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys
buildsys@lists.fedoraproject.org
-
Doug Reiland -
Mike Bonnet -
Mike McLean