GDBM upgrade in F17

[Jesse Keating]

On Oct 4, 2011, at 10:31 PM, Adam Williamson wrote:
> On Thu, 2011-09-29 at 09:39 -0400, Jesse Keating wrote:
>> On Sep 29, 2011, at 6:22 AM, Nils Philippsen wrote:
>>> 
>>> Ahh I'm late. Anyway, that's a side-effect of using the bodhi interface
>>> to manage something in koji then. It'd still be handy if we could use
>>> that for Rawhide so we don't break all dependent things for people who
>>> want to test something else than what we're breaking at the moment.
>> 
>> You're thinking of creating a /new/ build target and build tag/root.
>> That's not what bodhi does.
>> 
>> Buildroot overrides for the rawhide tags make no sense, as anything
>> built automatically lands in the build root.
> 
> So really want they want is a *tag*, not a buildroot override, right?
> 
> They can have a tag for Rawhide if they request one, I presume?


I don't even know what that would mean.

Tag is an overloaded term, even within koji itself.

A "tag" is an identifier that can be applied to specific builds.  It can also create a unique place to assign package ownership and package membership, e.g. "bash does not exist in tag f7-nobash" and "bash-3.2.5-2.fc17 has been tagged with f16".  Tag can also represent what collection of packages is used when a build is done for a certain build target.  The f17-candidate build target makes use of the f17-build tag to populate the build root.  After the build is done, the f17 tag is applied to the build.  Tags also have inheritance, so the f17-build tag inherits data from the f17 tag, which is why when a build is done and is tagged with f17, it will show up in the f17-build tag and be used in the build root.

We also make use of "override" tags, our own terminology here.  By injecting a tag between f17-build and f17, we have a way of 'overriding' what will normally appear in the build root by way of f17.  Thus we can temporarily 'override' a bad build, or in the case of Fedoras that are managed by updates-testing, we can short-circuit the time it would take for a certain build to wind up in the build root.

Further, we can create topic based tags that inherit from a base tag, but keep any builds from winding up in the base tag.  We've done this for perl rebuilds and other sweeping changes that would be very disruptive to the day to day activities.  For example, we could create a tag f17-perl that inherits data and builds from f17, create a build target f17-perl that makes use of the f17-perl tag to populate the build root, and all builds when finished would get tagged with f17-perl.  They would have their own little world in which to prepare a mass rebuild, and when done "move" it all into f17 proper.  I suspect this particular scenario is what some people are looking for in bodhi for rawhide, but there is a (high) cost in the form of repo regeneration.  Every time something gets tagged for f17 it will cause a newRepo call for any targets that use build tags that inherit from f17.  That would be f17-candidate, and now f17-perl, and any other topic tags that exist.  If there was an f18 tag at the time it would also get a repo regeneration, and any f18 topic tags.  newRepo tasks are one of the most expensive things our build system does, dealing with 12K~ source packages generating something like 50K rpms on each of our arches.  As much speedup as we've tried to do this is still a heavy process on the koji DB and the filesystems.  So much so that we have to limit them to only 3 at a time.  Once you start adding a lot of topic branches that 3 can be a hindrance and can cause significant delays in our build systems ability to keep its build roots up to date.  20 minutes or so per newRepo, 3 at a time.  If you have 9 tags to regenerate it can be over 1 hour between each newRepo for each particular target.

tl;dr if you're looking for bodhi to allow on-demand creation of topic-tags for isolating build efforts from rawhide, that's too expensive of an item to allow a free-for-all at this time, in my opinion.

- jlk