9:55 p.m.
Hey all,
I made a blog post[0] recently about bodhi[1], which is the name of the new
updates system for Fedora. I layed out a bunch of milestones[2] for the
project, 1.0 being the minimal functionality needed for Fedora 7.
If anyone is interested in helping out, feel free to grab any ticket;
and if you have questions or don't know where to begin, don't hesitate to ask.
I'm hoping to bring bodhi up to 1.0 (and hopefully beyond) during the hackfest
this weekend. We'll see what happens :)
luke
[0]: http://lewk.org/blog/2007/01/30/bodhi
[1]: https://hosted.fedoraproject.org/projects/bodhi
[2]: https://hosted.fedoraproject.org/projects/bodhi/roadmap
1:57 a.m.
On Wed, Jan 31, 2007 at 10:55:31PM -0500, Luke Macken wrote:
Hey all,
I made a blog post[0] recently about bodhi[1], which is the name of the new
updates system for Fedora. I layed out a bunch of milestones[2] for the
project, 1.0 being the minimal functionality needed for Fedora 7.
Very nice project.
If anyone is interested in helping out, feel free to grab any
ticket;
and if you have questions or don't know where to begin, don't hesitate to ask.
One question: Will there be a command line only way to push updates?
E.g something that takes type/bugs/cves/notes as arguments and/or
separate files?
Perhaps even a template file could reside in each folder waiting to be
filled out and make build refusing otherwise to build the package?
I'm hoping to bring bodhi up to 1.0 (and hopefully beyond) during
the hackfest
this weekend. We'll see what happens :)
luke
[0]: http://lewk.org/blog/2007/01/30/bodhi
[1]: https://hosted.fedoraproject.org/projects/bodhi
[2]: https://hosted.fedoraproject.org/projects/bodhi/roadmap
--
Axel.Thimm at ATrpms.net
12:50 p.m.
On Thu, Feb 01, 2007 at 08:57:50AM +0100, Axel Thimm wrote:
On Wed, Jan 31, 2007 at 10:55:31PM -0500, Luke Macken wrote:
> Hey all,
>
> I made a blog post[0] recently about bodhi[1], which is the name of the new
> updates system for Fedora. I layed out a bunch of milestones[2] for the
> project, 1.0 being the minimal functionality needed for Fedora 7.
Very nice project.
> If anyone is interested in helping out, feel free to grab any ticket;
> and if you have questions or don't know where to begin, don't hesitate to
ask.
One question: Will there be a command line only way to push updates?
E.g something that takes type/bugs/cves/notes as arguments and/or
separate files?
That's the plan. I opened a task[0] for the 1.1 milestone to write an XML-RPC
interface for interfacing with the system. From here, throwing together
some command line tools will be trivial.
Perhaps even a template file could reside in each folder waiting to
be
filled out and make build refusing otherwise to build the package?
That's a possibility; iirc, Brew has some way of keeping track of bugs
and such (via Makefile target parameters), but I'm not sure how it deals with
them (or how we would be able to deal with them). Until we get our hands on
the code, hacking up the Makefile to prompt the developer for this data (or using
some template, as you suggested) would be a fairly simple way of doing it.
luke
[0]: https://hosted.fedoraproject.org/projects/bodhi/ticket/5
5:56 a.m.
On Wed, 31 Jan 2007 22:55:31 -0500, Luke Macken wrote:
Hey all,
I made a blog post[0] recently about bodhi[1], which is the name of the new
updates system for Fedora. I layed out a bunch of milestones[2] for the
project, 1.0 being the minimal functionality needed for Fedora 7.
If anyone is interested in helping out, feel free to grab any ticket;
and if you have questions or don't know where to begin, don't hesitate to ask.
I'm hoping to bring bodhi up to 1.0 (and hopefully beyond) during the hackfest
this weekend. We'll see what happens :)
luke
[0]: http://lewk.org/blog/2007/01/30/bodhi
[1]: https://hosted.fedoraproject.org/projects/bodhi
[2]: https://hosted.fedoraproject.org/projects/bodhi/roadmap
The status updates raise a few more questions without that I've tried to
install the project locally:
* One of the screenshots,
https://hosted.fedoraproject.org/projects/bodhi/attachment/wiki/Screensho...
shows a "Push Console" where a single build job's packages are pushed and
createrepo is run afterwards. A huge difference to real-world pushing is
that this single operation does not take just 3 seconds, but several
minutes for each repository.
I fail to see how pushing packages like this can be a real-time operation
managed through a web interface. Unless a queueing server runs in the
background, which handles access to the repositories. Access for a variety
of operations, not limited to metadata creation. Add pruning, repoview,
multi-lib stuff. These operations are mutually exclusive.
* Currently, we push to a local master repository on the buildsys server
and sync that another machine at RDU. This requires SSH-based
authorization. What are the plans to change that within a web-based
updates system?
* "Staging area": I've browsed the bodhi wiki pages in search for
information on how to make better use of stages in the life-time of a
build-job's results.
Currently, we push from within a single staging area, plague's
"plague-results" tree. Earlier attempts, like moving packages from one
stage to another, have broken the buildsys (in particular the needsign
repo) too often, because it takes time for published packages to find
their way from fedoraproject.org to download.fedora.redhat.com, and
meanwhile, any pushed package we don't keep in the needsign repo breaks
the build servers' dep-chains. The work-around, that has worked flawlessly
since then, is to mark plague-results directories as "PUSHED" with an
empty file:
http://buildsys.fedoraproject.org/plague-results/fedora-6-extras/seedit/2...
That way we don't need to mess with the needsign repo rpms and metadata
and can keep the access read-only. With operations like "push" and
"unpush" and a testing repo it likely needs more to keep track of the
package life-cycle. What are the plans here?
4:54 p.m.
On Sat, Feb 10, 2007 at 12:56:11PM +0100, Michael Schwendt wrote:
The status updates raise a few more questions without that I've
tried to
install the project locally:
* One of the screenshots,
https://hosted.fedoraproject.org/projects/bodhi/attachment/wiki/Screensho...
shows a "Push Console" where a single build job's packages are pushed and
createrepo is run afterwards. A huge difference to real-world pushing is
that this single operation does not take just 3 seconds, but several
minutes for each repository.
The push process takes care of anything that needs to get done (ie
{,un}pushing, {,re}moving), based on developer requests (still due to
manual intervention for package signing), not just a single build job.
I fail to see how pushing packages like this can be a real-time
operation
managed through a web interface. Unless a queueing server runs in the
background, which handles access to the repositories. Access for a variety
of operations, not limited to metadata creation. Add pruning, repoview,
multi-lib stuff. These operations are mutually exclusive.
Right now there is no queueing server, and updates are pushed out in
batches by a single person, so being a real-time operation seems, and has
already half-assedly shown, to be feasible. Since few people actually
do the pushing, I'd like to implement whatever they want to use. Going
to a web site, clicking a button and waiting for a bit while being able
to see real time status updates hasn't gotten any complaints, but
With the current real-time status approach (python generators), it should
be trivial to be able to hook up any client to it, like a simple command-line
tool to do pushes. If Jesse would rather click a button and then get an email
at some point later on about the results, then that's fine too.
There is a lock on the repos that can be used to keep mutual exclusion with
regard to the pushing and sustainment scripts, but I agree that a queueing
service is an optimal approach.
* Currently, we push to a local master repository on the buildsys
server
and sync that another machine at RDU. This requires SSH-based
authorization. What are the plans to change that within a web-based
updates system?
That shouldn't change; the system is going to need to access the build
results of plague anyway, so I'm wondering if it would be best to
implement some sort of xmlrpc call to have plague stage the packages for
us? Since plague and bodhi (among other new pieces of infrastructure)
are in different co-locations, we definitely need some sort of way to
communicate. SSHFs was mentioned as a potentially viable solution, along
with rsync hackery; any suggestions?
* "Staging area": I've browsed the bodhi wiki pages in
search for
information on how to make better use of stages in the life-time of a
build-job's results.
How do you suggest we make better use of them? I'm fairly ignorant in
the ways of plague, so I don't know how the current staging of a
build-job results really works. At the moment there is only a single physical
updates-stage on disc. When a package is "pushed", it is copied from the build
result over to this updates-stage. Before the push an update is either
in 'testing', or is waiting to be signed/pushed/moved.
It seems like having some of these staging hooks in plague itself might be a
good idea, instead of pulling and moving packages out from under it.
Currently, we push from within a single staging area, plague's
"plague-results" tree. Earlier attempts, like moving packages from one
stage to another, have broken the buildsys (in particular the needsign
repo) too often, because it takes time for published packages to find
their way from fedoraproject.org to download.fedora.redhat.com, and
meanwhile, any pushed package we don't keep in the needsign repo breaks
the build servers' dep-chains. The work-around, that has worked flawlessly
since then, is to mark plague-results directories as "PUSHED" with an
empty file:
http://buildsys.fedoraproject.org/plague-results/fedora-6-extras/seedit/2...
That way we don't need to mess with the needsign repo rpms and metadata
and can keep the access read-only. With operations like "push" and
"unpush" and a testing repo it likely needs more to keep track of the
package life-cycle. What are the plans here?
Well, the current implementation of bodhi starts all updates off as
'testing', and from there it can be moved to final when requested (but
eventually after a given number of approvals or pushed through by the
security response team). From here an update can raise a
few requests: push, unpush, and move; but as far as the filesystem stage
is concerned, files only get written when updates are pushed to testing
or final.
John Polestra made a few diagrams of the update procedure that I'm in
the process of touching up. Hopefully with these, and eventually some
details on Koji, we can optimize the package lifecycle quite a bit.
luke
2:10 a.m.
On Tue, 13 Feb 2007 17:54:46 -0500, Luke Macken wrote:
the system is going to need to access the build
results of plague anyway, so I'm wondering if it would be best to
implement some sort of xmlrpc call to have plague stage the packages for
us? Since plague and bodhi (among other new pieces of infrastructure)
are in different co-locations, we definitely need some sort of way to
communicate. SSHFs was mentioned as a potentially viable solution, along
with rsync hackery; any suggestions?
Access to the build results so far has been on the local fs. An exclusive
lock file is shared between plague and the pushscript to handle access to
the needsign repositories.
As soon as build results are to be mirrored from another location, it
would need something different in order to work only on complete build
results.
Querying plague would be beneficial anyway, since else you cannot map
build job numbers to packages and vice versa. And, of course, what is
currently implemented in a brute-force way is the removal of old and
pushed build results from the needsign repo after two days. With a change
in infrastructure, plague could do that itself. But before it could, it
needs input from the push front-end (like appropriate status feedback
similar to "plague-client finish <jobid>", which needs admin privileges
so
far, however).
> * "Staging area": I've browsed the bodhi wiki
pages in search for
> information on how to make better use of stages in the life-time of a
> build-job's results.
How do you suggest we make better use of them? I'm fairly ignorant in
the ways of plague, so I don't know how the current staging of a
build-job results really works. At the moment there is only a single physical
updates-stage on disc. When a package is "pushed", it is copied from the build
result over to this updates-stage. Before the push an update is either
in 'testing', or is waiting to be signed/pushed/moved.
It seems like having some of these staging hooks in plague itself might be a
good idea, instead of pulling and moving packages out from under it.
Package status so far:
1 - needsign
1.2 - needsign/EXCLUDED
2 - needsign/PUSHED
In the future, it needs more stages, doesn't it? (and the concept
of collections adds on top of that)
1 - needsign
1.3 - signed
2 - target updates testing (signed or unsigned)
3 - target updates released (signed or unsigned)
4 - unpush (signed or unsigned)
5 - finished
1 is the multi-arch repository which is available to all build servers.
5 is not used yet, since telling plague about it needs admin privileges,
and at the level of the pushscript we don't know about job numbers.
For 2-4, it needs a database to keep track of the current package status
and location, and it also needs the places where to store the packages
temporarily. Where is that? Automatic or manual signing aside, packages
need to be available to the buildsys all of the time, regardless of
whether they are signed or unsigned or pushed to a public repository or
moved from "testing" to "released" (so it is possible to build test
updates against eachother, but also to avoid building final updates
against test updates - or this a grey area?).
6:09 p.m.
Luke Macken said the following on 02/13/2007 02:54 PM Pacific Time:
John Polestra made a few diagrams of the update procedure that
I'm in
the process of touching up. Hopefully with these, and eventually some
details on Koji, we can optimize the package lifecycle quite a bit.
I just added the diagrams to the bodhi project page:
https://hosted.fedoraproject.org/projects/bodhi under the "Design" section. All
feedback is welcome!
I've also added the original oodraw files to bodhi's source control under the
"docs" directory.
John
3:28 p.m.
Luke Macken (lmacken(a)redhat.com) said:
> * Currently, we push to a local master repository on the
buildsys server
> and sync that another machine at RDU. This requires SSH-based
> authorization. What are the plans to change that within a web-based
> updates system?
That shouldn't change; the system is going to need to access the build
results of plague anyway, so I'm wondering if it would be best to
implement some sort of xmlrpc call to have plague stage the packages for
us? Since plague and bodhi (among other new pieces of infrastructure)
are in different co-locations, we definitely need some sort of way to
communicate. SSHFs was mentioned as a potentially viable solution, along
with rsync hackery; any suggestions?
Well, *assuming* we're going to tie bodhi to koji (as opposed to
having it target both), it depends on how we set koji up.
> * "Staging area": I've browsed the bodhi wiki
pages in search for
> information on how to make better use of stages in the life-time of a
> build-job's results.
How do you suggest we make better use of them? I'm fairly ignorant in
the ways of plague, so I don't know how the current staging of a
build-job results really works. At the moment there is only a single physical
updates-stage on disc. When a package is "pushed", it is copied from the build
result over to this updates-stage. Before the push an update is either
in 'testing', or is waiting to be signed/pushed/moved.
It seems like having some of these staging hooks in plague itself might be a
good idea, instead of pulling and moving packages out from under it.
So, I think we're going to need a middle-stage tool between koji and
bodhi/pungi; there's logic like multilib selection, package selection,
interaction with signing, etc. that should all be in one place.
Bill
4:14 p.m.
On Friday 16 February 2007 16:28, Bill Nottingham wrote:
So, I think we're going to need a middle-stage tool between koji
and
bodhi/pungi; there's logic like multilib selection, package selection,
interaction with signing, etc. that should all be in one place.
That's a big +10 there partner. This same middle-stage tool could be used to
create the rawhide repo too, pungi need only get involved when say kernel or
anaconda gets bumped... (/me wanders off into the woods)
--
Jesse Keating
Release Engineer: Fedora
6281
days inactive
6296
days old
infrastructure@lists.fedoraproject.org
9 comments
6 participants
participants (6)
-
Axel Thimm -
Bill Nottingham -
Jesse Keating -
John Poelstra -
Luke Macken -
Michael Schwendt