I have spent significant portions of the last couple of days trying to weap my mind around what Fedora is doing. Let me start off by saying that I think the direction of the project is wonderful and much needed. As a system administrator managing three boxes, I love the prospect of being able to use apt-get or yum to continuously upgrade my systems without having to go through periodic CD shuffles and reboots. Thus, I've joined fedora-devel to help.
However, at the moment there is at least one serious practical problem that loses me as a package contributor. That is the use of Bugzilla (or any other method that requires manual click-and-confirm) for RPM submission. Most people maintain one or two packages at most; they can live with a manual submission process. At last count, I maintained thirty-seven packages -- thus, I can't. (This is also why I don't publish through SourceForge.)
What I need is to be able to write an "upload" script for each of my projects *once* (per project) that remote-scripts the RPM submission process and all other things I need to do to publish a release. So, what I need from Fedora is for there to be a submission front end (call it 'fedora-submit') which, given a collection of RPMS as arguments, does in a batch mode all that is necessary to upload them and put them on a submission queue at fedora. It is OK if fedora-submit requires additional metadata as long as it can all be specified at start of run by command-line switches.
I asked about this on the IRC channel and was told that Bugzilla has an XML-RPC interface. If so, writing fedora-submit as an XML-RPC client ought not be too difficult; I would be surprised if it required more than 150-200 lines of Python.
I am actually willing to write this myself, if anyone can point me at documentation for the XML-RPC interface to Bugzilla and is willing to answer questions about places where the documentation is inadequate. In the process, I would be willing to help improve the (presently inadequate) documentation on the submission procedure.
In case it's not obvious, I think the result ought to ship as part of fedora-core in order to reduce the entry barriers for independent packagers as much as possible.
On Sun, 28 Sep 2003, Eric S. Raymond wrote: [...]
However, at the moment there is at least one serious practical problem that loses me as a package contributor. That is the use of Bugzilla (or any other method that requires manual click-and-confirm) for RPM submission. Most people maintain one or two packages at most; they can live with a manual submission process. At last count, I maintained thirty-seven packages -- thus, I can't. (This is also why I don't publish through SourceForge.)
[...]
I don't think there is an official "RPM submission procedure" yet. The unofficial (only!) one is to send mails to Red Hat folks, or add a bug in Bugzilla.
But I think the final one will be much, much better .. once we get there. :-)
Pekka Savola pekkas@netcore.fi:
I don't think there is an official "RPM submission procedure" yet. The unofficial (only!) one is to send mails to Red Hat folks, or add a bug in Bugzilla.
But I think the final one will be much, much better .. once we get there. :-)
Indeed it will, because I'm already on the problem :-) I am stepping up with an offer to write, document, and maintain the client software for doing package submissions.
I had a long IRC chat with some key Bugzilla developers this afternoon. The conclusion of the chat was that Bugzilla ought to have an XML-RPC interface for posting bugs. But, pending that, Christian Reis gave me a scratch Python script for submitting bugs via scripted CGI that I am in the process of reworking into a production-quality tool. It can be used with Bugzilla as it is now. It's called `bug-bugzilla'.
Once I get bug-bugzilla done, I'll write a thin wrapper around it called `fedora-submit' for submitting RPMS through Fedora Bugzilla. If your channel for RPM submissions changes, I will modify fedora-submit accordingly. Someday it might be an XML-RPC client, but users need not care.
Fedora can ship both scripts as RPMs, fedora-submit will depend on bug-bugzilla until and unless that changes, and everybody should be happy. Including the Bugzilla people, who have had people clamoring for a production-quality tool of this kind forever.
Does this sound like a good plan?
On Mon, 29 Sep 2003, Eric S. Raymond wrote: [...]
Fedora can ship both scripts as RPMs, fedora-submit will depend on bug-bugzilla until and unless that changes, and everybody should be happy. Including the Bugzilla people, who have had people clamoring for a production-quality tool of this kind forever.
Does this sound like a good plan?
If a package in Fedora Core is maintained by non-RH person X, having person X go through Bugzilla to update his package (whether with a tool or not), implying having a Red Hat person look at the problem and apply it in a timely fashion, is IMHO *way* too heavy a process.
We need more than that, e.g. CVS commit access to the server housing the RPM spec files and patch files, etc.
I'm not saying that the tool you propose would not be useful, I just think that (at least as I see Fedora) it's still too heavy a process...
Pekka Savola pekkas@netcore.fi:
If a package in Fedora Core is maintained by non-RH person X, having person X go through Bugzilla to update his package (whether with a tool or not), implying having a Red Hat person look at the problem and apply it in a timely fashion, is IMHO *way* too heavy a process.
We need more than that, e.g. CVS commit access to the server housing the RPM spec files and patch files, etc.
I'm not saying that the tool you propose would not be useful, I just think that (at least as I see Fedora) it's still too heavy a process...
That's not a philosophical issue I'm equipped to have an opinion on, at this point. Fundamentally, I don't care (for purposes of designing this tool) whether the submission channel is Bugzilla or something else, and whether submissions are filtered by a human or not.
What I care about is that *there be a tool that allows remote-scripting the submission process*. I don't see that the interface of the tool needs to be any more complicated than
fedora-submit --tree {stable|testing|...} foo-1.2.3.fdr.0.rpm
regardless of how the process is actually implemented. My present plan is to implement fedora-submit as a wrapper around a script for submitting Bugzilla bugs (which script I have just finished writing) but that is an implementation detail about which the user should not have to care. If you guys want to change the implementation mechanism, you just tell me and I'll make fedora-submit use it.
So let's *not* get sidetracked onto whether the submission process is right or not. Just tell me what it actually *is*.
Eric S. Raymond wrote:
regardless of how the process is actually implemented. My present plan is to implement fedora-submit as a wrapper around a script for submitting Bugzilla bugs (which script I have just finished writing) but that is an implementation detail about which the user should not have to care. If you guys want to change the implementation mechanism, you just tell me and I'll make fedora-submit use it.
So let's *not* get sidetracked onto whether the submission process is right or not. Just tell me what it actually *is*.
Fedora.us is not "The Fedora Project". Fedora.us is the older packaging project for united volunteer packagers around the RH platform.
(Fedora.us continues in operation during the next few months while the new Fedora is in progress. No packages will be published from the new Fedora for quite a while because all the infrastructure, policies, standards and procedures need to be formed. It has been unofficially suggested by some RH elders to point people at fedora.us for package submission and approval during these next few months because fedora.redhat.com is NOT READY. fedora.us is recognized as having some technical clue and over-paranoid reluctance in accepting stuff, so stuff that is accepted is likely to be accepted into Fedora Extras much more readily at a later date .... after legal approvals and stuff... stuff stuff stuff)
It was our (fedora.us) intention from the beginning to better automate the submission process with command-line based tools, and eventually write a (RDBMS) database driven management system. Until the effort was put forward to make that happen, Fedora.us had used a Bugzilla based package submission, QA discussion and tracking system. It worked great for the publication of hundreds of packages, with almost 200 more currently needing QA approval.
However this being said, there has been almost zero discussion about whether the new fedora.redhat.com will use anything like fedora.us' submission and approval process. While we don't know yet fedora.redhat.com will become, now is the time to experiment with fedora.us to learn more about "What works" and "What doesn't".
Your suggestion of a fedora-submit script sounds like a good way to reduce overhead in submitting Bugzilla reports for packages. I am glad somebody finally wants to implement it. I suggest to you to read through many of the fedora.us reports to see the steps involved in this process. Your script would need to (off the top of my head)
1) Check for duplicates. 2) Submit a new report for a package, along with GPG signatures. 3) Submit revised packages along with new GPG signatures after some discussion points out flaws. 4) Upload packages to *somewhere* which we still haven't worked out. We obviously cannot give CVS access to everyone, and there may be issues with a public upload location at an official server. I much prefer the GPG signatures + submitter controlled URL process that fedora.us currently uses, but perhaps a future process may have a hybrid. The more senior trusted packagers have upload access to an official staging area or CVS, while everyone else uses GPG + URLs.
(Sopwith, the "hostile build" requirement with mach+vserver would be a boon in the "everyone else" category, because automation greatly reduces the amount of man-hours needed for package development while reducing the cost of entry for new-comers.)
Regarding your other comment regarding the over-complexity of package naming requirements in fedora.us documents: May I warn the feeling of "too-complex" may be partly from ignorance of the complexity of avoiding clashes. There are a great many common problems that happen when people do not THINK when creating their package.
Those requirements for seemingly over-complex naming requirements was the result of arguing for almost 4 months with no useful packaging happening. The guidelines were mainly about preventing common cases where "Epoch" needs to be incremented for poor reasons. Unfortunately a large part of the other requirements were based upon assumption that Red Hat would not pay attention to our naming scheme or packages, and we needed to avoid any chance of conflicting with future RH updates. In almost all cases the naming scheme has successfully avoided naming and version clashes with RH released, errata and beta packages.
Fortunately now that we are no longer an unaffilited 3rd party, I believe we no longer need the many weird corner case naming requirements. The naming document can be greatly simplified, dropping the leading "X.fdr." part of the %release tag and all related naming policies because they are not needed any longer. (We would need to have some common agreement with RH and fedora.us packagers in order to make this official, but I believe that will be an easy sell after I post my upcoming draft for "Fedora Project package naming conventions" for fedora.redhat.com.)
In the mean time, please submit your packages to fedora.us in any matter that you wish. Don't worry if you don't understand the overly complex naming guidelines, because the QA people will quickly point out errors.
You mention that you would like to help in rewriting that documentation. We would greatly appreciate your help in doing so (and I personally like your writing skills), but may I highly suggest seeing the actual old and inefficient process in motion for a while before making any assumptions.
Also it is my opinion that while our current GPG requirements and manual Bugzilla usage seems like a "waste of time", that time requirement pales in comparison to the QA approval process. fedora-submit and improved command line tools integrated with GPG would speed up the Bugzilla report communication process would be a plus to that process.
BTW, your fedora-submit tools may be good for our RPM development toolkit package "fedora-rpmdevtools" which is briefly documented here: http://www.fedora.us/wiki/FedoraRPMDevTools
Warren Togami warren@togami.com
Warren Togami warren@togami.com:
Your suggestion of a fedora-submit script sounds like a good way to reduce overhead in submitting Bugzilla reports for packages. I am glad somebody finally wants to implement it. I suggest to you to read through many of the fedora.us reports to see the steps involved in this process. Your script would need to (off the top of my head)
- Check for duplicates.
- Submit a new report for a package, along with GPG signatures.
- Submit revised packages along with new GPG signatures after some
discussion points out flaws. 4) Upload packages to *somewhere* which we still haven't worked out. We obviously cannot give CVS access to everyone, and there may be issues with a public upload location at an official server. I much prefer the GPG signatures + submitter controlled URL process that fedora.us currently uses, but perhaps a future process may have a hybrid. The more senior trusted packagers have upload access to an official staging area or CVS, while everyone else uses GPG + URLs.
On point 1, the design of fedora-submit should not try to wire in repository policy. In particular, putting dup checking in there would probably be a bad idea, as your concept of a dup is likely to evolve over time. For example, is it a dup when the same-named RPM is submitted for two different trees? You want to be able to change the answers to questions like that without replacing every client instance in the world.
Points 2 and 3 look right to me. As a toolbuilder I'm agnostic on point 4, but I share your preference. There's no reason for your server to have to pay storage costs for RPMs not yet accepted when a URL pointer can be functionally equivalent to having a copy.
Fortunately now that we are no longer an unaffilited 3rd party, I believe we no longer need the many weird corner case naming requirements. The naming document can be greatly simplified, dropping the leading "X.fdr." part of the %release tag and all related naming policies because they are not needed any longer.
Good. Then by all means nuke 'em.
In the mean time, please submit your packages to fedora.us in any matter that you wish. Don't worry if you don't understand the overly complex naming guidelines, because the QA people will quickly point out errors.
OK. I'll write fedora-submit and do some testing, then. I think I've essentially finished the bug-bugzilla tool that the first implementation of fedora-submit will use; it's now in evaluation and testing with Christian Reis of the Bogzilla project.
BTW, your fedora-submit tools may be good for our RPM development toolkit package "fedora-rpmdevtools" which is briefly documented here: http://www.fedora.us/wiki/FedoraRPMDevTools
Ahh, that's good to know. Yes, that is obviously the right place. I see I even got the naming convention right :-).
-
Eric S. Raymond -
Pekka Savola -
Warren Togami