From esr at thyrsus.com Sun Sep 28 16:18:34 2003 Content-Type: multipart/mixed; boundary="===============3116241079492109879==" MIME-Version: 1.0 From: Eric S. Raymond To: devel at lists.fedoraproject.org Subject: Showstopper in the RPM submission procesdure Date: Sun, 28 Sep 2003 16:22:15 -0400 Message-ID: <20030928202215.GA5175@thyrsus.com> --===============3116241079492109879== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable I have spent significant portions of the last couple of days trying to weap my mind around what Fedora is doing. Let me start off by saying that I think the direction of the project is wonderful and much needed. As a system administrator managing three boxes, I love the prospect of being able to use apt-get or yum to continuously upgrade my systems without having to go through periodic CD shuffles and reboots. Thus, I've joined fedora-devel to help. However, at the moment there is at least one serious practical problem that loses me as a package contributor. That is the use of Bugzilla (or any other method that requires manual click-and-confirm) for RPM submission. Most people maintain one or two packages at most; they can live with a manual submission process. At last count, I maintained thirty-seven packages -- thus, I can't. (This is also why I don't publish through SourceForge.) What I need is to be able to write an "upload" script for each of my projects *once* (per project) that remote-scripts the RPM submission process and all other things I need to do to publish a release. So, what I need from Fedora is for there to be a submission front end (call it 'fedora-submit') which, given a collection of RPMS as arguments, does in a batch mode all that is necessary to upload them and put them on a submission queue at fedora. It is OK if fedora-submit requires additional metadata as long as it can all be specified at start of run by command-line switches. I asked about this on the IRC channel and was told that Bugzilla has an XML-RPC interface. If so, writing fedora-submit as an XML-RPC client ought not be too difficult; I would be surprised if it required more than 150-200 lines of Python. I am actually willing to write this myself, if anyone can point me at = documentation for the XML-RPC interface to Bugzilla and is willing to = answer questions about places where the documentation is inadequate. In the process, I would be willing to help improve the (presently = inadequate) documentation on the submission procedure. In case it's not obvious, I think the result ought to ship as part of fedora-core in order to reduce the entry barriers for independent = packagers as much as possible. -- = Eric S. Raymond --===============3116241079492109879==-- From pekkas at netcore.fi Mon Sep 29 02:32:43 2003 Content-Type: multipart/mixed; boundary="===============8394330250712119154==" MIME-Version: 1.0 From: Pekka Savola To: devel at lists.fedoraproject.org Subject: Re: Showstopper in the RPM submission procesdure Date: Mon, 29 Sep 2003 09:36:18 +0300 Message-ID: In-Reply-To: 20030928202215.GA5175@thyrsus.com --===============8394330250712119154== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On Sun, 28 Sep 2003, Eric S. Raymond wrote: [...] > However, at the moment there is at least one serious practical problem > that loses me as a package contributor. That is the use of Bugzilla > (or any other method that requires manual click-and-confirm) for RPM > submission. Most people maintain one or two packages at most; they > can live with a manual submission process. At last count, I > maintained thirty-seven packages -- thus, I can't. (This is also why > I don't publish through SourceForge.) [...] I don't think there is an official "RPM submission procedure" yet. The = unofficial (only!) one is to send mails to Red Hat folks, or add a bug in = Bugzilla. But I think the final one will be much, much better .. once we get there. = :-) -- = Pekka Savola "You each name yourselves king, yet the Netcore Oy kingdom bleeds." Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings --===============8394330250712119154==-- From esr at thyrsus.com Mon Sep 29 02:55:27 2003 Content-Type: multipart/mixed; boundary="===============6031835689206870857==" MIME-Version: 1.0 From: Eric S. Raymond To: devel at lists.fedoraproject.org Subject: Re: Showstopper in the RPM submission procesdure Date: Mon, 29 Sep 2003 02:59:08 -0400 Message-ID: <20030929065908.GA8062@thyrsus.com> In-Reply-To: Pine.LNX.4.44.0309290934210.15163-100000@netcore.fi --===============6031835689206870857== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Pekka Savola : > I don't think there is an official "RPM submission procedure" yet. The = > unofficial (only!) one is to send mails to Red Hat folks, or add a bug in = > Bugzilla. > = > But I think the final one will be much, much better .. once we get there. = > :-) Indeed it will, because I'm already on the problem :-) I am stepping up wi= th an offer to write, document, and maintain the client software for doing = package submissions. I had a long IRC chat with some key Bugzilla developers this afternoon. = The conclusion of the chat was that Bugzilla ought to have an XML-RPC = interface for posting bugs. But, pending that, Christian Reis gave me = a scratch Python script for submitting bugs via scripted CGI that I = am in the process of reworking into a production-quality tool. It can be used with Bugzilla as it is now. It's called `bug-bugzilla'. Once I get bug-bugzilla done, I'll write a thin wrapper around it called `fedora-submit' for submitting RPMS through Fedora Bugzilla. If your = channel for RPM submissions changes, I will modify fedora-submit accordingly. Someday it might be an XML-RPC client, but users need not care. Fedora can ship both scripts as RPMs, fedora-submit will depend on bug-bugzilla until and unless that changes, and everybody should be happy. Including the Bugzilla people, who have had people clamoring for a production-quality tool of this kind forever. Does this sound like a good plan? -- = Eric S. Raymond --===============6031835689206870857==-- From pekkas at netcore.fi Mon Sep 29 04:57:48 2003 Content-Type: multipart/mixed; boundary="===============2779664869819550724==" MIME-Version: 1.0 From: Pekka Savola To: devel at lists.fedoraproject.org Subject: Re: Showstopper in the RPM submission procesdure Date: Mon, 29 Sep 2003 12:01:24 +0300 Message-ID: In-Reply-To: 20030929065908.GA8062@thyrsus.com --===============2779664869819550724== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On Mon, 29 Sep 2003, Eric S. Raymond wrote: [...] > Fedora can ship both scripts as RPMs, fedora-submit will depend on > bug-bugzilla until and unless that changes, and everybody should be > happy. Including the Bugzilla people, who have had people clamoring > for a production-quality tool of this kind forever. > = > Does this sound like a good plan? If a package in Fedora Core is maintained by non-RH person X, having person X go through Bugzilla to update his package (whether with a tool or not), implying having a Red Hat person look at the problem and apply it in a timely fashion, is IMHO *way* too heavy a process. We need more than that, e.g. CVS commit access to the server housing the RPM spec files and patch files, etc. I'm not saying that the tool you propose would not be useful, I just think = that (at least as I see Fedora) it's still too heavy a process... -- = Pekka Savola "You each name yourselves king, yet the Netcore Oy kingdom bleeds." Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings --===============2779664869819550724==-- From esr at thyrsus.com Mon Sep 29 05:26:41 2003 Content-Type: multipart/mixed; boundary="===============0105758788049130662==" MIME-Version: 1.0 From: Eric S. Raymond To: devel at lists.fedoraproject.org Subject: Re: Showstopper in the RPM submission procesdure Date: Mon, 29 Sep 2003 05:30:23 -0400 Message-ID: <20030929093023.GA9408@thyrsus.com> In-Reply-To: Pine.LNX.4.44.0309291157500.17160-100000@netcore.fi --===============0105758788049130662== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Pekka Savola : > If a package in Fedora Core is maintained by non-RH person X, having > person X go through Bugzilla to update his package (whether with a tool or > not), implying having a Red Hat person look at the problem and apply it in > a timely fashion, is IMHO *way* too heavy a process. > = > We need more than that, e.g. CVS commit access to the server housing the > RPM spec files and patch files, etc. > = > I'm not saying that the tool you propose would not be useful, I just thin= k = > that (at least as I see Fedora) it's still too heavy a process... That's not a philosophical issue I'm equipped to have an opinion on, at this point. Fundamentally, I don't care (for purposes of designing this tool) whether the submission channel is Bugzilla or something else, and whether submissions are filtered by a human or not. = What I care about is that *there be a tool that allows remote-scripting the submission process*. I don't see that the interface of the tool needs to be any more complicated than = fedora-submit --tree {stable|testing|...} foo-1.2.3.fdr.0.rpm regardless of how the process is actually implemented. My present plan is to implement fedora-submit as a wrapper around a script for submitting Bugzilla bugs (which script I have just finished writing) = but that is an implementation detail about which the user should not have to care. If you guys want to change the implementation mechanism, you just tell me and I'll make fedora-submit use it. So let's *not* get sidetracked onto whether the submission process = is right or not. Just tell me what it actually *is*. -- = Eric S. Raymond --===============0105758788049130662==-- From warren at togami.com Mon Sep 29 07:21:30 2003 Content-Type: multipart/mixed; boundary="===============1475717851112816145==" MIME-Version: 1.0 From: Warren Togami To: devel at lists.fedoraproject.org Subject: Re: Showstopper in the RPM submission procesdure Date: Mon, 29 Sep 2003 01:24:58 -1000 Message-ID: <3F78168A.4050508@togami.com> In-Reply-To: 20030929093023.GA9408@thyrsus.com --===============1475717851112816145== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Eric S. Raymond wrote: > = > regardless of how the process is actually implemented. My present > plan is to implement fedora-submit as a wrapper around a script for > submitting Bugzilla bugs (which script I have just finished writing) = > but that is an implementation detail about which the user should not > have to care. If you guys want to change the implementation mechanism, > you just tell me and I'll make fedora-submit use it. > = > So let's *not* get sidetracked onto whether the submission process = > is right or not. Just tell me what it actually *is*. Fedora.us is not "The Fedora Project". Fedora.us is the older packaging = project for united volunteer packagers around the RH platform. (Fedora.us continues in operation during the next few months while the = new Fedora is in progress. No packages will be published from the new = Fedora for quite a while because all the infrastructure, policies, = standards and procedures need to be formed. It has been unofficially = suggested by some RH elders to point people at fedora.us for package = submission and approval during these next few months because = fedora.redhat.com is NOT READY. fedora.us is recognized as having some = technical clue and over-paranoid reluctance in accepting stuff, so stuff = that is accepted is likely to be accepted into Fedora Extras much more = readily at a later date .... after legal approvals and stuff... stuff = stuff stuff) It was our (fedora.us) intention from the beginning to better automate = the submission process with command-line based tools, and eventually = write a (RDBMS) database driven management system. Until the effort = was put forward to make that happen, Fedora.us had used a Bugzilla based = package submission, QA discussion and tracking system. It worked great = for the publication of hundreds of packages, with almost 200 more = currently needing QA approval. However this being said, there has been almost zero discussion about = whether the new fedora.redhat.com will use anything like fedora.us' = submission and approval process. While we don't know yet = fedora.redhat.com will become, now is the time to experiment with = fedora.us to learn more about "What works" and "What doesn't". Your suggestion of a fedora-submit script sounds like a good way to = reduce overhead in submitting Bugzilla reports for packages. I am glad = somebody finally wants to implement it. I suggest to you to read = through many of the fedora.us reports to see the steps involved in this = process. Your script would need to (off the top of my head) 1) Check for duplicates. 2) Submit a new report for a package, along with GPG signatures. 3) Submit revised packages along with new GPG signatures after some = discussion points out flaws. 4) Upload packages to *somewhere* which we still haven't worked out. We = obviously cannot give CVS access to everyone, and there may be issues = with a public upload location at an official server. I much prefer the = GPG signatures + submitter controlled URL process that fedora.us = currently uses, but perhaps a future process may have a hybrid. The = more senior trusted packagers have upload access to an official staging = area or CVS, while everyone else uses GPG + URLs. (Sopwith, the "hostile build" requirement with mach+vserver would be a = boon in the "everyone else" category, because automation greatly reduces = the amount of man-hours needed for package development while reducing = the cost of entry for new-comers.) Regarding your other comment regarding the over-complexity of package = naming requirements in fedora.us documents: May I warn the feeling of = "too-complex" may be partly from ignorance of the complexity of avoiding = clashes. There are a great many common problems that happen when people = do not THINK when creating their package. Those requirements for seemingly over-complex naming requirements was = the result of arguing for almost 4 months with no useful packaging = happening. The guidelines were mainly about preventing common cases = where "Epoch" needs to be incremented for poor reasons. Unfortunately a = large part of the other requirements were based upon assumption that Red = Hat would not pay attention to our naming scheme or packages, and we = needed to avoid any chance of conflicting with future RH updates. In = almost all cases the naming scheme has successfully avoided naming and = version clashes with RH released, errata and beta packages. Fortunately now that we are no longer an unaffilited 3rd party, I = believe we no longer need the many weird corner case naming = requirements. The naming document can be greatly simplified, dropping = the leading "X.fdr." part of the %release tag and all related naming = policies because they are not needed any longer. (We would need to have some common agreement with RH and fedora.us = packagers in order to make this official, but I believe that will be an = easy sell after I post my upcoming draft for "Fedora Project package = naming conventions" for fedora.redhat.com.) In the mean time, please submit your packages to fedora.us in any matter = that you wish. Don't worry if you don't understand the overly complex = naming guidelines, because the QA people will quickly point out errors. You mention that you would like to help in rewriting that documentation. = We would greatly appreciate your help in doing so (and I personally = like your writing skills), but may I highly suggest seeing the actual = old and inefficient process in motion for a while before making any = assumptions. Also it is my opinion that while our current GPG requirements and manual = Bugzilla usage seems like a "waste of time", that time requirement pales = in comparison to the QA approval process. fedora-submit and improved = command line tools integrated with GPG would speed up the Bugzilla = report communication process would be a plus to that process. BTW, your fedora-submit tools may be good for our RPM development = toolkit package "fedora-rpmdevtools" which is briefly documented here: http://www.fedora.us/wiki/FedoraRPMDevTools Warren Togami warren(a)togami.com --===============1475717851112816145==-- From esr at thyrsus.com Mon Sep 29 15:53:50 2003 Content-Type: multipart/mixed; boundary="===============4915949815782495952==" MIME-Version: 1.0 From: Eric S. Raymond To: devel at lists.fedoraproject.org Subject: Re: Showstopper in the RPM submission procesdure Date: Mon, 29 Sep 2003 15:57:33 -0400 Message-ID: <20030929195733.GG11899@thyrsus.com> In-Reply-To: 3F78168A.4050508@togami.com --===============4915949815782495952== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Warren Togami : > Your suggestion of a fedora-submit script sounds like a good way to = > reduce overhead in submitting Bugzilla reports for packages. I am glad = > somebody finally wants to implement it. I suggest to you to read = > through many of the fedora.us reports to see the steps involved in this = > process. Your script would need to (off the top of my head) > = > 1) Check for duplicates. > 2) Submit a new report for a package, along with GPG signatures. > 3) Submit revised packages along with new GPG signatures after some = > discussion points out flaws. > 4) Upload packages to *somewhere* which we still haven't worked out. We = > obviously cannot give CVS access to everyone, and there may be issues = > with a public upload location at an official server. I much prefer the = > GPG signatures + submitter controlled URL process that fedora.us = > currently uses, but perhaps a future process may have a hybrid. The = > more senior trusted packagers have upload access to an official staging = > area or CVS, while everyone else uses GPG + URLs. On point 1, the design of fedora-submit should not try to wire in repository policy. In particular, putting dup checking in there would probably be a bad idea, as your concept of a dup is likely to evolve over time. For example, is it a dup when the same-named RPM is submitted for two different trees? You want to be able to change the answers to questions like that without replacing every client instance in the world. Points 2 and 3 look right to me. As a toolbuilder I'm agnostic on point 4, but I share your preference. There's no reason for your server to have to pay storage costs for RPMs not yet accepted when a URL pointer can be functionally equivalent to having a copy. = > Fortunately now that we are no longer an unaffilited 3rd party, I = > believe we no longer need the many weird corner case naming = > requirements. The naming document can be greatly simplified, dropping = > the leading "X.fdr." part of the %release tag and all related naming = > policies because they are not needed any longer. Good. Then by all means nuke 'em. = > In the mean time, please submit your packages to fedora.us in any matter = > that you wish. Don't worry if you don't understand the overly complex = > naming guidelines, because the QA people will quickly point out errors. OK. I'll write fedora-submit and do some testing, then. I think I've essentially finished the bug-bugzilla tool that the first implementation of fedora-submit will use; it's now in evaluation and testing with Christian Reis of the Bogzilla project. = > BTW, your fedora-submit tools may be good for our RPM development = > toolkit package "fedora-rpmdevtools" which is briefly documented here: > http://www.fedora.us/wiki/FedoraRPMDevTools Ahh, that's good to know. Yes, that is obviously the right place. = I see I even got the naming convention right :-). -- = Eric S. Raymond --===============4915949815782495952==--