Hello,
I have been spoken to by Marcela Mašláňová about "The Future of FTBFS". See this thread:
http://lists.fedoraproject.org/pipermail/devel/2011-April/150310.html
IIUC (Is there an abbreviation for "I'm not a developer"?) the problem is as follows:
* Matt Domsch from Dell used to rebuild *all* packages from Rawhide periodically (so-called "mass rebuild"). When some package failed to build, he reported errors against that package.
* This testing ensured we often find build problems early in the release process. Without it there is a chance that we discover the build failures only when a new build of that package is required, which may be shortly before final release or even after that. That's a problem.
* Mass-rebuilds in Koji are not done frequently (maybe once a year), so they can't cover this issue.
* Matt can't do this testing anymore. Marcela asked me whether AutoQA could be used for that. Matt's tools (scripts, etc) should be available.
* I asked Marcela to inquire more about some details. I have attached the discussion below (read from bottom up).
What are your thoughts? Is that something AutoQA can and should handle? Do we (will we) have enough hardware to be able to do that? According to our current priorities, is that even something we are able to implement in some reasonable time (under a year)?
As for the last question, I think it clearly fits our current effort to provide generic Fedora-related tests. OTOH we still have many generic tests to finish (either un-started or semi-finished) and before that we need to concentrate on architecture first (ResultDB etc.). I'm afraid to have complex tests running without solid architecture basis beneath it. In that respect unless we all agree this is a top-priority next-to-work-on test (and provided that we have enough hardware for it) I don't think we're able to run it soon.
Do we need some more information I should ask Matt for?
Thanks, Kamil
----- Forwarded Message ----- From: "Matt Domsch" Matt_Domsch@Dell.com To: mmaslano@redhat.com Cc: kparal@redhat.com, skvidal@fedoraproject.org Sent: Tuesday, April 12, 2011 7:30:14 PM Subject: RE: future of FTBFS
Seth was asking me the same question.
My environment consists of: Builders: 10 PowerEdge 1955 servers, each with 2x4core 3.0GHz CPUs, 8GB RAM, 2x144GB disks. Disk space on the builders is mostly used for the buildroots, and swap for large (or several parallel) jobs that can exceed 8GB in a tmpfs buildroot.
http and NFS server with space for the current rawhide tree (daily rsync), a hardlinked copy of the rawhide tree from the day the build starts (initially zero space, but growing to the size of the full rawhide tree as rawhide moves on), and space for the newly built tree results to land. ~250GB total.
One more server as the "master" that kicks off all the jobs to the builders. This can be anything, technically even one of the builders.
It takes this setup ~30 hours to build all 10,000 SRPMs twice, once for each of i386 and x86_64. That was before the tmpfs change in F14, which prevents mock from using tmpfs for its buildroots. Now it takes ~96 hours with disk-backed buildroots.
In my setup, each builder runs 4 jobs concurrently, two for each architecture. They're mostly I/O-bound, hence the disk-backed buildroots are so much slower. There is often plenty of memory and CPU left over; not always (depends on the size of the jobs that happen to be handed to each builder concurrently) - sometimes they're CPU-bound, but not mostly.
Seth was going to look into using cloud-based builders. I think this is a great idea, provided you have a place to store the build results outside of the builders themselves, and have network-local copy of the SRPM tree you're starting from and a copy of the buildroot repositories network-local too.
Thanks for your interest in taking this on! -Matt
-- Matt Domsch Technology Strategist Dell | Office of the CTO
-----Original Message----- From: Marcela Mašláňová [mailto:mmaslano@redhat.com] Sent: Tuesday, April 12, 2011 4:02 AM To: Domsch, Matt Cc: kparal@redhat.com Subject: future of FTBFS
Hello Matt, I was speaking with Fedora QA (Kamil Páral) about FTBFS. It might be possible to run it as one of project of QA, but they'd like to know some details.
What are the hw requirements (disk space, number of machines used to run it, how long it takes (days?), is it needed to have installed rawhide)?
Best regards, Marcela
-- Marcela Mašláňová BaseOS team Brno