Faster builds

[Michael Schwendt]

Some good news for package maintainers with many sub-packages:

Plague used to sleep at least 20 seconds for each (!) file it had to
download from a builder. Even downloading the several small log files used
to take at least 20 seconds *each*.

Reading and debugging the code, I found the source of the problem and
taught the build server to sleep less when files are being downloaded.
Interestingly, the speedup is noticeable even for tiny build-jobs, e.g.

  revisor down from 7m to 4m
  nagios down from 6m to 4m

but it will be incredible for build-jobs with a larger number of pkgs
(e.g. try moodle or nagios-plugins which are in EPEL already). That
means, such packagers get the build-results and the mail notification
much faster.

[Jeff Sheltren]

On Jan 3, 2008, at 5:59 PM, Michael Schwendt wrote:

>  Some good news for package maintainers with many sub-packages:
>
>  Plague used to sleep at least 20 seconds for each (!) file it had to
>  download from a builder. Even downloading the several small log  
>  files used
>  to take at least 20 seconds *each*.
>
>  Reading and debugging the code, I found the source of the problem and
>  taught the build server to sleep less when files are being downloaded.
>  Interestingly, the speedup is noticeable even for tiny build-jobs,  
>  e.g.
>
>   revisor down from 7m to 4m
>   nagios down from 6m to 4m
>
>  but it will be incredible for build-jobs with a larger number of pkgs
>  (e.g. try moodle or nagios-plugins which are in EPEL already). That
>  means, such packagers get the build-results and the mail notification
>  much faster.
>  
Michael, that's very cool.  This has always been something that bugged  
me when using plague, but I never got around to figuring out why the  
"downloading" code was so slow, even locally.  Would you mind posting  
a patch for plague with your changes?

Thanks,
Jeff

[Michael Schwendt]

On Thu, 3 Jan 2008 19:07:49 -0800, Jeff Sheltren wrote:

>  On Jan 3, 2008, at 5:59 PM, Michael Schwendt wrote:
>  
>  > Some good news for package maintainers with many sub-packages:
>  >
>  > Plague used to sleep at least 20 seconds for each (!) file it had to
>  > download from a builder. Even downloading the several small log  
>  > files used
>  > to take at least 20 seconds *each*.
>  >
>  > Reading and debugging the code, I found the source of the problem and
>  > taught the build server to sleep less when files are being downloaded.
>  > Interestingly, the speedup is noticeable even for tiny build-jobs,  
>  > e.g.
>  >
>  >  revisor down from 7m to 4m
>  >  nagios down from 6m to 4m
>  >
>  > but it will be incredible for build-jobs with a larger number of pkgs
>  > (e.g. try moodle or nagios-plugins which are in EPEL already). That
>  > means, such packagers get the build-results and the mail notification
>  > much faster.
>  >
>  
>  Michael, that's very cool.  This has always been something that bugged  
>  me when using plague, but I never got around to figuring out why the  
>  "downloading" code was so slow, even locally.  Would you mind posting  
>  a patch for plague with your changes? 
The less aggressive [*] patch I've applied is this:
http://mschwendt.fedorapeople.org/plague-0.4.4.1-faster-downloads.patch

With it, perl-Inline built in 1 minute while revisor built in 2 mins:
http://buildsys.fedoraproject.org/build-status/job.psp?uid=37772
http://buildsys.fedoraproject.org/build-status/job.psp?uid=37771

-- 
[*] "Less aggressive" with regard to the trade-off between polling
and sleeping.

[Jon Ciesla]

>  On Jan 3, 2008, at 5:59 PM, Michael Schwendt wrote: 
>  but it will be incredible for build-jobs with a larger number of
> pkgs
>  (e.g. try moodle or nagios-plugins which are in EPEL already). That
>  means, such packagers get the build-results and the mail notification
>  much faster.
>  
The Moodle maintainer thanks you! :))))))

-- 
novus ordo absurdum

[Thorsten Leemhuis]

On 04.01.2008 02:59, Michael Schwendt wrote:
>  Some good news for package maintainers with many sub-packages:
>  [...] 
Many thanks for this work Michael. Ohh, any many thx also for those
other changes you did to the system where plague-server is running.

CU
knurd