How to remove the large directory after build ?

[Jian Lee]

Hello,everybody!

My koji server has setup ok. but the mock dir is more and more large. How to
remove it ?
I mean if koji has this feature,I doesn't to do it myself. Is anyone can
help me ?

my mock directory info :
------------------------------------
651M    gtes11.2-build-10-9
18M     gtes11.2-build-13-10
5.6M    gtes11.2-build-15-15
549M    gtes11.2-build-1-6
5.5M    gtes11.2-build-16-15
18M     gtes11.2-build-18-18
18M     gtes11.2-build-19-18
579M    gtes11.2-build-21-20
560M    gtes11.2-build-22-20
3.5G    gtes11.2-build-23-26
548M    gtes11.2-build-2-6
567M    gtes11.2-build-3-6
583M    gtes11.2-build-5-9
562M    gtes11.2-build-6-9
670M    gtes11.2-build-9-9
6.2M    gtes11.3-build-24-28
6.1M    gtes11.3-build-25-28
637M    gtes11.3-build-26-30
609M    gtes11.3-build-28-30
3.5G    gtes11.3-build-30-30
3.0G    gtes11.3-build-31-30
1.8G    gtes11.3-build-32-32
1.8G    gtes11.3-build-34-32
583M    gtes11.3-build-35-33
562M    gtes11.3-build-37-33
888M    gtes11.3-build-39-34
852M    gtes11.3-build-40-34
408M    moblin2-build-41-38
417M    moblin2-build-42-39
==================================

Thanks!

[Mike Bonnet]

李建 wrote:
> Hello,everybody!
>
> My koji server has setup ok. but the mock dir is more and more large. How to
> remove it ?
> I mean if koji has this feature,I doesn't to do it myself. Is anyone can
> help me ?
> my mock directory info :
>
> 651M    gtes11.2-build-10-9
> 18M     gtes11.2-build-13-10
> 5.6M    gtes11.2-build-15-15
> 549M    gtes11.2-build-1-6
> 5.5M    gtes11.2-build-16-15
> 18M     gtes11.2-build-18-18
> 18M     gtes11.2-build-19-18
> 579M    gtes11.2-build-21-20
> 560M    gtes11.2-build-22-20
> 3.5G    gtes11.2-build-23-26
> 548M    gtes11.2-build-2-6
> 567M    gtes11.2-build-3-6
> 583M    gtes11.2-build-5-9
> 562M    gtes11.2-build-6-9
> 670M    gtes11.2-build-9-9
> 6.2M    gtes11.3-build-24-28
> 6.1M    gtes11.3-build-25-28
> 637M    gtes11.3-build-26-30
> 609M    gtes11.3-build-28-30
> 3.5G    gtes11.3-build-30-30
> 3.0G    gtes11.3-build-31-30
> 1.8G    gtes11.3-build-32-32
> 1.8G    gtes11.3-build-34-32
> 583M    gtes11.3-build-35-33
> 562M    gtes11.3-build-37-33
> 888M    gtes11.3-build-39-34
> 852M    gtes11.3-build-40-34
> 408M    moblin2-build-41-38
> 417M    moblin2-build-42-39
> ==================================

kojid manages the cleanup up the /var/lib/mock directory for builds it 
runs.  Buildroots for successful builds will be cleaned up almost 
immediately.  Buildroots for unsuccessful builds will be cleaned up 
after 4 hours, to give you a chance to debug the failure or copy 
relevant data out of the buildroot for later analysis.

[Jian Lee]

Now ,the directory is:
-----------------------------------------
[root@localhost mock]# du -hs *
651M    gtes11.2-build-10-9
18M     gtes11.2-build-13-10
5.6M    gtes11.2-build-15-15
549M    gtes11.2-build-1-6
5.5M    gtes11.2-build-16-15
18M     gtes11.2-build-18-18
18M     gtes11.2-build-19-18
579M    gtes11.2-build-21-20
560M    gtes11.2-build-22-20
3.5G    gtes11.2-build-23-26
548M    gtes11.2-build-2-6
567M    gtes11.2-build-3-6
583M    gtes11.2-build-5-9
562M    gtes11.2-build-6-9
670M    gtes11.2-build-9-9
6.2M    gtes11.3-build-24-28
6.1M    gtes11.3-build-25-28
637M    gtes11.3-build-26-30
609M    gtes11.3-build-28-30
3.5G    gtes11.3-build-30-30
3.0G    gtes11.3-build-31-30
1.8G    gtes11.3-build-32-32
1.8G    gtes11.3-build-34-32
583M    gtes11.3-build-35-33
562M    gtes11.3-build-37-33
888M    gtes11.3-build-39-34
852M    gtes11.3-build-40-34
408M    moblin2-build-41-38
417M    moblin2-build-42-39
539M    moblin2-build-43-39
==========================

My builder is ok:
--------------------------------------------------------------------------------
[root@localhost mock]# koji list-hosts
Hostname                     Enb Rdy Load/Cap Arches           Last Update
kojibuilder1                 N   N    0.0/2.0 i386,x86_64      2009-05-25
19:08:25
kojibuilder2                 Y   Y    0.0/2.0 ia64             2009-05-29
14:18:59
kojibuilder3                 Y   Y    0.0/2.0 x86_64,i386      2009-06-02
10:09:00
==============================================================

Why ? Because I use a special disk space,and I use Selinux, So I modified
the /var/lib/mock to /dist/mock on kojihub and kojiweb and kojibuilder3
host. but the kojibuilder2's mock directory is /var/lib/mock ,a link to
/dist/mock on it's host.

the kojihub and kojiweb and kojibuilder3's /mnt/koji directory is modified
to /data/koji also . but the other kojibuilder*'s directory is /mnt/koji by
nfs mount.

Why ? It isn't change or removed after 1 days.

Thanks!

2009/6/1 Mike Bonnet
> 李建 wrote:
> > Hello,everybody!
> >
> > My koji server has setup ok. but the mock dir is more and more large. How
> > to
> > remove it ?
> > I mean if koji has this feature,I doesn't to do it myself. Is anyone can
> > help me ?
> > my mock directory info :
> >
> > 651M    gtes11.2-build-10-9
> > 18M     gtes11.2-build-13-10
> > 5.6M    gtes11.2-build-15-15
> > 549M    gtes11.2-build-1-6
> > 5.5M    gtes11.2-build-16-15
> > 18M     gtes11.2-build-18-18
> > 18M     gtes11.2-build-19-18
> > 579M    gtes11.2-build-21-20
> > 560M    gtes11.2-build-22-20
> > 3.5G    gtes11.2-build-23-26
> > 548M    gtes11.2-build-2-6
> > 567M    gtes11.2-build-3-6
> > 583M    gtes11.2-build-5-9
> > 562M    gtes11.2-build-6-9
> > 670M    gtes11.2-build-9-9
> > 6.2M    gtes11.3-build-24-28
> > 6.1M    gtes11.3-build-25-28
> > 637M    gtes11.3-build-26-30
> > 609M    gtes11.3-build-28-30
> > 3.5G    gtes11.3-build-30-30
> > 3.0G    gtes11.3-build-31-30
> > 1.8G    gtes11.3-build-32-32
> > 1.8G    gtes11.3-build-34-32
> > 583M    gtes11.3-build-35-33
> > 562M    gtes11.3-build-37-33
> > 888M    gtes11.3-build-39-34
> > 852M    gtes11.3-build-40-34
> > 408M    moblin2-build-41-38
> > 417M    moblin2-build-42-39
> > ==================================
>
> kojid manages the cleanup up the /var/lib/mock directory for builds it
> runs.  Buildroots for successful builds will be cleaned up almost
> immediately.  Buildroots for unsuccessful builds will be cleaned up after 4
> hours, to give you a chance to debug the failure or copy relevant data out
> of the buildroot for later analysis.
>
> --
> Fedora-buildsys-list mailing list
> [e-mail]
> https://www.redhat.com/mailman/listinfo/fedora-buildsys-list

[Mike McLean]

李建 wrote:
> Now ,the directory is:
>
> [root@localhost mock]# du -hs *
> 651M    gtes11.2-build-10-9
> 18M     gtes11.2-build-13-10
> 5.6M    gtes11.2-build-15-15
> 549M    gtes11.2-build-1-6
> 5.5M    gtes11.2-build-16-15
> 18M     gtes11.2-build-18-18
> 18M     gtes11.2-build-19-18

Due to a potentially dangerous bug in rpmlib [1], kojid removes the 
mockdir directories in two stages.

First the buildroot (e.g. gtes11.2-build-16-15/root)is cleared of 
content, but the directory itself (as well the mock results dir) 
remains. This clearing occurs almost immediately after most builds, but 
is delayed four hours for failed builds.

After 24 hours, the entire mock directory for the chroot is removed. 
This allows any stray rpm transactions plenty of time to exit. This may 
seem like an extreme wait, but the consequences of triggering the rpm 
bug are very, very bad.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=192153

It appears that at least some of these directories (e.g. the 18M ones) 
are still in that 24 hour wait. As for the others, I cannot be sure from 
this small amount of data.

...
> Why ? Because I use a special disk space,and I use Selinux, So I modified
> the /var/lib/mock to /dist/mock on kojihub and kojiweb and kojibuilder3
> host. but the kojibuilder2's mock directory is /var/lib/mock ,a link to
> /dist/mock on it's host.
>
> the kojihub and kojiweb and kojibuilder3's /mnt/koji directory is modified
> to /data/koji also . but the other kojibuilder*'s directory is /mnt/koji by
> nfs mount.
>
> Why ? It isn't change or removed after 1 days.

As pointed out above, it is normal for koji's mock directories to 
survive (in a stripped form) for up to 24 hours. If the problem 
persists, I recommend cranking up the debugging on kojid and watching 
the logs (look for lines containing rootdir or buildroot).

[Jian Lee]

Mike McLean  writes:

There were much work to do recently, I haven't resolved this question. I
just remove those large rootdir manually. It's too silly. :)

I've see the kojid source code :

-----------------------------------------------------------------------------------------
 741                 #note: https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=192153)
 742                 #If rpmlib is installing in this chroot, removing it entirely
 743                 #can lead to a world of hurt.
 744                 #We remove the rootdir contents but leave the rootdir unless it
 745                 #is really old
 746                 if age > 3600*24:
 747                     #dir untouched for a day
 748                     self.logger.info("Removing buildroot: %s" % desc)
 749                     if topdir and safe_rmtree(topdir, unmount=True, strict=False) != 0:
 750                         continue
 751                     #also remove the config
 752                     try:
 753                         os.unlink(data['cfg'])
 754                     except OSError, e:
 755                         self.logger.warn("%s: can't remove config: %s" % (desc, e))
 756                 elif age > 120:
 757                     if rootdir:
 758                         try:
 759                             flist = os.listdir(rootdir)
 760                         except OSError, e:
 761                             self.logger.warn("%s: can't list rootdir: %s" % (desc, e))
 762                             continue
 763                         if flist:                                                                                    
 764                             self.logger.info("%s: clearing rootdir" % desc)                                          
 765                         for fn in flist:                                                                             
 766                             safe_rmtree("%s/%s" % (rootdir,fn), unmount=True, strict=False)
 767                 else:
 768                     self.logger.debug("Recent buildroot: %s: %i seconds" % (desc,age))
==============================================================================================

The 3600*24 must be mean 24 hours, but the 120 mean what ?

and , There are code following:

-----------------------------------
 801             # see if there's a dir for the buildroot                                               
 802             vardir = "/var/lib/mock/%s" % name                                                   
 803             #XXX
 804             buildroots[id] = {}
 805             buildroots[id]['name'] = name
 806             buildroots[id]['cfg'] = fn
 807             buildroots[id]['dir'] = None
 808             if os.path.isdir(vardir):
 809                 buildroots[id]['dir'] = vardir
 810         return buildroots
===============================================================

I've not use the /var/lib/mock, and I's not a link to /dist/mock. My
kojid.conf have following configure:

--------------------------------
; The directory root where work data can be found from the koji hub
topdir=/data/koji

; The directory root for temporary storage
workdir=/tmp/koji

; The directory root for mock
mockdir=/dist/mock

; The user to run as when doing builds
mockuser=kojibuilder
==============================================

Is the /var/lib/mock must exits and point to the mockdir ? And wheather
the kojid remove chrootdir use mockuser privilige ?

Thanks,
> 李建 wrote:
> > Now ,the directory is:
> >
> > [root@localhost mock]# du -hs *
> > 651M    gtes11.2-build-10-9
> > 18M     gtes11.2-build-13-10
> > 5.6M    gtes11.2-build-15-15
> > 549M    gtes11.2-build-1-6
> > 5.5M    gtes11.2-build-16-15
> > 18M     gtes11.2-build-18-18
> > 18M     gtes11.2-build-19-18
>
> Due to a potentially dangerous bug in rpmlib [1], kojid removes the
> mockdir directories in two stages.
>
> First the buildroot (e.g. gtes11.2-build-16-15/root)is cleared of
> content, but the directory itself (as well the mock results dir)
> remains. This clearing occurs almost immediately after most builds,
> but is delayed four hours for failed builds.
>
> After 24 hours, the entire mock directory for the chroot is removed.
> This allows any stray rpm transactions plenty of time to exit. This
> may seem like an extreme wait, but the consequences of triggering the
> rpm bug are very, very bad.
>
> [1] https://bugzilla.redhat.com/show_bug.cgi?id=192153
>
> It appears that at least some of these directories (e.g. the 18M ones)
> are still in that 24 hour wait. As for the others, I cannot be sure
> from this small amount of data.
>
> ...
> > Why ? Because I use a special disk space,and I use Selinux, So I modified
> > the /var/lib/mock to /dist/mock on kojihub and kojiweb and kojibuilder3
> > host. but the kojibuilder2's mock directory is /var/lib/mock ,a link to
> > /dist/mock on it's host.
> >
> > the kojihub and kojiweb and kojibuilder3's /mnt/koji directory is modified
> > to /data/koji also . but the other kojibuilder*'s directory is /mnt/koji by
> > nfs mount.
> >
> > Why ? It isn't change or removed after 1 days.
>
> As pointed out above, it is normal for koji's mock directories to
> survive (in a stripped form) for up to 24 hours. If the problem
> persists, I recommend cranking up the debugging on kojid and watching
> the logs (look for lines containing rootdir or buildroot).
>
> --
> Fedora-buildsys-list mailing list
> [e-mail]
> https://www.redhat.com/mailman/listinfo/fedora-buildsys-list