cobbler reposync download / upload rate limit

[Lanton Hulisani Vhengani]

Hi

I was wondering if there is a way of passing rsync options for cobbler
reposync. I am particularly interested in the rsync's --bwlimit=KBPS
option. I would like to limit the transfer rate to a minimun so that
repository downloads should not impact our network.

And if not possible, I have noticed that Cobbler generates commands such
as rsync -avz <site>:/var/www/cobbler/repo_mirror /var/www/cobbler
when performing a reposync. May someone please point me to where the
-avz options are set?

Thanks in advance,
Lanton

[mh]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi

>  I was wondering if there is a way of passing rsync options for
> cobbler
>  reposync. I am particularly interested in the rsync's --bwlimit=KBPS
>  option. I would like to limit the transfer rate to a minimun so that
>  repository downloads should not impact our network.
>  
>  And if not possible, I have noticed that Cobbler generates commands such
>  as rsync -avz <site>:/var/www/cobbler/repo_mirror /var/www/cobbler
>  when performing a reposync. May someone please point me to where the
>  -avz options are set? 
cobbler calls reposync and afaik reposync doesn't support passing
additional rsync options to it and the -avz is hardcoded in the reposync
code.

cheers Marcel
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (GNU/Linux)
> Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
>
> iEYEARECAAYFAklPbfMACgkQxqnHThA4pEm20gCg2r0fqet2DMgkhyUG/FmvHGXW
> YLIAnA3Ndd6i70784RpTz8aS+CjcXHEe
> =pU7c
> -----END PGP SIGNATURE-----

[Michael DeHaan]

haerry+et(a)puzzle.ch wrote:
>  -----BEGIN PGP SIGNED MESSAGE-----
>  Hash: SHA1
>
>  Hi
>
>    
> > I was wondering if there is a way of passing rsync options for cobbler
> > reposync. I am particularly interested in the rsync's --bwlimit=KBPS
> > option. I would like to limit the transfer rate to a minimun so that
> > repository downloads should not impact our network.
> >
> > And if not possible, I have noticed that Cobbler generates commands such
> > as rsync -avz <site>:/var/www/cobbler/repo_mirror /var/www/cobbler
> > when performing a reposync. May someone please point me to where the
> > -avz options are set?
> >     
>
>  cobbler calls reposync and afaik reposync doesn't support passing
>  additional rsync options to it and the -avz is hardcoded in the reposync
>  code.
>     
I've considered adding --rsync-flags to 'cobbler repo add'.

(FYI -- if you were concerned about internal bandwith, "cobbler repo 
edit --yum-opts" is what you'd want to use, though this does not apply 
to mirroring, and especially not to rsync)


>  cheers Marcel
> >  -----BEGIN PGP SIGNATURE-----
> >  Version: GnuPG v1.4.9 (GNU/Linux)
> >  Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
> >
> >  iEYEARECAAYFAklPbfMACgkQxqnHThA4pEm20gCg2r0fqet2DMgkhyUG/FmvHGXW
> >  YLIAnA3Ndd6i70784RpTz8aS+CjcXHEe
> >  =pU7c
> >  -----END PGP SIGNATURE----- 
>  _______________________________________________
>  cobbler mailing list
>  cobbler(a)lists.fedorahosted.org
>  https://fedorahosted.org/mailman/listinfo/cobbler
>

[Lanton Hulisani Vhengani]

Hi Marcel

I located the following code:

<CODE>
def rsync_it(self,from_path,to_path):
        from_path = "%s:%s" % (self.host, from_path)
        cmd = "rsync -avz %s %s" % (from_path, to_path)
        print _("- %s") % cmd
        rc = sub_process.call(cmd, shell=True)
        if rc !=0:
            raise CX(_("rsync failed"))
   
def scp_it(self,from_path,to_path):
        from_path = "%s:%s" % (self.host, from_path)
        cmd = "scp %s %s" % (from_path, to_path)
        print _("- %s") % cmd
        rc = sub_process.call(cmd, shell=True)
        if rc !=0:
            raise CX(_("scp failed"))
<CODE>
from /usr/lib/python2.4/site-packages/cobbler/action_replicate.py

I would like to alter it to:
<CODE>
def rsync_it(self,from_path,to_path):
        from_path = "%s:%s" % (self.host, from_path)
        cmd = "rsync -avz --bwlimit=KBPS %s %s" % (from_path, to_path)
        print _("- %s") % cmd
        rc = sub_process.call(cmd, shell=True)
        if rc !=0:
            raise CX(_("rsync failed"))
   
def scp_it(self,from_path,to_path):
        from_path = "%s:%s" % (self.host, from_path)
        cmd = "scp -l KBPS %s %s" % (from_path, to_path)
        print _("- %s") % cmd
        rc = sub_process.call(cmd, shell=True)
        if rc !=0:
            raise CX(_("scp failed"))
<CODE>

Is this advisable? But does address my problem.

Regards
Lanton


On Mon, 2008-12-22 at 11:37 +0100, haerry+et(a)puzzle.ch wrote:
>  -----BEGIN PGP SIGNED MESSAGE-----
>  Hash: SHA1
>  
>  Hi
>  
>  > I was wondering if there is a way of passing rsync options for cobbler
>  > reposync. I am particularly interested in the rsync's --bwlimit=KBPS
>  > option. I would like to limit the transfer rate to a minimun so that
>  > repository downloads should not impact our network.
>  > 
>  > And if not possible, I have noticed that Cobbler generates commands such
>  > as rsync -avz <site>:/var/www/cobbler/repo_mirror /var/www/cobbler
>  > when performing a reposync. May someone please point me to where the
>  > -avz options are set?
>  
>  cobbler calls reposync and afaik reposync doesn't support passing
>  additional rsync options to it and the -avz is hardcoded in the reposync
>  code.
>  
>  cheers Marcel
> >  -----BEGIN PGP SIGNATURE-----
> >  Version: GnuPG v1.4.9 (GNU/Linux)
> >  Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
> >  
> >  iEYEARECAAYFAklPbfMACgkQxqnHThA4pEm20gCg2r0fqet2DMgkhyUG/FmvHGXW
> >  YLIAnA3Ndd6i70784RpTz8aS+CjcXHEe
> >  =pU7c
> >  -----END PGP SIGNATURE----- 
>

[Michael DeHaan]

Lanton Hulisani Vhengani wrote:
>  Hi Marcel
>
>  I located the following code:
>
>  <CODE>
>  def rsync_it(self,from_path,to_path):
>          from_path = "%s:%s" % (self.host, from_path)
>          cmd = "rsync -avz %s %s" % (from_path, to_path)
>          print _("- %s") % cmd
>          rc = sub_process.call(cmd, shell=True)
>          if rc !=0:
>              raise CX(_("rsync failed"))
>     
>  def scp_it(self,from_path,to_path):
>          from_path = "%s:%s" % (self.host, from_path)
>          cmd = "scp %s %s" % (from_path, to_path)
>          print _("- %s") % cmd
>          rc = sub_process.call(cmd, shell=True)
>          if rc !=0:
>              raise CX(_("scp failed"))
>  <CODE>
>  from /usr/lib/python2.4/site-packages/cobbler/action_replicate.py
>
>  I would like to alter it to:
>  <CODE>
>  def rsync_it(self,from_path,to_path):
>          from_path = "%s:%s" % (self.host, from_path)
>          cmd = "rsync -avz --bwlimit=KBPS %s %s" % (from_path, to_path)
>          print _("- %s") % cmd
>          rc = sub_process.call(cmd, shell=True)
>          if rc !=0:
>              raise CX(_("rsync failed"))
>     
>  def scp_it(self,from_path,to_path):
>          from_path = "%s:%s" % (self.host, from_path)
>          cmd = "scp -l KBPS %s %s" % (from_path, to_path)
>          print _("- %s") % cmd
>          rc = sub_process.call(cmd, shell=True)
>          if rc !=0:
>              raise CX(_("scp failed"))
>  <CODE>
>
>  Is this advisable? But does address my problem.
>
>  Regards
>  Lanton
>     
Longer term I'd think we'd rather add --rsync-flags to "cobbler repo 
add" to generically solve the problem of not being able to add flags (or 
at least to /etc/cobbler/settings), but there's nothing wrong with you 
modifying things yourself provided it solves things for you. The next 
time you update the RPM you'd have to make the change again, which is 
the main drawback.

--Michael