Cobbler sync very slow, timing out with multiple interfaces

[Chris Weaver]

I am working with the bonded interfaces option for 1.4.  For one server it
works great, I can add two bonded interfaces and a couple of non-bonded
ones.  Everything comes up rosy.

However cobbler sync is taking about 8 seconds when I have one interface,
and about 20 seconds with six.  I add another system with another 6
interfaces, it takes 3 minutes and times out.  If I delete an interface from
each system, sync takes around 2 minutes but at least it succeeds.  It is
definitely related to dhcp since if I disable that in settings, sync comes
back right away.

This is the error I get (after 3 minutes):

ng00ud01# cobbler sync

join() is too long for a Python string
  File "/usr/lib/python2.3/site-packages/cobbler/cobbler.py", line 83, in
main
    rc = BootCLI().run(sys.argv)

  File "/usr/lib/python2.3/site-packages/cobbler/cobbler.py", line 61, in
run
    return self.loader.run(args)

  File "/usr/lib/python2.3/site-packages/cobbler/commands.py", line 122, in
run
    return fn.run()

  File "/usr/lib/python2.3/site-packages/cobbler/modules/cli_misc.py", line
183, in run
    return self.api.sync()

  File "/usr/lib/python2.3/site-packages/cobbler/api.py", line 523, in sync
    return sync.run()

  File "/usr/lib/python2.3/site-packages/cobbler/action_sync.py", line 101,
in run
    self.dhcp.write_dhcp_file()

  File "/usr/lib/python2.3/site-packages/cobbler/modules/manage_isc.py",
line 262, in write_dhcp_file
    self.templar.render(template_data, metadata, settings_file, None)

  File "/usr/lib/python2.3/site-packages/cobbler/templar.py", line 133, in
render
    data_out = data_out.replace("@@%s@@" % str(x),
str(search_table[str(x)]))


Any ideas?? TIA...

-- 
... Chris Weaver

[Steven Wagner]

Inspired by this message, I tried turning off manage_dhcp in my settings
and cobbler sync went from taking upwards of 90 seconds to about 5-10
seconds.
 
I don't use any bonded interfaces, but I do have one or two multihomed
boxes being tracked by cobbler...


________________________________

	From: cobbler-bounces(a)lists.fedorahosted.org
[mailto:[e-mail]] On Behalf Of Chris
Weaver
	Sent: Wednesday, December 31, 2008 9:06 AM
	To: cobbler(a)lists.fedorahosted.org
	Subject: Cobbler sync very slow, timing out with multiple
interfaces
	
	
	I am working with the bonded interfaces option for 1.4.  For one
server it works great, I can add two bonded interfaces and a couple of
non-bonded ones.  Everything comes up rosy.
	
	However cobbler sync is taking about 8 seconds when I have one
interface, and about 20 seconds with six.  I add another system with
another 6 interfaces, it takes 3 minutes and times out.  If I delete an
interface from each system, sync takes around 2 minutes but at least it
succeeds.  It is definitely related to dhcp since if I disable that in
settings, sync comes back right away.
	
	This is the error I get (after 3 minutes):
	
	ng00ud01# cobbler sync
	
	join() is too long for a Python string
	  File "/usr/lib/python2.3/site-packages/cobbler/cobbler.py",
line 83, in main
	    rc = BootCLI().run(sys.argv)
	
	  File "/usr/lib/python2.3/site-packages/cobbler/cobbler.py",
line 61, in run
	    return self.loader.run(args)
	
	  File "/usr/lib/python2.3/site-packages/cobbler/commands.py",
line 122, in run
	    return fn.run()
	
	  File
"/usr/lib/python2.3/site-packages/cobbler/modules/cli_misc.py", line
183, in run
	    return self.api.sync()
	
	  File "/usr/lib/python2.3/site-packages/cobbler/api.py", line
523, in sync
	    return sync.run()
	
	  File
"/usr/lib/python2.3/site-packages/cobbler/action_sync.py", line 101, in
run
	    self.dhcp.write_dhcp_file()
	
	  File
"/usr/lib/python2.3/site-packages/cobbler/modules/manage_isc.py", line
262, in write_dhcp_file
	    self.templar.render(template_data, metadata, settings_file,
None)
	
	  File "/usr/lib/python2.3/site-packages/cobbler/templar.py",
line 133, in render
	    data_out = data_out.replace("@@%s@@" % str(x),
str(search_table[str(x)]))
	
	
	Any ideas?? TIA...
	
	-- 
	... Chris Weaver

[Jeff Schroeder]

On Wed, Dec 31, 2008 at 10:35 AM, Steven Wagner <swagner(a)convera.com&gt; wrote:
>  Inspired by this message, I tried turning off manage_dhcp in my
> settings and
>  cobbler sync went from taking upwards of 90 seconds to about 5-10 seconds. 
Are you using OMAPI or having cobbler write out a static
/etc/dhcpd.conf? This might be the root of your problem whereas
disabling manage_dhcp is a bandaid.

>  I don't use any bonded interfaces, but I do have one or two
> multihomed boxes
>  being tracked by cobbler...
>
>  ________________________________
>  From: cobbler-bounces(a)lists.fedorahosted.org
>  [mailto:[e-mail]] On Behalf Of Chris Weaver
>  Sent: Wednesday, December 31, 2008 9:06 AM
>  To: cobbler(a)lists.fedorahosted.org
>  Subject: Cobbler sync very slow, timing out with multiple interfaces
>
>  I am working with the bonded interfaces option for 1.4.  For one server it
>  works great, I can add two bonded interfaces and a couple of non-bonded
>  ones.  Everything comes up rosy.
>
>  However cobbler sync is taking about 8 seconds when I have one interface,
>  and about 20 seconds with six.  I add another system with another 6
>  interfaces, it takes 3 minutes and times out.  If I delete an interface from
>  each system, sync takes around 2 minutes but at least it succeeds.  It is
>  definitely related to dhcp since if I disable that in settings, sync comes
>  back right away.
>
>  This is the error I get (after 3 minutes):
>
>  ng00ud01# cobbler sync
>
>  join() is too long for a Python string
>    File "/usr/lib/python2.3/site-packages/cobbler/cobbler.py", line 83, in
>  main
>      rc = BootCLI().run(sys.argv)
>
>    File "/usr/lib/python2.3/site-packages/cobbler/cobbler.py", line 61, in
>  run
>      return self.loader.run(args)
>
>    File "/usr/lib/python2.3/site-packages/cobbler/commands.py", line 122, in
>  run
>      return fn.run()
>
>    File "/usr/lib/python2.3/site-packages/cobbler/modules/cli_misc.py", line
>  183, in run
>      return self.api.sync()
>
>    File "/usr/lib/python2.3/site-packages/cobbler/api.py", line 523, in sync
>      return sync.run()
>
>    File "/usr/lib/python2.3/site-packages/cobbler/action_sync.py", line 101,
>  in run
>      self.dhcp.write_dhcp_file()
>
>    File "/usr/lib/python2.3/site-packages/cobbler/modules/manage_isc.py",
>  line 262, in write_dhcp_file
>      self.templar.render(template_data, metadata, settings_file, None)
>
>    File "/usr/lib/python2.3/site-packages/cobbler/templar.py", line 133, in
>  render
>      data_out = data_out.replace("@@%s@@" % str(x),
>  str(search_table[str(x)]))
>
>
>  Any ideas?? TIA...
>
>  --
>  ... Chris Weaver
>
>  _______________________________________________
>  cobbler mailing list
>  cobbler(a)lists.fedorahosted.org
>  https://fedorahosted.org/mailman/listinfo/cobbler
>
>  


-- 
Jeff Schroeder

Don't drink and derive, alcohol and analysis don't mix.
http://www.digitalprognosis.com

[Steven Wagner]

Ah, I should have said... I was using OMAPI back when I first set up
cobbler, but ran into some weird problems with the lease database
getting corrupted.  It's off and has been off for some time.  

So turning off manage_dhcp is, as far as I know, only skipping the
invocation of cheetah to turn my DHCP template file into a valid ISC
dhcpd.conf.

>  -----Original Message-----
>  From: cobbler-bounces(a)lists.fedorahosted.org 
>  [mailto:[e-mail]] On Behalf Of 
>  Jeff Schroeder
>  Sent: Wednesday, December 31, 2008 10:41 AM
>  To: cobbler mailing list
>  Subject: Re: Cobbler sync very slow, timing out with multiple 
>  interfaces
>  
>  On Wed, Dec 31, 2008 at 10:35 AM, Steven Wagner 
>  <swagner(a)convera.com&gt; wrote:
>  > Inspired by this message, I tried turning off manage_dhcp in my 
>  > settings and cobbler sync went from taking upwards of 90 
>  seconds to about 5-10 seconds.
>  
>  Are you using OMAPI or having cobbler write out a static 
>  /etc/dhcpd.conf? This might be the root of your problem 
>  whereas disabling manage_dhcp is a bandaid.
>  
>  > I don't use any bonded interfaces, but I do have one or two 
>  multihomed 
>  > boxes being tracked by cobbler...
>  >
>  > ________________________________
>  > From: cobbler-bounces(a)lists.fedorahosted.org
>  > [mailto:[e-mail]] On Behalf Of Chris 
>  > Weaver
>  > Sent: Wednesday, December 31, 2008 9:06 AM
>  > To: cobbler(a)lists.fedorahosted.org
>  > Subject: Cobbler sync very slow, timing out with multiple interfaces
>  >
>  > I am working with the bonded interfaces option for 1.4.  For one 
>  > server it works great, I can add two bonded interfaces and 
>  a couple of 
>  > non-bonded ones.  Everything comes up rosy.
>  >
>  > However cobbler sync is taking about 8 seconds when I have one 
>  > interface, and about 20 seconds with six.  I add another 
>  system with 
>  > another 6 interfaces, it takes 3 minutes and times out.  If 
>  I delete 
>  > an interface from each system, sync takes around 2 minutes but at 
>  > least it succeeds.  It is definitely related to dhcp since if I 
>  > disable that in settings, sync comes back right away.
>  >
>  > This is the error I get (after 3 minutes):
>  >
>  > ng00ud01# cobbler sync
>  >
>  > join() is too long for a Python string
>  >   File 
>  "/usr/lib/python2.3/site-packages/cobbler/cobbler.py", line 83, 
>  > in main
>  >     rc = BootCLI().run(sys.argv)
>  >
>  >   File 
>  "/usr/lib/python2.3/site-packages/cobbler/cobbler.py", line 61, 
>  > in run
>  >     return self.loader.run(args)
>  >
>  >   File "/usr/lib/python2.3/site-packages/cobbler/commands.py", line 
>  > 122, in run
>  >     return fn.run()
>  >
>  >   File 
>  "/usr/lib/python2.3/site-packages/cobbler/modules/cli_misc.py", 
>  > line 183, in run
>  >     return self.api.sync()
>  >
>  >   File "/usr/lib/python2.3/site-packages/cobbler/api.py", 
>  line 523, in sync
>  >     return sync.run()
>  >
>  >   File 
>  "/usr/lib/python2.3/site-packages/cobbler/action_sync.py", line 
>  > 101, in run
>  >     self.dhcp.write_dhcp_file()
>  >
>  >   File 
>  > "/usr/lib/python2.3/site-packages/cobbler/modules/manage_isc.py",
>  > line 262, in write_dhcp_file
>  >     self.templar.render(template_data, metadata, 
>  settings_file, None)
>  >
>  >   File "/usr/lib/python2.3/site-packages/cobbler/templar.py", line 
>  > 133, in render
>  >     data_out = data_out.replace("@@%s@@" % str(x),
>  > str(search_table[str(x)]))
>  >
>  >
>  > Any ideas?? TIA...
>  >
>  > --
>  > ... Chris Weaver
>  >
>  > _______________________________________________
>  > cobbler mailing list
>  > cobbler(a)lists.fedorahosted.org
>  > https://fedorahosted.org/mailman/listinfo/cobbler
>  >
>  >
>  
>  
>  
>  --
>  Jeff Schroeder
>  
>  Don't drink and derive, alcohol and analysis don't mix.
>  http://www.digitalprognosis.com
>  _______________________________________________
>  cobbler mailing list
>  cobbler(a)lists.fedorahosted.org
>  https://fedorahosted.org/mailman/listinfo/cobbler
>

[Jeff Schroeder]

On Wed, Dec 31, 2008 at 10:48 AM, Steven Wagner <swagner(a)convera.com&gt; wrote:
>  Ah, I should have said... I was using OMAPI back when I first set up
>  cobbler, but ran into some weird problems with the lease database
>  getting corrupted.  It's off and has been off for some time.
>
>  So turning off manage_dhcp is, as far as I know, only skipping the
>  invocation of cheetah to turn my DHCP template file into a valid ISC
>  dhcpd.conf. 
Alright, then this is a michael question : )

-- 
Jeff Schroeder

Don't drink and derive, alcohol and analysis don't mix.
http://www.digitalprognosis.com

[Michael DeHaan]

Steven Wagner wrote:
>  Ah, I should have said... I was using OMAPI back when I first set up
>  cobbler, but ran into some weird problems with the lease database
>  getting corrupted.  It's off and has been off for some time.  
>
>  So turning off manage_dhcp is, as far as I know, only skipping the
>  invocation of cheetah to turn my DHCP template file into a valid ISC
>  dhcpd.conf.
>
>    
> > -----Original Message-----
> > From: cobbler-bounces(a)lists.fedorahosted.org 
> > [mailto:[e-mail]] On Behalf Of 
> > Jeff Schroeder
> > Sent: Wednesday, December 31, 2008 10:41 AM
> > To: cobbler mailing list
> > Subject: Re: Cobbler sync very slow, timing out with multiple 
> > interfaces
> >
> > On Wed, Dec 31, 2008 at 10:35 AM, Steven Wagner 
> > <swagner(a)convera.com&gt; wrote:
> >     
> >> Inspired by this message, I tried turning off manage_dhcp in my 
> >> settings and cobbler sync went from taking upwards of 90 
> >>       
> > seconds to about 5-10 seconds.
> >
> > Are you using OMAPI or having cobbler write out a static 
> > /etc/dhcpd.conf? This might be the root of your problem 
> > whereas disabling manage_dhcp is a bandaid.
> >
> >     
> >> I don't use any bonded interfaces, but I do have one or two 
> >>       
> > multihomed 
> >     
> >> boxes being tracked by cobbler...
> >>
> >> ________________________________
> >> From: cobbler-bounces(a)lists.fedorahosted.org
> >> [mailto:[e-mail]] On Behalf Of Chris 
> >> Weaver
> >> Sent: Wednesday, December 31, 2008 9:06 AM
> >> To: cobbler(a)lists.fedorahosted.org
> >> Subject: Cobbler sync very slow, timing out with multiple interfaces
> >>
> >> I am working with the bonded interfaces option for 1.4.  For one 
> >> server it works great, I can add two bonded interfaces and 
> >>       
> > a couple of 
> >     
> >> non-bonded ones.  Everything comes up rosy.
> >>
> >> However cobbler sync is taking about 8 seconds when I have one 
> >> interface, and about 20 seconds with six.  I add another 
> >>       
> > system with 
> >     
> >> another 6 interfaces, it takes 3 minutes and times out.  If 
> >>       
> > I delete 
> >     
> >> an interface from each system, sync takes around 2 minutes but at 
> >> least it succeeds.  It is definitely related to dhcp since if I 
> >> disable that in settings, sync comes back right away.
> >>
> >> This is the error I get (after 3 minutes):
> >>
> >> ng00ud01# cobbler sync
> >>
> >> join() is too long for a Python string
> >>   File 
> >>       
> > "/usr/lib/python2.3/site-packages/cobbler/cobbler.py", line 83, 
> >     
> >> in main
> >>     rc = BootCLI().run(sys.argv)
> >>
> >>   File 
> >>       
> > "/usr/lib/python2.3/site-packages/cobbler/cobbler.py", line 61, 
> >     
> >> in run
> >>     return self.loader.run(args)
> >>
> >>   File "/usr/lib/python2.3/site-packages/cobbler/commands.py", line 
> >> 122, in run
> >>     return fn.run()
> >>
> >>   File 
> >>       
> > "/usr/lib/python2.3/site-packages/cobbler/modules/cli_misc.py", 
> >     
> >> line 183, in run
> >>     return self.api.sync()
> >>
> >>   File "/usr/lib/python2.3/site-packages/cobbler/api.py", 
> >>       
> > line 523, in sync
> >     
> >>     return sync.run()
> >>
> >>   File 
> >>       
> > "/usr/lib/python2.3/site-packages/cobbler/action_sync.py", line 
> >     
> >> 101, in run
> >>     self.dhcp.write_dhcp_file()
> >>
> >>   File 
> >> "/usr/lib/python2.3/site-packages/cobbler/modules/manage_isc.py",
> >> line 262, in write_dhcp_file
> >>     self.templar.render(template_data, metadata, 
> >>       
> > settings_file, None)
> >     
> >>   File "/usr/lib/python2.3/site-packages/cobbler/templar.py", line 
> >> 133, in render
> >>     data_out = data_out.replace("@@%s@@" % str(x),
> >> str(search_table[str(x)]))
> >>
> >>
> >> Any ideas?? TIA...
> >>
> >> --
> >> ... Chris Weaver
> >>
> >> _______________________________________________
> >> cobbler mailing list
> >> cobbler(a)lists.fedorahosted.org
> >> https://fedorahosted.org/mailman/listinfo/cobbler
> >>
> >>
> >>       
> >
> > --
> > Jeff Schroeder
> >
> > Don't drink and derive, alcohol and analysis don't mix.
> > http://www.digitalprognosis.com
> > _______________________________________________
> > cobbler mailing list
> > cobbler(a)lists.fedorahosted.org
> > https://fedorahosted.org/mailman/listinfo/cobbler
> >
> >     
>  _______________________________________________
>  cobbler mailing list
>  cobbler(a)lists.fedorahosted.org
>  https://fedorahosted.org/mailman/listinfo/cobbler
>     
Ok, your OMAPI being off is noted -- send the results of your cobbler 
system configs (tar up /var/lib/cobbler/config and send it to me off 
list if you like) and I'll see if I can reproduce things.

--Michael