server crash (389 server 1.2.1 on FC10 domu / centos 5.3 dom0)

[Mikael Kermorgant]

Hello,

We've recently migrated our old FDS 1.0.2/FC4 setup to a new one
hosted on a xen virtualised FC10. The dom0 is centos 5.3 and 389
server version is 1.2.1.

It has been working about a week, but we have recently faced some crashes.

First ones were the day after a schema modification on the master
which had not been weel propagated on the slaves. This could be the
explanation but I'm not really sure as it worked well several hours.

The last one was today when using the console.

Problem is we have no logs. Server suddenly freezes and is no more
available via ssh , vnc or xen console.

Would somebody have an idea about which steps to debug this situation ?
Could it be that virtualizing FC1 on Centos 5.3 was a bad idea ?

Regards,

--
Mikael Kermorgant

[Rich Megginson]

Mikael Kermorgant wrote:
>  Hello,
>
>  We've recently migrated our old FDS 1.0.2/FC4 setup to a new one
>  hosted on a xen virtualised FC10. The dom0 is centos 5.3 and 389
>  server version is 1.2.1.
>
>  It has been working about a week, but we have recently faced some crashes.
>
>  First ones were the day after a schema modification on the master
>  which had not been weel propagated on the slaves. 
How did you modify the schema on
the master?
>  This could be the
>  explanation but I'm not really sure as it worked well several hours.
>
>  The last one was today when using the console.
>     
Try console -D 9 -f console.log if you can reproduce the
>  Problem is we have no logs. Server suddenly freezes and is no more
>  available via ssh , vnc or xen console.
>     
So the entire VM freezes, not just the directory server?
So you see the server crashing and the VM freezing?
>  Would somebody have an idea about which steps to debug this situation
> ?
>     
Anything in the error log?  /var/log/messages ?  any other log files in 
/var/log?
>  Could it be that virtualizing FC1 on Centos 5.3 was a bad idea ?
>
>  Regards,
>
>  --
>  Mikael Kermorgant
>
>  --
>  389 users mailing list
>  389-users(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-directory-users
>

[Mikael Kermorgant]

On Fri, Oct 2, 2009 at 5:24 PM, Rich Megginson <rmeggins(a)redhat.com&gt; wrote:
>  Mikael Kermorgant wrote:
> >
> > Hello,
> >
> > First ones were the day after a schema modification on the master
> > which had not been weel propagated on the slaves.
>
>  How did you modify the schema on the master? 
Via the console, by adding an attribute.
Had to copy the schema files to the slaves later.

> >
> > This could be the
> > explanation but I'm not really sure as it worked well several hours.
> >
> > The last one was today when using the console.
> >
>
>  Try console -D 9 -f console.log if you can reproduce the 
Ok, I'll try that next time.

> > Problem is we have no logs. Server suddenly freezes and is no
> more
> > available via ssh , vnc or xen console.
> >
>
>  So the entire VM freezes, not just the directory server?
>  So you see the server crashing and the VM freezing? 
Yes, sort of. I still had control of the cursor but could not ctrl+c
the console on the xterm


> > Would somebody have an idea about which steps to debug this
> situation ?
> >
>
>  Anything in the error log?  /var/log/messages ?  any other log files in
>  /var/log? 
Only in /var/log/xen/xend.log which indicates that this is quite offtopic.

I'll see what I can find on the xen side,

Thanks,

Mikael

[Rich Megginson]

Mikael Kermorgant wrote:
>  On Fri, Oct 2, 2009 at 5:24 PM, Rich Megginson
> <rmeggins(a)redhat.com&gt; wrote:
>    
> > Mikael Kermorgant wrote:
> >     
> >> Hello,
> >>
> >> First ones were the day after a schema modification on the master
> >> which had not been weel propagated on the slaves.
> >>       
> > How did you modify the schema on the master?
> >     
>
>  Via the console, by adding an attribute.
>  Had to copy the schema files to the slaves later.
>     
Were there any messages in the errors logs on the supplier or consumers 
about not being able to replicate the schema?
>    
> >> This could be the
> >> explanation but I'm not really sure as it worked well several hours.
> >>
> >> The last one was today when using the console.
> >>
> >>       
> > Try console -D 9 -f console.log if you can reproduce the
> >     
>
>  Ok, I'll try that next time.
>
>    
> >> Problem is we have no logs. Server suddenly freezes and is no more
> >> available via ssh , vnc or xen console.
> >>
> >>       
> > So the entire VM freezes, not just the directory server?
> > So you see the server crashing and the VM freezing?
> >     
>
>  Yes, sort of. I still had control of the cursor but could not ctrl+c
>  the console on the xterm
>
>
>    
> >> Would somebody have an idea about which steps to debug this situation ?
> >>
> >>       
> > Anything in the error log?  /var/log/messages ?  any other log files in
> > /var/log?
> >     
>
>  Only in /var/log/xen/xend.log which indicates that this is quite offtopic.
>
>  I'll see what I can find on the xen side,
>
>  Thanks,
>
>  Mikael
>
>  --
>  389 users mailing list
>  389-users(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-directory-users
>

[Mikael Kermorgant]

Hello,

On Fri, Oct 2, 2009 at 10:40 PM, Rich Megginson <rmeggins(a)redhat.com&gt; wrote:

> >>> First ones were the day after a schema modification on
> the master
> >>> which had not been weel propagated on the slaves.
> >>>
> >>
> >> How did you modify the schema on the master?
> >>
> >
> > Via the console, by adding an attribute.
> > Had to copy the schema files to the slaves later.
> >
>
>  Were there any messages in the errors logs on the supplier or consumers
>  about not being able to replicate the schema? 
Yep, there were some :

[23/Sep/2009:10:59:58 +0200] NSMMReplicationPlugin -
agmtlist_add_callback: Can't start agreement "cn=supann0& -
bsupann01,cn=replica,cn="dc=dede, dc=deded, dc=fr",cn=mapping
tree,cn=config"

And many of these :

[23/Sep/2009:14:37:13 +0200] NSMMReplicationPlugin -
agmt="cn=bsupann01" (192:389): Replica has a different generation ID
than the local data.
[23/Sep/2009:14:37:17 +0200] NSMMReplicationPlugin -
agmt="cn=bsupann01" (192:389): Replica has a different generation ID
than the local data.


However, I now really want to migrate to some safer virtualisation
environment. Maybe a bit over pessimistic but I'm affraid of the FC11
domU on centos dom0...


Regards,

Mikael

[Rich Megginson]

Mikael Kermorgant wrote:
>  Hello,
>
>  On Fri, Oct 2, 2009 at 10:40 PM, Rich Megginson <rmeggins(a)redhat.com&gt; wrote:
>
>    
> >>>> First ones were the day after a schema modification on the master
> >>>> which had not been weel propagated on the slaves.
> >>>>
> >>>>           
> >>> How did you modify the schema on the master?
> >>>
> >>>         
> >> Via the console, by adding an attribute.
> >> Had to copy the schema files to the slaves later.
> >>
> >>       
> > Were there any messages in the errors logs on the supplier or consumers
> > about not being able to replicate the schema?
> >     
>
>  Yep, there were some :
>
>  [23/Sep/2009:10:59:58 +0200] NSMMReplicationPlugin -
>  agmtlist_add_callback: Can't start agreement "cn=supann0& -
>  bsupann01,cn=replica,cn="dc=dede, dc=deded, dc=fr",cn=mapping
>  tree,cn=config"
>     
That's really odd.  Do you get that at startup?  That indicates some 
sort of configuration problem.  You could try starting the server with 
/usr/lib/dirsrv/slapd-instance/start-slapd -d 8192 to see if there is 
more detailed information.
>  And many of these :
>
>  [23/Sep/2009:14:37:13 +0200] NSMMReplicationPlugin -
>  agmt="cn=bsupann01" (192:389): Replica has a different generation ID
>  than the local data.
>  [23/Sep/2009:14:37:17 +0200] NSMMReplicationPlugin -
>  agmt="cn=bsupann01" (192:389): Replica has a different generation ID
>  than the local data.
>     
This usually means the consumer has not been initialized.
>  However, I now really want to migrate to some safer virtualisation
>  environment. Maybe a bit over pessimistic but I'm affraid of the FC11
>  domU on centos dom0...
>
>
>  Regards,
>
>  Mikael
>
>  --
>  389 users mailing list
>  389-users(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-directory-users
>