Issues with devstack on both Fedora 17 and 18

[Kyle Mestery (kmestery)]

Tried this on the openstack-dev list, but didn't hear anything, thought I'd try
the Fedora Cloud mailing list since this is Fedora specific.

On both Fedora 17 and Fedora 18, when I use devstack, after a devstack run my D-Bus daemon
has died such that systemctl no longer works. If I try to run systemctl, I see this:

[kmestery@fedora-mac devstack]$ systemctl 
Failed to get D-Bus connection: No connection to service manager.
[kmestery@fedora-mac devstack]$ 

I've googled around for this, but haven't seen anything concrete. Does anyone have
any idea what might be happening, or where I could look to understand why this is
happening? A reboot of the host fixes things, but another devstack run leaves the system
in the same state.

Thanks!
Kyle

[Daniel P. Berrange]

On Fri, Feb 08, 2013 at 02:23:07PM +0000, Kyle Mestery (kmestery) wrote:
>  Tried this on the openstack-dev list, but didn't hear anything,
> thought I'd try the Fedora Cloud mailing list since this is Fedora specific.
>  
>  On both Fedora 17 and Fedora 18, when I use devstack, after a devstack run my D-Bus
> daemon has died such that systemctl no longer works. If I try to run systemctl, I see
> this:
>  
>  [kmestery@fedora-mac devstack]$ systemctl 
>  Failed to get D-Bus connection: No connection to service manager.
>  [kmestery@fedora-mac devstack]$ 
>  
>  I've googled around for this, but haven't seen anything concrete.
>  Does anyone have any idea what might be happening, or where I
>  could look to understand why this is happening? A reboot of the
>  host fixes things, but another devstack run leaves the system in
>  the same state. 
I've never experianced this issue with DevStack / DBus on any F17
or F18 system I've used. I guess it could be something peculiar to
your particular devstack config - what is in your 'localrc' ?

I successfully run devstack on F18/F17 using the following, on a
regular basis

$ cd $HOME/cloud/devstack
$ cat localrc
DEST=$HOME/src/cloud
DATA_DIR=$DEST/data

LOGFILE=$DATA_DIR/logs/stack.log
SCREEN_LOGDIR=$DATA_DIR/logs
VERBOSE=False

disable_service rabbit
enable_service qpid
disable_service tempest

# If i want to test spice instead of VNC
#enable_service n-spice
#disable_service n-novnc
#disable_service n-xvnc

# If I want to test quantum instead of nova-network
#Q_PLUGIN=linuxbridge
#Q_PLUGIN=openvswitch
#disable_service n-net
#enable_service q-svc
#enable_service q-agt
#enable_service q-dhcp
#enable_service q-l3
#enable_service q-meta
#enable_service quantum

MYSQL_PASSWORD=123456
SERVICE_TOKEN=123456
SERVICE_PASSWORD=123456
ADMIN_PASSWORD=123456

IMAGE_URLS="http://berrange.fedorapeople.org/images/2012-02-29/f16-x...



NB, this results in everything being checked out under $HOME/cloud/XXXXX
and not /opt, because I don't like polluting /opt, and puts all devstack
logs in $HOME/cloud/data/logs

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

[Kyle Mestery (kmestery)]

On Feb 8, 2013, at 8:31 AM, Daniel P. Berrange <berrange(a)redhat.com&gt; wrote:
>  On Fri, Feb 08, 2013 at 02:23:07PM +0000, Kyle Mestery (kmestery)
> wrote:
> > Tried this on the openstack-dev list, but didn't hear anything, thought I'd
> try the Fedora Cloud mailing list since this is Fedora specific.
> > 
> > On both Fedora 17 and Fedora 18, when I use devstack, after a devstack run my D-Bus
> daemon has died such that systemctl no longer works. If I try to run systemctl, I see
> this:
> > 
> > [kmestery@fedora-mac devstack]$ systemctl 
> > Failed to get D-Bus connection: No connection to service manager.
> > [kmestery@fedora-mac devstack]$ 
> > 
> > I've googled around for this, but haven't seen anything concrete.
> > Does anyone have any idea what might be happening, or where I
> > could look to understand why this is happening? A reboot of the
> > host fixes things, but another devstack run leaves the system in
> > the same state.
>  
>  I've never experianced this issue with DevStack / DBus on any F17
>  or F18 system I've used. I guess it could be something peculiar to
>  your particular devstack config - what is in your 'localrc' ?
>  
>  I successfully run devstack on F18/F17 using the following, on a
>  regular basis
>  
>  $ cd $HOME/cloud/devstack
>  $ cat localrc
>  DEST=$HOME/src/cloud
>  DATA_DIR=$DEST/data
>  
>  LOGFILE=$DATA_DIR/logs/stack.log
>  SCREEN_LOGDIR=$DATA_DIR/logs
>  VERBOSE=False
>  
>  disable_service rabbit
>  enable_service qpid
>  disable_service tempest
>  
>  # If i want to test spice instead of VNC
>  #enable_service n-spice
>  #disable_service n-novnc
>  #disable_service n-xvnc
>  
>  # If I want to test quantum instead of nova-network
>  #Q_PLUGIN=linuxbridge
>  #Q_PLUGIN=openvswitch
>  #disable_service n-net
>  #enable_service q-svc
>  #enable_service q-agt
>  #enable_service q-dhcp
>  #enable_service q-l3
>  #enable_service q-meta
>  #enable_service quantum
>  
>  MYSQL_PASSWORD=123456
>  SERVICE_TOKEN=123456
>  SERVICE_PASSWORD=123456
>  ADMIN_PASSWORD=123456
>  
>
> IMAGE_URLS="http://berrange.fedorapeople.org/images/2012-02-29/f16-x...
>  
>  
>  
>  NB, this results in everything being checked out under $HOME/cloud/XXXXX
>  and not /opt, because I don't like polluting /opt, and puts all devstack
>  logs in $HOME/cloud/data/logs
>  
>  Regards,
>  Daniel 

My localrc looks like the below. Note I am storing things in /opt, and I am using
Quantum with the Ryu plugin as well. I was using rabbitmq, but switched to qpid
recently, though this fails the same way with either one.

[kmestery@fedora-mac devstack]$ cat localrc 
LOGFILE=stack.sh.log
#OFFLINE=True
#RECLONE=yes
disable_service n-net
enable_service q-svc
enable_service q-agt
enable_service q-dhcp
enable_service q-l3
enable_service quantum
enable_service ryu
disable_service rabbit
enable_service qpid

HOST_NAME=$(hostname)
SERVICE_HOST_NAME=${HOST_NAME}
SERVICE_HOST=192.168.56.101

FLOATING_RANGE=192.168.100.0/24
#Q_PLUGIN=openvswitch
Q_PLUGIN=ryu
# ryu
RYU_API_HOST=$SERVICE_HOST
RYU_OFP_HOST=$SERVICE_HOST
RYU_APPS=ryu.app.gre_tunnel,ryu.app.quantum_adapter,ryu.app.rest,ryu.app.rest_conf_switch,ryu.app.rest_tunnel,ryu.app.tunnel_port_updater,ryu.app.rest_quantum
RYU_REPO=${GIT_BASE}/ykaneko/ryu.git
RYU_BRANCH=live-migration

Q_HOST=$SERVICE_HOST
MYSQL_HOST=$SERVICE_HOST
RABBIT_HOST=$SERVICE_HOST
GLANCE_HOSTPORT=$SERVICE_HOST:9292
KEYSTONE_AUTH_HOST=$SERVICE_HOST
KEYSTONE_SERVICE_HOST=$SERVICE_HOST

MYSQL_PASSWORD=mysql
#RABBIT_PASSWORD=rabbit
SERVICE_TOKEN=service
SERVICE_PASSWORD=admin
ADMIN_PASSWORD=admin

[kmestery@fedora-mac devstack]$ 

Thanks,
Kyle

[Daniel P. Berrange]

On Fri, Feb 08, 2013 at 02:35:24PM +0000, Kyle Mestery (kmestery) wrote:
>  On Feb 8, 2013, at 8:31 AM, Daniel P. Berrange
> <berrange(a)redhat.com&gt; wrote:
>  > On Fri, Feb 08, 2013 at 02:23:07PM +0000, Kyle Mestery (kmestery) wrote:
>  My localrc looks like the below. Note I am storing things in /opt, and I am using
>  Quantum with the Ryu plugin as well. I was using rabbitmq, but switched to qpid
>  recently, though this fails the same way with either one. 
I don't see anything particularly wrong with your config here. The only
key difference from mine seems to be your use of Quantum+Ryu. So I guess
my inclination would be to try taking this out of the equation, by trying
todo a setup using just nova-network, and disable Quantum entirely. If
that works, then you'll know it is something quantum related, and can then
try different plugins to see if it is Ryu related or not.

>  
>  [kmestery@fedora-mac devstack]$ cat localrc 
>  LOGFILE=stack.sh.log
>  #OFFLINE=True
>  #RECLONE=yes
>  disable_service n-net
>  enable_service q-svc
>  enable_service q-agt
>  enable_service q-dhcp
>  enable_service q-l3
>  enable_service quantum
>  enable_service ryu
>  disable_service rabbit
>  enable_service qpid
>  
>  HOST_NAME=$(hostname)
>  SERVICE_HOST_NAME=${HOST_NAME}
>  SERVICE_HOST=192.168.56.101
>  
>  FLOATING_RANGE=192.168.100.0/24
>  #Q_PLUGIN=openvswitch
>  Q_PLUGIN=ryu
>  # ryu
>  RYU_API_HOST=$SERVICE_HOST
>  RYU_OFP_HOST=$SERVICE_HOST
>
> RYU_APPS=ryu.app.gre_tunnel,ryu.app.quantum_adapter,ryu.app.rest,ryu.app.rest_conf_switch,ryu.app.rest_tunnel,ryu.app.tunnel_port_updater,ryu.app.rest_quantum
>  RYU_REPO=${GIT_BASE}/ykaneko/ryu.git
>  RYU_BRANCH=live-migration
>  
>  Q_HOST=$SERVICE_HOST
>  MYSQL_HOST=$SERVICE_HOST
>  RABBIT_HOST=$SERVICE_HOST
>  GLANCE_HOSTPORT=$SERVICE_HOST:9292
>  KEYSTONE_AUTH_HOST=$SERVICE_HOST
>  KEYSTONE_SERVICE_HOST=$SERVICE_HOST
>  
>  MYSQL_PASSWORD=mysql
>  #RABBIT_PASSWORD=rabbit
>  SERVICE_TOKEN=service
>  SERVICE_PASSWORD=admin
>  ADMIN_PASSWORD=admin
>   

Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

[Dennis Jacobfeuerborn]

On 02/08/2013 03:49 PM, Daniel P. Berrange wrote:
>  On Fri, Feb 08, 2013 at 02:35:24PM +0000, Kyle Mestery (kmestery)
> wrote:
> > On Feb 8, 2013, at 8:31 AM, Daniel P. Berrange <berrange(a)redhat.com&gt; wrote:
> >> On Fri, Feb 08, 2013 at 02:23:07PM +0000, Kyle Mestery (kmestery) wrote:
> > My localrc looks like the below. Note I am storing things in /opt, and I am using
> > Quantum with the Ryu plugin as well. I was using rabbitmq, but switched to qpid
> > recently, though this fails the same way with either one.
>  
>  I don't see anything particularly wrong with your config here. The only
>  key difference from mine seems to be your use of Quantum+Ryu. So I guess
>  my inclination would be to try taking this out of the equation, by trying
>  todo a setup using just nova-network, and disable Quantum entirely. If
>  that works, then you'll know it is something quantum related, and can then
>  try different plugins to see if it is Ryu related or not.
>  
> >
> > [kmestery@fedora-mac devstack]$ cat localrc 
> > LOGFILE=stack.sh.log
> > #OFFLINE=True
> > #RECLONE=yes
> > disable_service n-net
> > enable_service q-svc
> > enable_service q-agt
> > enable_service q-dhcp
> > enable_service q-l3
> > enable_service quantum
> > enable_service ryu
> > disable_service rabbit
> > enable_service qpid
> >
> > HOST_NAME=$(hostname)
> > SERVICE_HOST_NAME=${HOST_NAME}
> > SERVICE_HOST=192.168.56.101
> >
> > FLOATING_RANGE=192.168.100.0/24
> > #Q_PLUGIN=openvswitch
> > Q_PLUGIN=ryu
> > # ryu
> > RYU_API_HOST=$SERVICE_HOST
> > RYU_OFP_HOST=$SERVICE_HOST
> >
> RYU_APPS=ryu.app.gre_tunnel,ryu.app.quantum_adapter,ryu.app.rest,ryu.app.rest_conf_switch,ryu.app.rest_tunnel,ryu.app.tunnel_port_updater,ryu.app.rest_quantum
> > RYU_REPO=${GIT_BASE}/ykaneko/ryu.git
> > RYU_BRANCH=live-migration
> >
> > Q_HOST=$SERVICE_HOST
> > MYSQL_HOST=$SERVICE_HOST
> > RABBIT_HOST=$SERVICE_HOST
> > GLANCE_HOSTPORT=$SERVICE_HOST:9292
> > KEYSTONE_AUTH_HOST=$SERVICE_HOST
> > KEYSTONE_SERVICE_HOST=$SERVICE_HOST
> >
> > MYSQL_PASSWORD=mysql
> > #RABBIT_PASSWORD=rabbit
> > SERVICE_TOKEN=service
> > SERVICE_PASSWORD=admin
> > ADMIN_PASSWORD=admin 
I've seen something similar and quantum was the culprit. I don't know what
quantum does that messes this up but when i disabled the quantum services
the problem went away and when I re-enabled them the problem popped back up.

Regards,
  Dennis

[Kyle Mestery (kmestery)]

On Feb 8, 2013, at 8:55 AM, Dennis Jacobfeuerborn <dennisml(a)conversis.de&gt;
 wrote:
>  On 02/08/2013 03:49 PM, Daniel P. Berrange wrote:
> > On Fri, Feb 08, 2013 at 02:35:24PM +0000, Kyle Mestery (kmestery) wrote:
> >> On Feb 8, 2013, at 8:31 AM, Daniel P. Berrange <berrange(a)redhat.com&gt;
> wrote:
> >>> On Fri, Feb 08, 2013 at 02:23:07PM +0000, Kyle Mestery (kmestery) wrote:
> >> My localrc looks like the below. Note I am storing things in /opt, and I am
> using
> >> Quantum with the Ryu plugin as well. I was using rabbitmq, but switched to qpid
> >> recently, though this fails the same way with either one.
> > 
> > I don't see anything particularly wrong with your config here. The only
> > key difference from mine seems to be your use of Quantum+Ryu. So I guess
> > my inclination would be to try taking this out of the equation, by trying
> > todo a setup using just nova-network, and disable Quantum entirely. If
> > that works, then you'll know it is something quantum related, and can then
> > try different plugins to see if it is Ryu related or not.
> > 
> >> 
> >> [kmestery@fedora-mac devstack]$ cat localrc 
> >> LOGFILE=stack.sh.log
> >> #OFFLINE=True
> >> #RECLONE=yes
> >> disable_service n-net
> >> enable_service q-svc
> >> enable_service q-agt
> >> enable_service q-dhcp
> >> enable_service q-l3
> >> enable_service quantum
> >> enable_service ryu
> >> disable_service rabbit
> >> enable_service qpid
> >> 
> >> HOST_NAME=$(hostname)
> >> SERVICE_HOST_NAME=${HOST_NAME}
> >> SERVICE_HOST=192.168.56.101
> >> 
> >> FLOATING_RANGE=192.168.100.0/24
> >> #Q_PLUGIN=openvswitch
> >> Q_PLUGIN=ryu
> >> # ryu
> >> RYU_API_HOST=$SERVICE_HOST
> >> RYU_OFP_HOST=$SERVICE_HOST
> >>
> RYU_APPS=ryu.app.gre_tunnel,ryu.app.quantum_adapter,ryu.app.rest,ryu.app.rest_conf_switch,ryu.app.rest_tunnel,ryu.app.tunnel_port_updater,ryu.app.rest_quantum
> >> RYU_REPO=${GIT_BASE}/ykaneko/ryu.git
> >> RYU_BRANCH=live-migration
> >> 
> >> Q_HOST=$SERVICE_HOST
> >> MYSQL_HOST=$SERVICE_HOST
> >> RABBIT_HOST=$SERVICE_HOST
> >> GLANCE_HOSTPORT=$SERVICE_HOST:9292
> >> KEYSTONE_AUTH_HOST=$SERVICE_HOST
> >> KEYSTONE_SERVICE_HOST=$SERVICE_HOST
> >> 
> >> MYSQL_PASSWORD=mysql
> >> #RABBIT_PASSWORD=rabbit
> >> SERVICE_TOKEN=service
> >> SERVICE_PASSWORD=admin
> >> ADMIN_PASSWORD=admin
>  
>  I've seen something similar and quantum was the culprit. I don't know what
>  quantum does that messes this up but when i disabled the quantum services
>  the problem went away and when I re-enabled them the problem popped back up.
>   
Thanks Dennis and Daniel, you are both right, it is Quantum causing this
issue. I'll file a bug with the Quantum project and take a look at fixing this.
I'm not sure what's causing it now, but it was getting annoying to constantly
be seeing it. Thanks for helping me debug this!

Kyle


>  Regards,
>   Dennis
>  
>  _______________________________________________
>  cloud mailing list
>  cloud(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/cloud

[Daniel P. Berrange]

On Fri, Feb 08, 2013 at 03:03:43PM +0000, Kyle Mestery (kmestery) wrote:
>  On Feb 8, 2013, at 8:55 AM, Dennis Jacobfeuerborn
> <dennisml(a)conversis.de&gt;
>   wrote:
>  > On 02/08/2013 03:49 PM, Daniel P. Berrange wrote:
>  >> On Fri, Feb 08, 2013 at 02:35:24PM +0000, Kyle Mestery (kmestery) wrote:
>  >>> On Feb 8, 2013, at 8:31 AM, Daniel P. Berrange <berrange(a)redhat.com&gt;
> wrote:
>  >>>> On Fri, Feb 08, 2013 at 02:23:07PM +0000, Kyle Mestery (kmestery)
> wrote:
>  >>> My localrc looks like the below. Note I am storing things in /opt, and I am
> using
>  >>> Quantum with the Ryu plugin as well. I was using rabbitmq, but switched to
> qpid
>  >>> recently, though this fails the same way with either one.
>  >> 
>  >> I don't see anything particularly wrong with your config here. The only
>  >> key difference from mine seems to be your use of Quantum+Ryu. So I guess
>  >> my inclination would be to try taking this out of the equation, by trying
>  >> todo a setup using just nova-network, and disable Quantum entirely. If
>  >> that works, then you'll know it is something quantum related, and can then
>  >> try different plugins to see if it is Ryu related or not.
>  >> 
>  >>> 
>  >>> [kmestery@fedora-mac devstack]$ cat localrc 
>  >>> LOGFILE=stack.sh.log
>  >>> #OFFLINE=True
>  >>> #RECLONE=yes
>  >>> disable_service n-net
>  >>> enable_service q-svc
>  >>> enable_service q-agt
>  >>> enable_service q-dhcp
>  >>> enable_service q-l3
>  >>> enable_service quantum
>  >>> enable_service ryu
>  >>> disable_service rabbit
>  >>> enable_service qpid
>  >>> 
>  >>> HOST_NAME=$(hostname)
>  >>> SERVICE_HOST_NAME=${HOST_NAME}
>  >>> SERVICE_HOST=192.168.56.101
>  >>> 
>  >>> FLOATING_RANGE=192.168.100.0/24
>  >>> #Q_PLUGIN=openvswitch
>  >>> Q_PLUGIN=ryu
>  >>> # ryu
>  >>> RYU_API_HOST=$SERVICE_HOST
>  >>> RYU_OFP_HOST=$SERVICE_HOST
>  >>>
> RYU_APPS=ryu.app.gre_tunnel,ryu.app.quantum_adapter,ryu.app.rest,ryu.app.rest_conf_switch,ryu.app.rest_tunnel,ryu.app.tunnel_port_updater,ryu.app.rest_quantum
>  >>> RYU_REPO=${GIT_BASE}/ykaneko/ryu.git
>  >>> RYU_BRANCH=live-migration
>  >>> 
>  >>> Q_HOST=$SERVICE_HOST
>  >>> MYSQL_HOST=$SERVICE_HOST
>  >>> RABBIT_HOST=$SERVICE_HOST
>  >>> GLANCE_HOSTPORT=$SERVICE_HOST:9292
>  >>> KEYSTONE_AUTH_HOST=$SERVICE_HOST
>  >>> KEYSTONE_SERVICE_HOST=$SERVICE_HOST
>  >>> 
>  >>> MYSQL_PASSWORD=mysql
>  >>> #RABBIT_PASSWORD=rabbit
>  >>> SERVICE_TOKEN=service
>  >>> SERVICE_PASSWORD=admin
>  >>> ADMIN_PASSWORD=admin
>  > 
>  > I've seen something similar and quantum was the culprit. I don't know what
>  > quantum does that messes this up but when i disabled the quantum services
>  > the problem went away and when I re-enabled them the problem popped back up.
>  > 
>  Thanks Dennis and Daniel, you are both right, it is Quantum causing this
>  issue. I'll file a bug with the Quantum project and take a look at fixing this.
>  I'm not sure what's causing it now, but it was getting annoying to constantly
>  be seeing it. Thanks for helping me debug this! 
CC me on the bug you file - I'd be very interested to find out what the
cause is. Even if quantum is what's triggering it, it smells like it might
actually be a bug in DBus - something like Quantum ought not to be able
to crash dbus after all !

My money would be on something namespace related, since IIUC, quantum
does some playing around with network namespaces at least ?

Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

[Kyle Mestery (kmestery)]

On Feb 8, 2013, at 9:13 AM, "Daniel P. Berrange" <berrange(a)redhat.com&gt;
 wrote:
>  On Fri, Feb 08, 2013 at 03:03:43PM +0000, Kyle Mestery (kmestery)
> wrote:
> > On Feb 8, 2013, at 8:55 AM, Dennis Jacobfeuerborn <dennisml(a)conversis.de&gt;
> > wrote:
> >> On 02/08/2013 03:49 PM, Daniel P. Berrange wrote:
> >>> On Fri, Feb 08, 2013 at 02:35:24PM +0000, Kyle Mestery (kmestery) wrote:
> >>>> On Feb 8, 2013, at 8:31 AM, Daniel P. Berrange
> <berrange(a)redhat.com&gt; wrote:
> >>>>> On Fri, Feb 08, 2013 at 02:23:07PM +0000, Kyle Mestery (kmestery)
> wrote:
> >>>> My localrc looks like the below. Note I am storing things in /opt, and I
> am using
> >>>> Quantum with the Ryu plugin as well. I was using rabbitmq, but switched
> to qpid
> >>>> recently, though this fails the same way with either one.
> >>> 
> >>> I don't see anything particularly wrong with your config here. The only
> >>> key difference from mine seems to be your use of Quantum+Ryu. So I guess
> >>> my inclination would be to try taking this out of the equation, by trying
> >>> todo a setup using just nova-network, and disable Quantum entirely. If
> >>> that works, then you'll know it is something quantum related, and can
> then
> >>> try different plugins to see if it is Ryu related or not.
> >>> 
> >>>> 
> >>>> [kmestery@fedora-mac devstack]$ cat localrc 
> >>>> LOGFILE=stack.sh.log
> >>>> #OFFLINE=True
> >>>> #RECLONE=yes
> >>>> disable_service n-net
> >>>> enable_service q-svc
> >>>> enable_service q-agt
> >>>> enable_service q-dhcp
> >>>> enable_service q-l3
> >>>> enable_service quantum
> >>>> enable_service ryu
> >>>> disable_service rabbit
> >>>> enable_service qpid
> >>>> 
> >>>> HOST_NAME=$(hostname)
> >>>> SERVICE_HOST_NAME=${HOST_NAME}
> >>>> SERVICE_HOST=192.168.56.101
> >>>> 
> >>>> FLOATING_RANGE=192.168.100.0/24
> >>>> #Q_PLUGIN=openvswitch
> >>>> Q_PLUGIN=ryu
> >>>> # ryu
> >>>> RYU_API_HOST=$SERVICE_HOST
> >>>> RYU_OFP_HOST=$SERVICE_HOST
> >>>>
> RYU_APPS=ryu.app.gre_tunnel,ryu.app.quantum_adapter,ryu.app.rest,ryu.app.rest_conf_switch,ryu.app.rest_tunnel,ryu.app.tunnel_port_updater,ryu.app.rest_quantum
> >>>> RYU_REPO=${GIT_BASE}/ykaneko/ryu.git
> >>>> RYU_BRANCH=live-migration
> >>>> 
> >>>> Q_HOST=$SERVICE_HOST
> >>>> MYSQL_HOST=$SERVICE_HOST
> >>>> RABBIT_HOST=$SERVICE_HOST
> >>>> GLANCE_HOSTPORT=$SERVICE_HOST:9292
> >>>> KEYSTONE_AUTH_HOST=$SERVICE_HOST
> >>>> KEYSTONE_SERVICE_HOST=$SERVICE_HOST
> >>>> 
> >>>> MYSQL_PASSWORD=mysql
> >>>> #RABBIT_PASSWORD=rabbit
> >>>> SERVICE_TOKEN=service
> >>>> SERVICE_PASSWORD=admin
> >>>> ADMIN_PASSWORD=admin
> >> 
> >> I've seen something similar and quantum was the culprit. I don't know
> what
> >> quantum does that messes this up but when i disabled the quantum services
> >> the problem went away and when I re-enabled them the problem popped back up.
> >> 
> > Thanks Dennis and Daniel, you are both right, it is Quantum causing this
> > issue. I'll file a bug with the Quantum project and take a look at fixing this.
> > I'm not sure what's causing it now, but it was getting annoying to
> constantly
> > be seeing it. Thanks for helping me debug this!
>  
>  CC me on the bug you file - I'd be very interested to find out what the
>  cause is. Even if quantum is what's triggering it, it smells like it might
>  actually be a bug in DBus - something like Quantum ought not to be able
>  to crash dbus after all !
>  
>  My money would be on something namespace related, since IIUC, quantum
>  does some playing around with network namespaces at least ?
>  
>  Daniel 
Thanks Daniel. I just subscribed you to the bug. I'm trying a run with
namespaces turned off now to see if that helps or not. I think that may be
the culprit, I recall running a devstack with Quantum but without namespaces
and it working a while back. Will let you know how it goes.

Thanks,
Kyle

[Kyle Mestery (kmestery)]

On Feb 8, 2013, at 9:16 AM, "Kyle Mestery (kmestery)"
<kmestery(a)cisco.com&gt;
 wrote:
>  On Feb 8, 2013, at 9:13 AM, "Daniel P. Berrange"
> <berrange(a)redhat.com&gt;
>  wrote:
> > On Fri, Feb 08, 2013 at 03:03:43PM +0000, Kyle Mestery (kmestery) wrote:
> >> On Feb 8, 2013, at 8:55 AM, Dennis Jacobfeuerborn <dennisml(a)conversis.de&gt;
> >> wrote:
> >>> On 02/08/2013 03:49 PM, Daniel P. Berrange wrote:
> >>>> On Fri, Feb 08, 2013 at 02:35:24PM +0000, Kyle Mestery (kmestery) wrote:
> >>>>> On Feb 8, 2013, at 8:31 AM, Daniel P. Berrange
> <berrange(a)redhat.com&gt; wrote:
> >>>>>> On Fri, Feb 08, 2013 at 02:23:07PM +0000, Kyle Mestery (kmestery)
> wrote:
> >>>>> My localrc looks like the below. Note I am storing things in /opt,
> and I am using
> >>>>> Quantum with the Ryu plugin as well. I was using rabbitmq, but
> switched to qpid
> >>>>> recently, though this fails the same way with either one.
> >>>> 
> >>>> I don't see anything particularly wrong with your config here. The
> only
> >>>> key difference from mine seems to be your use of Quantum+Ryu. So I guess
> >>>> my inclination would be to try taking this out of the equation, by
> trying
> >>>> todo a setup using just nova-network, and disable Quantum entirely. If
> >>>> that works, then you'll know it is something quantum related, and can
> then
> >>>> try different plugins to see if it is Ryu related or not.
> >>>> 
> >>>>> 
> >>>>> [kmestery@fedora-mac devstack]$ cat localrc 
> >>>>> LOGFILE=stack.sh.log
> >>>>> #OFFLINE=True
> >>>>> #RECLONE=yes
> >>>>> disable_service n-net
> >>>>> enable_service q-svc
> >>>>> enable_service q-agt
> >>>>> enable_service q-dhcp
> >>>>> enable_service q-l3
> >>>>> enable_service quantum
> >>>>> enable_service ryu
> >>>>> disable_service rabbit
> >>>>> enable_service qpid
> >>>>> 
> >>>>> HOST_NAME=$(hostname)
> >>>>> SERVICE_HOST_NAME=${HOST_NAME}
> >>>>> SERVICE_HOST=192.168.56.101
> >>>>> 
> >>>>> FLOATING_RANGE=192.168.100.0/24
> >>>>> #Q_PLUGIN=openvswitch
> >>>>> Q_PLUGIN=ryu
> >>>>> # ryu
> >>>>> RYU_API_HOST=$SERVICE_HOST
> >>>>> RYU_OFP_HOST=$SERVICE_HOST
> >>>>>
> RYU_APPS=ryu.app.gre_tunnel,ryu.app.quantum_adapter,ryu.app.rest,ryu.app.rest_conf_switch,ryu.app.rest_tunnel,ryu.app.tunnel_port_updater,ryu.app.rest_quantum
> >>>>> RYU_REPO=${GIT_BASE}/ykaneko/ryu.git
> >>>>> RYU_BRANCH=live-migration
> >>>>> 
> >>>>> Q_HOST=$SERVICE_HOST
> >>>>> MYSQL_HOST=$SERVICE_HOST
> >>>>> RABBIT_HOST=$SERVICE_HOST
> >>>>> GLANCE_HOSTPORT=$SERVICE_HOST:9292
> >>>>> KEYSTONE_AUTH_HOST=$SERVICE_HOST
> >>>>> KEYSTONE_SERVICE_HOST=$SERVICE_HOST
> >>>>> 
> >>>>> MYSQL_PASSWORD=mysql
> >>>>> #RABBIT_PASSWORD=rabbit
> >>>>> SERVICE_TOKEN=service
> >>>>> SERVICE_PASSWORD=admin
> >>>>> ADMIN_PASSWORD=admin
> >>> 
> >>> I've seen something similar and quantum was the culprit. I don't know
> what
> >>> quantum does that messes this up but when i disabled the quantum services
> >>> the problem went away and when I re-enabled them the problem popped back up.
> >>> 
> >> Thanks Dennis and Daniel, you are both right, it is Quantum causing this
> >> issue. I'll file a bug with the Quantum project and take a look at fixing
> this.
> >> I'm not sure what's causing it now, but it was getting annoying to
> constantly
> >> be seeing it. Thanks for helping me debug this!
> > 
> > CC me on the bug you file - I'd be very interested to find out what the
> > cause is. Even if quantum is what's triggering it, it smells like it might
> > actually be a bug in DBus - something like Quantum ought not to be able
> > to crash dbus after all !
> > 
> > My money would be on something namespace related, since IIUC, quantum
> > does some playing around with network namespaces at least ?
> > 
> > Daniel
>  
>  Thanks Daniel. I just subscribed you to the bug. I'm trying a run with
>  namespaces turned off now to see if that helps or not. I think that may be
>  the culprit, I recall running a devstack with Quantum but without namespaces
>  and it working a while back. Will let you know how it goes.
>  
>  Thanks,
>  Kyle
>   
It's definitely namespace related. When I turned namespaces off in my
localrc,
DBus stayed up and things worked ok. Now the digging on how they two are
interacting in such a destructive way can begin!

Thanks,
Kyle

>  _______________________________________________
>  cloud mailing list
>  cloud(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/cloud

[Alan Pevec]

2013/2/8 Kyle Mestery (kmestery) <kmestery(a)cisco.com&gt;:
>  It's definitely namespace related. When I turned namespaces off
> in my localrc,
>  DBus stayed up and things worked ok. Now the digging on how they two are
>  interacting in such a destructive way can begin! 
Maybe https://bugzilla.redhat.com/show_bug.cgi?id=882047 ?

[Matthias Runge]

On 02/10/2013 10:08 PM, Alan Pevec wrote:
>  2013/2/8 Kyle Mestery (kmestery) <kmestery(a)cisco.com&gt;:
> > It's definitely namespace related. When I turned namespaces off in my localrc,
> > DBus stayed up and things worked ok. Now the digging on how they two are
> > interacting in such a destructive way can begin!
>  
>  Maybe https://bugzilla.redhat.com/show_bug.cgi?id=882047 ? 
Sounds like.

Some observations:

- this has nothing to do especially with devstack, this also happens
e.g. on f18.
- it is caused by quantum calling internally ip netns exec
- ip remounts /sys and destroys /sys/cgroups
- systemctl tries to access /sys/cgroups and returns that misleading
error message:
Failed to get D-Bus connection: No connection to service manager.
- no ability to shut down the system cleanly anymore.

So, I think, these are at least two errors:
- ip remounting /sys and destroying the hierarchy below that mountpoint
- (at least) misleading error message from systemctl. My proposal here
would be to try to remount /sys/cgroups etc. because this error might
happen even by accident

Matthias


-- 
Matthias Runge <mrunge(a)matthias-runge.de&gt;
               <mrunge(a)fedoraproject.org&gt;

[Daniel P. Berrange]

On Sun, Feb 10, 2013 at 10:08:11PM +0100, Alan Pevec wrote:
>  2013/2/8 Kyle Mestery (kmestery) <kmestery(a)cisco.com&gt;:
>  > It's definitely namespace related. When I turned namespaces off in my localrc,
>  > DBus stayed up and things worked ok. Now the digging on how they two are
>  > interacting in such a destructive way can begin!
>  
>  Maybe https://bugzilla.redhat.com/show_bug.cgi?id=882047 ? 
Yes, I'd say that would definitely cause this kind of behaviour. Their
fix of re-mounting the / with PRIVATE flag is the correct solution to
this bug. So hopefully if Kyle applies that update, this should work
correctly again.

Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

[Kyle Mestery (kmestery)]

On Feb 11, 2013, at 3:43 AM, Daniel P. Berrange <berrange(a)redhat.com&gt; wrote:
>  On Sun, Feb 10, 2013 at 10:08:11PM +0100, Alan Pevec wrote:
> > 2013/2/8 Kyle Mestery (kmestery) <kmestery(a)cisco.com&gt;:
> >> It's definitely namespace related. When I turned namespaces off in my
> localrc,
> >> DBus stayed up and things worked ok. Now the digging on how they two are
> >> interacting in such a destructive way can begin!
> > 
> > Maybe https://bugzilla.redhat.com/show_bug.cgi?id=882047 ?
>  
>  Yes, I'd say that would definitely cause this kind of behaviour. Their
>  fix of re-mounting the / with PRIVATE flag is the correct solution to
>  this bug. So hopefully if Kyle applies that update, this should work
>  correctly again.
>  
>  Daniel 
That was indeed the problem, and the update to iproute fixed my problem.
Thanks for pointing me in the right direction! I'll close the Quantum bug I
opened and reference this Fedora bug.

Kyle