dead dashboard after update & reboot

Hi,

Today after a "yum update" which did upgrade parts of my openstack 
installation and a reboot, I ended up with a "500 internal server 
error".

Here's last 100 log entried in httpd/error_log

centos 6.3 + epel (up to date :> ), selinux is in permissive mode.

-- 
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

dead dashboard after update & reboot (nova-api doesn't start)

On 11.01.2013 21:30, Nux! wrote:
>  Hi,
>
>  Today after a "yum update" which did upgrade parts of my openstack
>  installation and a reboot, I ended up with a "500 internal server
>  error".
>
>  Here's last 100 log entried in httpd/error_log
>
>  centos 6.3 + epel (up to date :> ), selinux is in permissive mode. 
Hi,

Apparently nova-api doesn't start anymore. The initscript reports "OK" 
but a second later it's dead:
openstack-nova-api dead but pid file exists

api.log fragment here http://fpaste.org/iZJr/

==
2013-01-12 11:45:20 19519 INFO nova.osapi_compute.wsgi.server [-] 
(19519) wsgi starting up on http://0.0.0.0:8774/
2013-01-12 11:45:20 19481 CRITICAL nova [-] [Errno 98] Address already 
in use
2013-01-12 11:45:21 19488 INFO nova.service [-] Parent process has died 
unexpectedly, exiting
==

But there is nothing listening on 8774 according to netstat and lsof, 
so why is it complaining?

-- 
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

dead dashboard after update & reboot (nova-api doesn't start)

[Matthias Runge]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 01/12/2013 10:52 AM, Nux! wrote:
>  On 11.01.2013 21:30, Nux! wrote:
> > Hi,
> > 
> > Today after a "yum update" which did upgrade parts of my
> > openstack installation and a reboot, I ended up with a "500
> > internal server error".
> > 
> > Here's last 100 log entried in httpd/error_log
> > 
> > centos 6.3 + epel (up to date :> ), selinux is in permissive
> > mode.
>  
>  Hi,
>  
>  Apparently nova-api doesn't start anymore. The initscript reports
>  "OK" but a second later it's dead: openstack-nova-api dead but pid
>  file exists
>  
>  api.log fragment here http://fpaste.org/iZJr/
>  
>  == 2013-01-12 11:45:20 19519 INFO nova.osapi_compute.wsgi.server
>  [-] (19519) wsgi starting up on http://0.0.0.0:8774/ 2013-01-12
>  11:45:20 19481 CRITICAL nova [-] [Errno 98] Address already in use 
>  2013-01-12 11:45:21 19488 INFO nova.service [-] Parent process has
>  died unexpectedly, exiting ==
>  
>  But there is nothing listening on 8774 according to netstat and
>  lsof, so why is it complaining?
>   
By any chance, did you an upgrade from Essex to Folsom release without
doing migration steps?

Another idea: Did you install OpenStack with openstack-demo-install?
Cinder/nova-volume requires a Volume group named cinder-volume
You might also want to stop nova-volume service and start
cinder-volumes/nova-volumes. If you installed OpenStack following the
TestDay page, cinder/nova-volume service is not started by default,
and also a volume group is not created persistantly.

Dashboard in it's current version requires a volume service enabled
(and running/answering to requests!)

An output of openstack-status and from keystone service-list will help
us here.
- -- 
Matthias Runge <mrunge(a)matthias-runge.de&gt;
               <mrunge(a)fedoraproject.org&gt;
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.13 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQEcBAEBAgAGBQJQ8V5hAAoJEOnz8qQwcaIWyCYH/2fnNZ/YlRbQiqqYiMpk5DU5
> YEU7Q3KGx/LjLLw0r9z2WITcIAP09RFRXY6HX0sVblQx/jJor3el/ZW8krzTD7hN
> D8B4kIqeDDXq9RIEP84SQ8qouhfYPQLmojaMGBagrksTfHf16H1a+wqx8ShQwmdi
> QwT0+Q+mH8MHpzDeXFNcWVs1TrvaRd1ZkSp2Ks+d1cTxrNQyl2HO2rPePjGQDOZB
> aqPVPtKzD+6o8s7KUan9DnH7T4DxITZk6SR9MLhHMl3ZSg3iGPtyyk135n+wxaQ7
> bGXL4ImY0V8T4Oi0gt1Ua++7xfch1s/eBzQDRthCLSuXokQjGPaeqVmvWsXUWd4=
> =zba/
> -----END PGP SIGNATURE-----

dead dashboard after update & reboot (nova-api doesn't start)

On 12.01.2013 13:00, Matthias Runge wrote:
>  -----BEGIN PGP SIGNED MESSAGE-----
>  Hash: SHA1
>
>  On 01/12/2013 10:52 AM, Nux! wrote:
> > On 11.01.2013 21:30, Nux! wrote:
> >> Hi,
> >>
> >> Today after a "yum update" which did upgrade parts of my
> >> openstack installation and a reboot, I ended up with a "500
> >> internal server error".
> >>
> >> Here's last 100 log entried in httpd/error_log
> >>
> >> centos 6.3 + epel (up to date :> ), selinux is in permissive
> >> mode.
> >
> > Hi,
> >
> > Apparently nova-api doesn't start anymore. The initscript reports
> > "OK" but a second later it's dead: openstack-nova-api dead but pid
> > file exists
> >
> > api.log fragment here http://fpaste.org/iZJr/
> >
> > == 2013-01-12 11:45:20 19519 INFO nova.osapi_compute.wsgi.server
> > [-] (19519) wsgi starting up on http://0.0.0.0:8774/ 2013-01-12
> > 11:45:20 19481 CRITICAL nova [-] [Errno 98] Address already in use
> > 2013-01-12 11:45:21 19488 INFO nova.service [-] Parent process has
> > died unexpectedly, exiting ==
> >
> > But there is nothing listening on 8774 according to netstat and
> > lsof, so why is it complaining?
> >
>  By any chance, did you an upgrade from Essex to Folsom release 
>  without
>  doing migration steps?
>
>  Another idea: Did you install OpenStack with openstack-demo-install?
>  Cinder/nova-volume requires a Volume group named cinder-volume
>  You might also want to stop nova-volume service and start
>  cinder-volumes/nova-volumes. If you installed OpenStack following the
>  TestDay page, cinder/nova-volume service is not started by default,
>  and also a volume group is not created persistantly.
>
>  Dashboard in it's current version requires a volume service enabled
>  (and running/answering to requests!)
>
>  An output of openstack-status and from keystone service-list will 
>  help
>  us here.
>  - --
>  Matthias Runge <mrunge(a)matthias-runge.de&gt;
>                 <mrunge(a)fedoraproject.org&gt;
> >  -----BEGIN PGP SIGNATURE-----
> >  Version: GnuPG v1.4.13 (GNU/Linux)
> >  Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
> >
> >  iQEcBAEBAgAGBQJQ8V5hAAoJEOnz8qQwcaIWyCYH/2fnNZ/YlRbQiqqYiMpk5DU5
> >  YEU7Q3KGx/LjLLw0r9z2WITcIAP09RFRXY6HX0sVblQx/jJor3el/ZW8krzTD7hN
> >  D8B4kIqeDDXq9RIEP84SQ8qouhfYPQLmojaMGBagrksTfHf16H1a+wqx8ShQwmdi
> >  QwT0+Q+mH8MHpzDeXFNcWVs1TrvaRd1ZkSp2Ks+d1cTxrNQyl2HO2rPePjGQDOZB
> >  aqPVPtKzD+6o8s7KUan9DnH7T4DxITZk6SR9MLhHMl3ZSg3iGPtyyk135n+wxaQ7
> >  bGXL4ImY0V8T4Oi0gt1Ua++7xfch1s/eBzQDRthCLSuXokQjGPaeqVmvWsXUWd4=
> >  =zba/
> >  -----END PGP SIGNATURE----- 
>  _______________________________________________
>  cloud mailing list
>  cloud(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/cloud 
Hello Matthias,

This is not an upgrade from Essex, it was/is Folsom, "yum update" 
brought in some minor version changes.
The software was installed via openstack-demo-install indeed, but I'm 
not using cinder or swift (the services are stopped). I have started 
however all cinder services and created the cinder-volumes vg, just to 
rule cinder out, no luck.
Let's say the dashboard requires cinder/nova-volume running, but why is 
nova-api service not starting and complaining about port 8774 being in 
use when it's not?


openstack-status http://fpaste.org/R15w/

keystone service-list

+----------------------------------+-------------+--------------+---------------------------+
|                id                |     name    |     type     |       
description        |
+----------------------------------+-------------+--------------+---------------------------+
| 145f1f198b934b65b5d66f78915eca6e |     ec2     |     ec2      |  EC2 
Compatibility Layer  |
| 19379eb755c04a90866c695a9a7dafb6 |   keystone  |   identity   | 
Keystone Identity Service |
| 8923f7ab3cce464d99151d39aa26e17e |     nova    |   compute    |    
Nova Compute Service   |
| 94f2aa75d6384bf08152dc629ee108dc | nova-volume |    volume    |    
Nova Volume Service    |
| c1e219f0c680419bbddce335fb2228e7 |    glance   |    image     |    
Glance Image Service   |
| dbfe96a9a2f04bf69f62e3855e85279c |   horizon   |  dashboard   |    
OpenStack Dashboard    |
| dfb76c3d811b4563aea37cfedf65220f |    cinder   |    volume    |       
Cinder Service      |
| e218dc54e2944fb0bd48eb7ddd70c203 |    swift    | object-store |       
Swift Service       |
+----------------------------------+-------------+--------------+---------------------------+



-- 
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

[Pádraig Brady]

On 01/11/2013 09:30 PM, Nux! wrote:
>  Hi,
>
>  Today after a "yum update" which did upgrade parts of my openstack installation
> and a reboot, I ended up with a "500 internal server error". 
What were the old and new versions of nova?

>  Here's last 100 log entried in httpd/error_log
>
>  centos 6.3 + epel (up to date :> ), selinux is in permissive mode. 
This could be due to trying to start the metadata service twice:

https://lists.launchpad.net/openstack/msg19294.html
https://bugzilla.redhat.com/show_bug.cgi?id=877606

You can probably get around the issue by disabling
the separate metadata service or adjusting the
enabled_apis option in nova.conf

thanks,
Pádraig.

[Pádraig Brady]

On 01/13/2013 01:58 AM, Pádraig Brady wrote:
>  On 01/11/2013 09:30 PM, Nux! wrote:
> > Hi,
> >
> > Today after a "yum update" which did upgrade parts of my openstack
> installation and a reboot, I ended up with a "500 internal server error".
>
>  What were the old and new versions of nova?
>
> > Here's last 100 log entried in httpd/error_log
> >
> > centos 6.3 + epel (up to date :> ), selinux is in permissive mode.
>
>  This could be due to trying to start the metadata service twice:
>
>  https://lists.launchpad.net/openstack/msg19294.html
>  https://bugzilla.redhat.com/show_bug.cgi?id=877606
>
>  You can probably get around the issue by disabling
>  the separate metadata service or adjusting the
>  enabled_apis option in nova.conf 
By the way, did you manually enable the metadata service,
as openstack-demo-install doesn't seem to at least.

thanks,
Pádraig.

On 13.01.2013 02:11, Pádraig Brady wrote:
>  On 01/13/2013 01:58 AM, Pádraig Brady wrote:
> > On 01/11/2013 09:30 PM, Nux! wrote:
> >> Hi,
> >>
> >> Today after a "yum update" which did upgrade parts of my openstack 
> >> installation and a reboot, I ended up with a "500 internal server 
> >> error".
> >
> > What were the old and new versions of nova?
> >
> >> Here's last 100 log entried in httpd/error_log
> >>
> >> centos 6.3 + epel (up to date :> ), selinux is in permissive mode.
> >
> > This could be due to trying to start the metadata service twice:
> >
> > https://lists.launchpad.net/openstack/msg19294.html
> > https://bugzilla.redhat.com/show_bug.cgi?id=877606
> >
> > You can probably get around the issue by disabling
> > the separate metadata service or adjusting the
> > enabled_apis option in nova.conf
>
>  By the way, did you manually enable the metadata service,
>  as openstack-demo-install doesn't seem to at least.
>
>  thanks,
>  Pádraig.
>  _______________________________________________
>  cloud mailing list
>  cloud(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/cloud 
That was probably by mistake. :-/

The dashboard is half-functioning now (no CSS, ajax etc) and 
nova-compute refuses to start:
2013-01-13 10:54:32 INFO nova.compute.manager 
[req-9c31b571-69e4-4e07-8cb2-cdf394837b72 None None] [instance: 
9aec8016-e904-41ac-80ad-ee2cf95ab393] Rebooting instance after 
nova-compute restart.
2013-01-13 10:54:32 INFO nova.virt.libvirt.firewall 
[req-9c31b571-69e4-4e07-8cb2-cdf394837b72 None None] [instance: 
9aec8016-e904-41ac-80ad-ee2cf95ab393] Called setup_basic_filtering in 
nwfilter
2013-01-13 10:54:32 INFO nova.virt.libvirt.firewall 
[req-9c31b571-69e4-4e07-8cb2-cdf394837b72 None None] [instance: 
9aec8016-e904-41ac-80ad-ee2cf95ab393] Ensuring static filters
2013-01-13 10:54:36 8943 CRITICAL nova [-] Requested operation is not 
valid: domain is already running

All the VMs are of course running, I'm using 
"start_guests_on_host_boot=true".


-- 
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

On 13.01.2013 08:56, Nux! wrote:
>  On 13.01.2013 02:11, Pádraig Brady wrote:
> > On 01/13/2013 01:58 AM, Pádraig Brady wrote:
> >> On 01/11/2013 09:30 PM, Nux! wrote:
> >>> Hi,
> >>>
> >>> Today after a "yum update" which did upgrade parts of my openstack
>
> >>> installation and a reboot, I ended up with a "500 internal server 
> >>> error".
> >>
> >> What were the old and new versions of nova?
> >>
> >>> Here's last 100 log entried in httpd/error_log
> >>>
> >>> centos 6.3 + epel (up to date :> ), selinux is in permissive mode.
> >>
> >> This could be due to trying to start the metadata service twice:
> >>
> >> https://lists.launchpad.net/openstack/msg19294.html
> >> https://bugzilla.redhat.com/show_bug.cgi?id=877606
> >>
> >> You can probably get around the issue by disabling
> >> the separate metadata service or adjusting the
> >> enabled_apis option in nova.conf
> >
> > By the way, did you manually enable the metadata service,
> > as openstack-demo-install doesn't seem to at least.
> >
> > thanks,
> > Pádraig.
> > _______________________________________________
> > cloud mailing list
> > cloud(a)lists.fedoraproject.org
> > https://admin.fedoraproject.org/mailman/listinfo/cloud
>
>  That was probably by mistake. :-/
>
>  The dashboard is half-functioning now (no CSS, ajax etc) and
>  nova-compute refuses to start:
>  2013-01-13 10:54:32 INFO nova.compute.manager
>  [req-9c31b571-69e4-4e07-8cb2-cdf394837b72 None None] [instance:
>  9aec8016-e904-41ac-80ad-ee2cf95ab393] Rebooting instance after
>  nova-compute restart.
>  2013-01-13 10:54:32 INFO nova.virt.libvirt.firewall
>  [req-9c31b571-69e4-4e07-8cb2-cdf394837b72 None None] [instance:
>  9aec8016-e904-41ac-80ad-ee2cf95ab393] Called setup_basic_filtering in
>  nwfilter
>  2013-01-13 10:54:32 INFO nova.virt.libvirt.firewall
>  [req-9c31b571-69e4-4e07-8cb2-cdf394837b72 None None] [instance:
>  9aec8016-e904-41ac-80ad-ee2cf95ab393] Ensuring static filters
>  2013-01-13 10:54:36 8943 CRITICAL nova [-] Requested operation is not
>  valid: domain is already running
>
>  All the VMs are of course running, I'm using
>  "start_guests_on_host_boot=true". 
I think I'll leave this off as it creates issues and rely on 
libvirt-guests service to start them up.

-- 
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

[Matthias Runge]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 01/13/2013 09:56 AM, Nux! wrote:
>  
>  That was probably by mistake. :-/
>  
>  The dashboard is half-functioning now (no CSS, ajax etc) and 
Sounds like /static from your web root is not accessible any more.
That should be a config option in
/etc/httpd/conf.d/openstack-dashboard.conf
Please also validate /usr/share/openstack-dashboard/static is readable.

If you simply reinstall openstack-dashboard, I'd do:
yum remove openstack-dashboard
rm -rf /etc/openstack-dashboard
rm -rf /usr/share/openstack-dashboard
rm -f /etc/httpd/conf.d/openstack-dashboard.conf
yum install openstack-dashboard

That should solve that issue.

Matthias
- -- 
Matthias Runge <mrunge(a)matthias-runge.de&gt;
               <mrunge(a)fedoraproject.org&gt;
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.13 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQEcBAEBAgAGBQJQ8yMXAAoJEOnz8qQwcaIWhowH/1EEJ+Y/CdTyJ10wRuO4nk0Y
> UMR4MJn5SWaYED6N9hFRNnsd60KmNCYaPTfQnn0LX+ZFVmbcCMXe+enKYY4T3hHP
> T2UvwG7RwcSOw2z98Bl+eEBEgLhAaurmPyfHpckbHsW8D5VwsA89hdLpkyGSWKvz
> VD1b77ZcFG2bM1hXH8hFRouCtli5jgmbC5ywMaQibZiaGnbpIJMDBBtFpQ65G3/2
> o7hIo/H4S7Q68CPcOGihvO6Rv/DRA3OUcfbEmYGjj7N9O2PjYgbOyqk6oKdoSi2L
> cvI9I5Vi0j9LHL55dsZkMVsKb0WCQFeSG5hX5ZBZRSrWwgmd7JDMYxmMdsx2XYQ=
> =UF63
> -----END PGP SIGNATURE-----

On 13.01.2013 21:11, Matthias Runge wrote:
>  -----BEGIN PGP SIGNED MESSAGE-----
>  Hash: SHA1
>
>  On 01/13/2013 09:56 AM, Nux! wrote:
> >
> > That was probably by mistake. :-/
> >
> > The dashboard is half-functioning now (no CSS, ajax etc) and
>
>  Sounds like /static from your web root is not accessible any more.
>  That should be a config option in
>  /etc/httpd/conf.d/openstack-dashboard.conf
>  Please also validate /usr/share/openstack-dashboard/static is 
>  readable.
>
>  If you simply reinstall openstack-dashboard, I'd do:
>  yum remove openstack-dashboard
>  rm -rf /etc/openstack-dashboard
>  rm -rf /usr/share/openstack-dashboard
>  rm -f /etc/httpd/conf.d/openstack-dashboard.conf
>  yum install openstack-dashboard
>
>  That should solve that issue. 
Indeed it did. Thanks!
The static dir was readable, nothing seemed broken in an obvious 
manner. :-/


-- 
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

On 13.01.2013 01:58, Pádraig Brady wrote:
>  On 01/11/2013 09:30 PM, Nux! wrote:
> > Hi,
> >
> > Today after a "yum update" which did upgrade parts of my openstack 
> > installation and a reboot, I ended up with a "500 internal server 
> > error".
>
>  What were the old and new versions of nova?
>
> > Here's last 100 log entried in httpd/error_log
> >
> > centos 6.3 + epel (up to date :> ), selinux is in permissive mode.
>
>  This could be due to trying to start the metadata service twice:
>
>  https://lists.launchpad.net/openstack/msg19294.html
>  https://bugzilla.redhat.com/show_bug.cgi?id=877606 
That was it. Thanks! Don't know why these sites never came up in my 
searches. :(
Anyway, now the dashboard looks like from the 90s again, maybe because 
I fiddled with "python manage.py compress"? How do I fix it?



-- 
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro