[Bug 1144100] New: rabbitmq restarts fail randomly

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1144100

            Bug ID: 1144100
           Summary: rabbitmq restarts fail randomly
           Product: Fedora EPEL
           Version: epel7
         Component: rabbitmq-server
          Assignee: lemenkov(a)gmail.com
          Reporter: bnemec(a)redhat.com
        QA Contact: extras-qa(a)fedoraproject.org
                CC: bnemec(a)redhat.com, erlang(a)lists.fedoraproject.org,
                    extras-qa(a)fedoraproject.org,
                    hubert.plociniczak(a)gmail.com, imcleod(a)redhat.com,
                    jeckersb(a)redhat.com, josh(a)fornwall.com,
                    lemenkov(a)gmail.com, lnie(a)redhat.com,
                    ohadlevy(a)redhat.com, rjones(a)redhat.com, s(a)shk.io
        Depends On: 1059028



+++ This bug was initially created as a clone of Bug #1059028 +++

This still appears to be a problem in the EPEL version of rabbitmq-server.

Description of problem:

I originally encountered this issue when installing OpenStack via devstack but
have since been able to reproduce it by simply executing commands inside of a
fresh F20 install.

Version-Release number of selected component (if applicable):

rabbitmq-server-3.1.5-1.fc20.noarch

How reproducible:

Occurs regularly but not 100% of the time

Steps to Reproduce:

/sbin/service rabbitmq-server stop
/sbin/service rabbitmq-server start
rabbitmqctl change_password guest newpassword


Actual results:

About half the time, on a freshly installed F20, this will fail, claiming the
node cannot be contacted.  A typical error message:

[root@cob-dell5 ~]# rabbitmqctl change_password guest ozrootpw
Changing password for user "guest" ...
Error: unable to connect to node 'rabbit@cob-dell5': nodedown

DIAGNOSTICS
===========

nodes in question: ['rabbit@cob-dell5']

hosts, their running nodes and ports:
- cob-dell5: [{rabbitmqctl2648,48609}]

current node details:
- node name: 'rabbitmqctl2648@cob-dell5'
- home dir: /var/lib/rabbitmq
- cookie hash: 8DNoVu56TqDYWypW7YXDJw==

Expected results:

Changing password for user "guest" ...
...done.

--- Additional comment from Ian McLeod on 2014-01-28 21:36:45 EST ---

See the following thread on rdo-list for another example of this issue:

https://www.redhat.com/archives/rdo-list/2013-December/msg00058.html

As well as this more recent one, indicating tha the problem persists:

https://www.redhat.com/archives/rdo-list/2014-January/msg00164.html

--- Additional comment from Ian McLeod on 2014-02-27 17:51:21 EST ---

Note to anyone following.  The devstack installer has a commit that attempts to
workaround this issue by retrying the rabbitmq restart:

https://github.com/openstack-dev/devstack/commit/f6c001faf6ac5728e07c6bbd...

Still, it would be nice to sort this out on the rabbitmq side eventually.

--- Additional comment from Fedora Admin XMLRPC Client on 2014-07-01 13:58:36
EDT ---

This package has changed ownership in the Fedora Package Database.  Reassigning
to the new owner of this component.

--- Additional comment from Fedora Update System on 2014-08-10 04:14:16 EDT ---

rabbitmq-server-3.1.5-9.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/rabbitmq-server-3.1.5-9.fc20

--- Additional comment from lnie on 2014-08-11 03:00:36 EDT ---

rabbitmq-server-3.1.5-9.fc20 works

--- Additional comment from Fedora Update System on 2014-08-14 22:35:38 EDT ---

Package rabbitmq-server-3.1.5-9.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing rabbitmq-server-3.1.5-9.fc20'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2014-9337/rabbitmq-server-...
then log in and leave karma (feedback).

--- Additional comment from Fedora Update System on 2014-08-15 20:28:02 EDT ---

rabbitmq-server-3.1.5-9.fc20 has been pushed to the Fedora 20 stable
repository.  If problems still persist, please make note of it in this bug
report.


Referenced Bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=1059028
[Bug 1059028] rabbitmq restarts fail randomly
-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug 1144100] rabbitmq restarts fail randomly

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1144100

Charles Crouch <ccrouch(a)redhat.com&gt; changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Severity|unspecified                 |high



-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug 1144100] rabbitmq restarts fail randomly

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1144100

John Eckersberg <jeckersb(a)redhat.com&gt; changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED
           Assignee|lemenkov(a)gmail.com          |jeckersb(a)redhat.com



-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug 1144100] rabbitmq restarts fail randomly

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1144100

Nathan G. Grennan <redhat-bugzilla(a)cygnusx-1.org&gt; changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |redhat-bugzilla(a)cygnusx-1.o
                   |                            |rg



--- Comment #1 from Nathan G. Grennan <redhat-bugzilla(a)cygnusx-1.org&gt; ---
I just ran into this issue. RHEL 7 Openstack has upgraded to 3.3.5-2. I
recompiled the rabbitmq-server 3.3.5-2 fc22 src.rpm from koji, and it fixed my
issue.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug 1144100] rabbitmq restarts fail randomly

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1144100

Ian Wienand <iwienand(a)redhat.com&gt; changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |iwienand(a)redhat.com



--- Comment #2 from Ian Wienand <iwienand(a)redhat.com&gt; ---
This is being seen in upstream testing for openstack

---
2014-12-15 19:16:20.015 | ++ sudo rabbitmqctl list_users
2014-12-15 19:16:20.239 | Error: unable to connect to node
'rabbit@devstack-centos7-rax-iad-100675': nodedown
2014-12-15 19:16:20.249 | 
2014-12-15 19:16:20.249 | DIAGNOSTICS
2014-12-15 19:16:20.249 | ===========
2014-12-15 19:16:20.249 | 
2014-12-15 19:16:20.249 | nodes in question:
['rabbit@devstack-centos7-rax-iad-100675']
2014-12-15 19:16:20.250 | 
2014-12-15 19:16:20.250 | hosts, their running nodes and ports:
2014-12-15 19:16:20.250 | - devstack-centos7-rax-iad-100675:
[{rabbitmqctl29293,39511}]
2014-12-15 19:16:20.250 | 
2014-12-15 19:16:20.250 | current node details:
2014-12-15 19:16:20.250 | - node name:
'rabbitmqctl29293@devstack-centos7-rax-iad-100675'
2014-12-15 19:16:20.250 | - home dir: /var/lib/rabbitmq
2014-12-15 19:16:20.250 | - cookie hash: KieJnx1pnllKbHVihGcDqA==
2014-12-15 19:16:20.250 | 
2014-12-15 19:16:20.256 | + out='Listing users ...'
2014-12-15 19:16:20.256 | + echo 'failed to list users'
---

This happens somewhat frequently on our centos 7 jobs (can search on [1] to see
examples)

[1] http://logstash.openstack.org

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug 1144100] rabbitmq restarts fail randomly

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1144100

Fedora Update System <updates(a)fedoraproject.org&gt; changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |MODIFIED



-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug 1144100] rabbitmq restarts fail randomly

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1144100



--- Comment #3 from Fedora Update System <updates(a)fedoraproject.org&gt; ---
rabbitmq-server-3.3.5-3.el7 has been submitted as an update for Fedora EPEL 7.
https://admin.fedoraproject.org/updates/rabbitmq-server-3.3.5-3.el7

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug 1144100] rabbitmq restarts fail randomly

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1144100

Fedora Update System <updates(a)fedoraproject.org&gt; changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|MODIFIED                    |ON_QA



--- Comment #4 from Fedora Update System <updates(a)fedoraproject.org&gt; ---
Package rabbitmq-server-3.3.5-3.el7:
* should fix your issue,
* was pushed to the Fedora EPEL 7 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=epel-testing rabbitmq-server-3.3.5-3.el7'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-EPEL-2014-4742/rabbitmq-se...
then log in and leave karma (feedback).

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug 1144100] rabbitmq restarts fail randomly

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1144100



--- Comment #5 from Fedora Update System <updates(a)fedoraproject.org&gt; ---
rabbitmq-server-3.3.5-4.el7 has been submitted as an update for Fedora EPEL 7.
https://admin.fedoraproject.org/updates/rabbitmq-server-3.3.5-4.el7

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug 1144100] rabbitmq restarts fail randomly

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1144100

Fedora Update System <updates(a)fedoraproject.org&gt; changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ON_QA                       |CLOSED
   Fixed In Version|                            |rabbitmq-server-3.3.5-4.el7
         Resolution|---                         |ERRATA
        Last Closed|                            |2015-01-05 21:09:08



--- Comment #6 from Fedora Update System <updates(a)fedoraproject.org&gt; ---
rabbitmq-server-3.3.5-4.el7 has been pushed to the Fedora EPEL 7 stable
repository.  If problems still persist, please make note of it in this bug
report.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug 1144100] rabbitmq restarts fail randomly

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1144100

David Hill <dhill(a)cloudops.com&gt; changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dhill(a)cloudops.com



--- Comment #7 from David Hill <dhill(a)cloudops.com&gt; ---
This created bugs in Ubuntu Vivid.

See : 
https://review.openstack.org/#/c/191090/

-- 
You are receiving this mail because:
You are on the CC list for the bug.