[Bug 1059913] New: Race condition creating .erlang.cookie

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1059913

            Bug ID: 1059913
           Summary: Race condition creating .erlang.cookie
           Product: Fedora
           Version: 20
         Component: rabbitmq-server
          Assignee: hubert.plociniczak(a)gmail.com
          Reporter: jeckersb(a)redhat.com
        QA Contact: extras-qa(a)fedoraproject.org
                CC: erlang(a)lists.fedoraproject.org,
                    hubert.plociniczak(a)gmail.com, lemenkov(a)gmail.com,
                    skottler(a)redhat.com



Description of problem:

There is a race condition when starting rabbitmq-server for the first time.

When the erlang runtime starts, it tries to read its cookie file (for rabbitmq,
/var/lib/rabbitmq/.erlang.cookie) and if it doesn't already exist, it generates
a new random cookie and creates the file.

The following two lines from the rabbitmq-service.service unit file are
involved:

ExecStart=/usr/lib/rabbitmq/bin/rabbitmq-server
ExecStartPost=/usr/lib/rabbitmq/bin/rabbitmqctl wait /var/run/rabbitmq/pid

The rabbitmq-server command returns before the service is up.  Therefor it is
required to exec the additional rabbitmqctl wait in order to make sure the
service starts all the way.  However both of these are erlang programs, and
they share the cookie startup code described previously.

There is variance on the order of events and the eventual error.  But generally
what happens is:

- ExecStart (rabbitmq-server) is run and exits.  The erlang runtime is now
booting in the background.

- ExecStartPost (rabbitmqctl) is run.

- rabbitmq-server determines the cookie file is not present, and generates a
new cookie.

- rabbitmqctl determines the cookie file is not present, and generates a new
cookie.

- rabbitmq-server writes the new cookie to disk and sets the file to read-only

- rabbitmqctl tries to open the cookie file read/write in order to write its
cookie, but errors with EACCESS because the file already exists and is read
only.

- The erlang runtime for rabbitmqctl crashes and the command returns with a
non-successful exit code.

- The entire service unit is marked as failed, and all of the processes are
killed by systemd.


Version-Release number of selected component (if applicable):

rabbitmq-server-3.1.5-1.fc20.noarch
erlang-R16B-03.1.fc20.x86_64

How reproducible:

There is some variability since it's a race.  I've provided my reproducer below
that works 100% of the time for me, inside a F20 VM.  In theory this behavior
should still exist if starting the service from the cli instead of rebooting,
but I can't reproduce it that way.

Steps to Reproduce:
1. install rabbitmq-server
2. systemctl enable rabbitmq-server.service
3. reboot

Actual results:
Service fails to start, see attachment of journalctl output for error

Expected results:
Service starts cleanly

Additional info:

This is really an erlang bug, but the workaround for rabbit is simple (I'll
post a patch in a followup).  I'll run down the erlang bit separately but it
will take longer, so it makes sense to apply the workaround here until erlang
is fixed.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug 1059913] Race condition creating .erlang.cookie

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1059913



--- Comment #1 from John Eckersberg <jeckersb(a)redhat.com&gt; ---
Created attachment 857656
  --> https://bugzilla.redhat.com/attachment.cgi?id=857656&action=edit
journalctl output from failed rabbitmq-server startup

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug 1059913] Race condition creating .erlang.cookie

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1059913



--- Comment #2 from John Eckersberg <jeckersb(a)redhat.com&gt; ---
Created attachment 857863
  --> https://bugzilla.redhat.com/attachment.cgi?id=857863&action=edit
rabbitmq-server-cookie-race.patch

This patch updates the systemd service to run `rabbitmqctl status` before
starting the rabbitmq-server process.  This ensures the erlang cookie is
created before starting the service.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug 1059913] Race condition creating .erlang.cookie

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1059913

Richard W.M. Jones <rjones(a)redhat.com&gt; changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |rjones(a)redhat.com
           Assignee|hubert.plociniczak(a)gmail.co |rjones(a)redhat.com
                   |m                           |



-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug 1059913] Race condition creating .erlang.cookie

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1059913



--- Comment #3 from Richard W.M. Jones <rjones(a)redhat.com&gt; ---
I performed the steps given in the bug description and the patch
works for me in a Rawhide VM.

The patch also looks reasonable to me and low risk, so I'm going to
backport it to Fedora 20.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug 1059913] Race condition creating .erlang.cookie

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1059913

Fedora Update System <updates(a)fedoraproject.org&gt; changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |MODIFIED



-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug 1059913] Race condition creating .erlang.cookie

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1059913



--- Comment #4 from Fedora Update System <updates(a)fedoraproject.org&gt; ---
rabbitmq-server-3.1.5-5.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/rabbitmq-server-3.1.5-5.fc20

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug 1059913] Race condition creating .erlang.cookie

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1059913

Fedora Update System <updates(a)fedoraproject.org&gt; changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|MODIFIED                    |ON_QA



--- Comment #5 from Fedora Update System <updates(a)fedoraproject.org&gt; ---
Package rabbitmq-server-3.1.5-5.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing rabbitmq-server-3.1.5-5.fc20'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2014-4722/rabbitmq-server-...
then log in and leave karma (feedback).

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug 1059913] Race condition creating .erlang.cookie

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1059913



--- Comment #6 from Sam Kottler <skottler(a)redhat.com&gt; ---
I'll merge rawhide into EPEL7 when I'm at a machine with my certs on it.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug 1059913] Race condition creating .erlang.cookie

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1059913

Alan Pevec <apevec(a)redhat.com&gt; changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |apevec(a)redhat.com



--- Comment #7 from Alan Pevec <apevec(a)redhat.com&gt; ---
Instead of messing with ExecStartPost, it would be more reliable to switch
systemd service to Type=notify just need someone familiar with rabbitmq to tell
us where to put sd_notify to send notification when service is ready.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug 1059913] Race condition creating .erlang.cookie

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1059913

Fedora Update System <updates(a)fedoraproject.org&gt; changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ON_QA                       |CLOSED
   Fixed In Version|                            |rabbitmq-server-3.1.5-5.fc2
                   |                            |0
         Resolution|---                         |ERRATA
        Last Closed|                            |2014-04-14 18:47:32



--- Comment #8 from Fedora Update System <updates(a)fedoraproject.org&gt; ---
rabbitmq-server-3.1.5-5.fc20 has been pushed to the Fedora 20 stable
repository.  If problems still persist, please make note of it in this bug
report.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug 1059913] Race condition creating .erlang.cookie

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1059913

Peter Lemenkov <lemenkov(a)gmail.com&gt; changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Blocks|                            |1103524




Referenced Bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=1103524
[Bug 1103524] Change rabbitmq-server systemd service to Type=notify
-- 
You are receiving this mail because:
You are on the CC list for the bug.