[Bug 1313511] New: Erlang Port Mapper crashes at startup

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1313511

Bug ID: 1313511
           Summary: Erlang Port Mapper crashes at startup
           Product: Red Hat OpenStack
           Version: 7.0
         Component: erlang
          Assignee: [e-mail]
          Reporter: [e-mail]
        QA Contact: [e-mail]
                CC: [e-mail], [e-mail],
                    [e-mail], [e-mail],
                    [e-mail], [e-mail], [e-mail],
                    [e-mail], [e-mail], [e-mail]
        Depends On: 1310808, 1312452

+++ This bug was initially created as a clone of Bug #1312452 +++

+++ This bug was initially created as a clone of Bug #1310808 +++

Description of problem:

After upgrading from erlang-R16B-03.11 to erlang-R16B-03.13 epmd has started
crashing at start up in some of our machines.

Version-Release number of selected component (if applicable):

erlang-R16B-03.13

How reproducible:

Start epmd daemon in a machine without ipv6 address

Steps to Reproduce:
[root@myhost ~]# epmd -debug
epmd: Wed Feb 17 08:27:23 2016: epmd running - daemon = 0
*** buffer overflow detected ***: epmd terminated
======= Backtrace: =========
/lib64/libc.so.6(__fortify_fail+0x37)[0x7f01fe269ac7]
/lib64/libc.so.6(+0x10bc80)[0x7f01fe267c80]
/lib64/libc.so.6(+0x10da37)[0x7f01fe269a37]
epmd[0x403a96]
epmd[0x401915]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f01fe17db15]
epmd[0x401d0d]
======= Memory map: ========
00400000-0040c000 r-xp 00000000 fd:00 34108967                          
/usr/lib64/erlang/erts-5.10.4/bin/epmd
0060b000-0060c000 r--p 0000b000 fd:00 34108967                          
/usr/lib64/erlang/erts-5.10.4/bin/epmd
0060c000-0060d000 rw-p 0000c000 fd:00 34108967                          
/usr/lib64/erlang/erts-5.10.4/bin/epmd
0149f000-014c0000 rw-p 00000000 00:00 0                                  [heap]
7f01fcc70000-7f01fcc85000 r-xp 00000000 fd:00 50663                     
/usr/lib64/libz.so.1.2.7
7f01fcc85000-7f01fce84000 ---p 00015000 fd:00 50663                     
/usr/lib64/libz.so.1.2.7
7f01fce84000-7f01fce85000 r--p 00014000 fd:00 50663                     
/usr/lib64/libz.so.1.2.7
7f01fce85000-7f01fce86000 rw-p 00015000 fd:00 50663                     
/usr/lib64/libz.so.1.2.7
7f01fce86000-7f01fce95000 r-xp 00000000 fd:00 66728                     
/usr/lib64/libbz2.so.1.0.6
7f01fce95000-7f01fd094000 ---p 0000f000 fd:00 66728                     
/usr/lib64/libbz2.so.1.0.6
7f01fd094000-7f01fd095000 r--p 0000e000 fd:00 66728                     
/usr/lib64/libbz2.so.1.0.6
7f01fd095000-7f01fd096000 rw-p 0000f000 fd:00 66728                     
/usr/lib64/libbz2.so.1.0.6
7f01fd096000-7f01fd0ba000 r-xp 00000000 fd:00 16885                     
/usr/lib64/liblzma.so.5.0.99
7f01fd0ba000-7f01fd2b9000 ---p 00024000 fd:00 16885                     
/usr/lib64/liblzma.so.5.0.99
7f01fd2b9000-7f01fd2ba000 r--p 00023000 fd:00 16885                     
/usr/lib64/liblzma.so.5.0.99
7f01fd2ba000-7f01fd2bb000 rw-p 00024000 fd:00 16885                     
/usr/lib64/liblzma.so.5.0.99
7f01fd2bb000-7f01fd2d0000 r-xp 00000000 fd:00 66743                     
/usr/lib64/libelf-0.163.so
7f01fd2d0000-7f01fd4cf000 ---p 00015000 fd:00 66743                     
/usr/lib64/libelf-0.163.so
7f01fd4cf000-7f01fd4d0000 r--p 00014000 fd:00 66743                     
/usr/lib64/libelf-0.163.so
7f01fd4d0000-7f01fd4d1000 rw-p 00015000 fd:00 66743                     
/usr/lib64/libelf-0.163.so
7f01fd4d1000-7f01fd4d5000 r-xp 00000000 fd:00 66753                     
/usr/lib64/libattr.so.1.1.0
7f01fd4d5000-7f01fd6d4000 ---p 00004000 fd:00 66753                     
/usr/lib64/libattr.so.1.1.0
7f01fd6d4000-7f01fd6d5000 r--p 00003000 fd:00 66753                     
/usr/lib64/libattr.so.1.1.0
7f01fd6d5000-7f01fd6d6000 rw-p 00004000 fd:00 66753                     
/usr/lib64/libattr.so.1.1.0
7f01fd6d6000-7f01fd6ec000 r-xp 00000000 fd:00 16866                     
/usr/lib64/libpthread-2.17.so
7f01fd6ec000-7f01fd8ec000 ---p 00016000 fd:00 16866                     
/usr/lib64/libpthread-2.17.so
7f01fd8ec000-7f01fd8ed000 r--p 00016000 fd:00 16866                     
/usr/lib64/libpthread-2.17.so
7f01fd8ed000-7f01fd8ee000 rw-p 00017000 fd:00 16866                     
/usr/lib64/libpthread-2.17.so
7f01fd8ee000-7f01fd8f2000 rw-p 00000000 00:00 0 
7f01fd8f2000-7f01fd907000 r-xp 00000000 fd:00 142                       
/usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f01fd907000-7f01fdb06000 ---p 00015000 fd:00 142                       
/usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f01fdb06000-7f01fdb07000 r--p 00014000 fd:00 142                       
/usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f01fdb07000-7f01fdb08000 rw-p 00015000 fd:00 142                       
/usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f01fdb08000-7f01fdb4d000 r-xp 00000000 fd:00 82543                     
/usr/lib64/libdw-0.163.so
7f01fdb4d000-7f01fdd4c000 ---p 00045000 fd:00 82543                     
/usr/lib64/libdw-0.163.so
7f01fdd4c000-7f01fdd4e000 r--p 00044000 fd:00 82543                     
/usr/lib64/libdw-0.163.so
7f01fdd4e000-7f01fdd4f000 rw-p 00046000 fd:00 82543                     
/usr/lib64/libdw-0.163.so
7f01fdd4f000-7f01fdd53000 r-xp 00000000 fd:00 66757                     
/usr/lib64/libcap.so.2.22
7f01fdd53000-7f01fdf52000 ---p 00004000 fd:00 66757                     
/usr/lib64/libcap.so.2.22
7f01fdf52000-7f01fdf53000 r--p 00003000 fd:00 66757                     
/usr/lib64/libcap.so.2.22
7f01fdf53000-7f01fdf54000 rw-p 00004000 fd:00 66757                     
/usr/lib64/libcap.so.2.22
7f01fdf54000-7f01fdf5b000 r-xp 00000000 fd:00 16870                     
/usr/lib64/librt-2.17.so
7f01fdf5b000-7f01fe15a000 ---p 00007000 fd:00 16870                     
/usr/lib64/librt-2.17.so
7f01fe15a000-7f01fe15b000 r--p 00006000 fd:00 16870                     
/usr/lib64/librt-2.17.so
7f01fe15b000-7f01fe15c000 rw-p 00007000 fd:00 16870                     
/usr/lib64/librt-2.17.so
7f01fe15c000-7f01fe312000 r-xp 00000000 fd:00 16840                     
/usr/lib64/libc-2.17.so
7f01fe312000-7f01fe512000 ---p 001b6000 fd:00 16840                     
/usr/lib64/libc-2.17.so
7f01fe512000-7f01fe516000 r--p 001b6000 fd:00 16840                     
/usr/lib64/libc-2.17.so
7f01fe516000-7f01fe518000 rw-p 001ba000 fd:00 16840                     
/usr/lib64/libc-2.17.so
7f01fe518000-7f01fe51d000 rw-p 00000000 00:00 0 
7f01fe51d000-7f01fe61e000 r-xp 00000000 fd:00 16848                     
/usr/lib64/libm-2.17.so
7f01fe61e000-7f01fe81d000 ---p 00101000 fd:00 16848                     
/usr/lib64/libm-2.17.so
7f01fe81d000-7f01fe81e000 r--p 00100000 fd:00 16848                     
/usr/lib64/libm-2.17.so
7f01fe81e000-7f01fe81f000 rw-p 00101000 fd:00 16848                     
/usr/lib64/libm-2.17.so
7f01fe81f000-7f01fe822000 r-xp 00000000 fd:00 16846                     
/usr/lib64/libdl-2.17.so
7f01fe822000-7f01fea21000 ---p 00003000 fd:00 16846                     
/usr/lib64/libdl-2.17.so
7f01fea21000-7f01fea22000 r--p 00002000 fd:00 16846                     
/usr/lib64/libdl-2.17.so
7f01fea22000-7f01fea23000 rw-p 00003000 fd:00 16846                     
/usr/lib64/libdl-2.17.so
7f01fea23000-7f01fea25000 r-xp 00000000 fd:00 16874                     
/usr/lib64/libutil-2.17.so
7f01fea25000-7f01fec24000 ---p 00002000 fd:00 16874                     
/usr/lib64/libutil-2.17.so
7f01fec24000-7f01fec25000 r--p 00001000 fd:00 16874                     
/usr/lib64/libutil-2.17.soAborted (core dumped)

Actual results:

epmd crashes

Expected results:

epmd daemon starts

Additional info:

strace shows that before the crash:
bind(3, {sa_family=AF_INET, sin_port=htons(4369),
sin_addr=inet_addr("0.0.0.0")}, 16) = 0
listen(3, 128)                          = 0
socket(PF_INET6, SOCK_STREAM, IPPROTO_IP) = -1 EAFNOSUPPORT (Address family not
supported by protocol)
select(4, [3], NULL, NULL, {5, 0})      = 0 (Timeout)
open("/dev/tty", O_RDWR|O_NOCTTY|O_NONBLOCK) = 4
writev(4, [{"*** ", 4}, {"buffer overflow detected", 24}, {" ***: ", 6},
{"epmd", 4}, {" terminated\n", 12}], 5*** buffer overflow detected ***: epmd
terminated
) = 50

systemctl start epmd@0.0.0.0

works fine.

--- Additional comment from John Eckersberg on 2016-02-22 14:32:37 EST ---

Can you attach the core dump?  Merely starting epmd on a host with no IPv6
addresses doesn't seem to be enough to trigger the bug, I'm unable to
reproduce:

[root@centos7 ~]# rpm -q erlang-erts
erlang-erts-R16B-03.13.el7.x86_64
[root@centos7 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP
qlen 1000
    link/ether 52:54:00:87:2d:f5 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.147/24 brd 192.168.122.255 scope global dynamic eth0
       valid_lft 2850sec preferred_lft 2850sec
[root@centos7 ~]# epmd -debug
epmd: Mon Feb 22 14:27:20 2016: epmd running - daemon = 0

(no crash for me)

--- Additional comment from Raúl Andrés on 2016-02-23 01:44 EST ---

--- Additional comment from Raúl Andrés on 2016-02-23 02:59:59 EST ---

I have IPv6 support disabled:

[root@myhost tmp]#  cat /proc/net/if_inet6                                
cat: /proc/net/if_inet6: No such file or directory
[root@myhost tmp]#  modprobe -c| grep ipv6
options ipv6 disable=1

--- Additional comment from John Eckersberg on 2016-02-26 12:44:37 EST ---

This also reproduces on OSP:

[root@rhel7 ~]# rpm -q erlang-erts
erlang-erts-R16B-03.10min.5.el7ost.x86_64
[root@rhel7 ~]# epmd -d 2>&1 | head
epmd: Fri Feb 26 12:38:16 2016: epmd running - daemon = 0
*** buffer overflow detected ***: epmd terminated
======= Backtrace: =========
/lib64/libc.so.6(__fortify_fail+0x37)[0x7fc11d6f4b37]
/lib64/libc.so.6(+0x10bcf0)[0x7fc11d6f2cf0]
/lib64/libc.so.6(+0x10daa7)[0x7fc11d6f4aa7]
epmd[0x403a96]
epmd[0x401915]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7fc11d608b15]
epmd[0x401d0d]
...

Disable IPv6 first as per
https://access.redhat.com/solutions/8709?tour=6#rhel7disable in order to
reproduce.

Referenced Bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=1310808
[Bug 1310808] Erlang Port Mapper crashes at startup if IPv6 disabled
completely
https://bugzilla.redhat.com/show_bug.cgi?id=1312452
[Bug 1312452] Erlang Port Mapper crashes at startup

[Bug 1313511] Erlang Port Mapper crashes at startup

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1313511

John Eckersberg  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
   Fixed In Version|                            |erlang-R16B-03.10min.8.el7o
                   |                            |st

[Bug 1313511] Erlang Port Mapper crashes at startup

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1313511

John Eckersberg  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |MODIFIED

[Bug 1313511] Erlang Port Mapper crashes at startup

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1313511

John Eckersberg  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |MODIFIED

RHEL Product and Program Management  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
           Keywords|                            |ZStream

[Bug 1313511] Erlang Port Mapper crashes at startup

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1313511
Bug 1313511 depends on bug 1310808, which changed state.

Bug 1310808 Summary: Erlang Port Mapper crashes at startup if IPv6 disabled completely
https://bugzilla.redhat.com/show_bug.cgi?id=1310808

What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ON_QA                       |CLOSED
         Resolution|---                         |ERRATA

[Bug 1313511] Erlang Port Mapper crashes at startup

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1313511
Bug 1313511 depends on bug 1312452, which changed state.

Bug 1312452 Summary: Erlang Port Mapper crashes at startup
https://bugzilla.redhat.com/show_bug.cgi?id=1312452

What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|VERIFIED                    |CLOSED
         Resolution|---                         |ERRATA

[Bug 1313511] Erlang Port Mapper crashes at startup

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1313511

John Skeoch  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
         QA Contact|[e-mail]           |[e-mail]

[Bug 1313511] Erlang Port Mapper crashes at startup

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1313511

errata-xmlrpc  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|MODIFIED                    |ON_QA

[Bug 1313511] Erlang Port Mapper crashes at startup

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1313511

errata-xmlrpc  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|MODIFIED                    |ON_QA

[Bug 1313511] Erlang Port Mapper crashes at startup

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1313511

nlevinki  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[e-mail]
         QA Contact|[e-mail]          |[e-mail]

[Bug 1313511] Erlang Port Mapper crashes at startup

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1313511

Udi  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ON_QA                       |VERIFIED

--- Comment #3 from Udi  ---
Verified in erlang-erts-R16B-03.10min.9.el7ost.x86_64

[Bug 1313511] Erlang Port Mapper crashes at startup

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1313511

errata-xmlrpc  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|VERIFIED                    |RELEASE_PENDING

[Bug 1313511] Erlang Port Mapper crashes at startup

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1313511

errata-xmlrpc  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RELEASE_PENDING             |CLOSED
         Resolution|---                         |ERRATA
        Last Closed|                            |2016-06-01 08:32:01

--- Comment #5 from errata-xmlrpc  ---
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1191