Re: kernel 4.19.x is unstable (random panics) -> reproducer

[Reindl Harald]

see attachment of https://bugzilla.redhat.com/show_bug.cgi?id=1659706#c3

the problem seems to be in conntrack in general and seems to be easily
reproduceable combined with a ipset with a single port-range like
"create DEBUG_PORT bitmap:port range 10022-10022"

to trigger it that easy for me it needs the combination of "-t filter -A
INBOUND -p all -m set --match-set DEBUG_PORT dst" with the single-range
ipset *and* connlimit rule

[Mon Dec 24 16:08:04 2018] general protection fault: 0000 [#1] SMP PTI
[Mon Dec 24 16:08:04 2018] CPU: 0 PID: 890 Comm: iptables Not tainted
4.19.12-200.fc28.x86_64 #1
[Mon Dec 24 16:08:04 2018] Hardware name: VMware, Inc. VMware Virtual
Platform/440BX Desktop Reference Platform, BIOS 6.00 07/03/2018
[Mon Dec 24 16:08:04 2018] RIP: 0010:rb_erase+0x216/0x370
[Mon Dec 24 16:08:04 2018] Code: e9 6b fe ff ff 4d 89 48 10 e9 91 fe ff
ff c3 48 89 06 48 89 d0 48 8b 52 10 e9 b1 fe ff ff 48 8b 07 48 89 c1 48
83 e1 fc 74 53 <48> 3b 79 10 0f 84 94 00 00 00 4c 89 41 08 4d 85 c0 75
4c a8 01 0f
[Mon Dec 24 16:08:04 2018] RSP: 0018:ffffa8a840ac7d28 EFLAGS: 00010286
[Mon Dec 24 16:08:04 2018] RAX: 89a7851347204a0d RBX: ffff940e233086c0
RCX: 89a7851347204a0c
[Mon Dec 24 16:08:04 2018] RDX: 0000000000000000 RSI: ffff940e22502730
RDI: ffff940e233086c0
[Mon Dec 24 16:08:04 2018] RBP: ffff940e25942ec8 R08: 0000000000000000
R09: ffffffffc01533de
[Mon Dec 24 16:08:04 2018] R10: ffff940e24aac268 R11: 00000000000003c0
R12: ffff940e22502730
[Mon Dec 24 16:08:04 2018] R13: ffff940e22502808 R14: ffff940e22502000
R15: ffff940e233086e0
[Mon Dec 24 16:08:04 2018] FS:  00007f29173e7740(0000)
GS:ffff940e25e00000(0000) knlGS:0000000000000000
[Mon Dec 24 16:08:04 2018] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Mon Dec 24 16:08:04 2018] CR2: 000055f6ca2fab48 CR3: 000000002242e003
CR4: 00000000001606f0
[Mon Dec 24 16:08:04 2018] Call Trace:
[Mon Dec 24 16:08:04 2018]  nf_conncount_destroy+0x58/0xc0 [nf_conncount]
[Mon Dec 24 16:08:04 2018]  cleanup_match+0x45/0x70
[Mon Dec 24 16:08:04 2018]  cleanup_entry+0x3e/0xc0
[Mon Dec 24 16:08:04 2018]  __do_replace+0x1ca/0x230
[Mon Dec 24 16:08:04 2018]  do_ipt_set_ctl+0x146/0x1a2
[Mon Dec 24 16:08:04 2018]  nf_setsockopt+0x44/0x70
[Mon Dec 24 16:08:04 2018]  __sys_setsockopt+0x82/0xe0
[Mon Dec 24 16:08:04 2018]  __x64_sys_setsockopt+0x20/0x30
[Mon Dec 24 16:08:04 2018]  do_syscall_64+0x5b/0x160
[Mon Dec 24 16:08:04 2018]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[Mon Dec 24 16:08:04 2018] RIP: 0033:0x7f29163124ea
[Mon Dec 24 16:08:04 2018] Code: ff ff ff c3 48 8b 15 b5 d9 2b 00 f7 d8
64 89 02 48 c7 c0 ff ff ff ff eb b1 0f 1f 80 00 00 00 00 49 89 ca b8 36
00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 86 d9 2b 00 f7 d8
64 89 01 48
[Mon Dec 24 16:08:04 2018] RSP: 002b:00007ffcc79633b8 EFLAGS: 00000202
ORIG_RAX: 0000000000000036
[Mon Dec 24 16:08:04 2018] RAX: ffffffffffffffda RBX: 000055f6ca2f8268
RCX: 00007f29163124ea
[Mon Dec 24 16:08:04 2018] RDX: 0000000000000040 RSI: 0000000000000000
RDI: 0000000000000004
[Mon Dec 24 16:08:04 2018] RBP: 000055f6ca2f9e50 R08: 0000000000000cf8
R09: 0000000000000000
[Mon Dec 24 16:08:04 2018] R10: 000055f6ca2f9e50 R11: 0000000000000202
R12: 000055f6ca2f9eb0
[Mon Dec 24 16:08:04 2018] R13: 000055f6ca2f8268 R14: 0000000000000c98
R15: 000055f6ca2f8260
[Mon Dec 24 16:08:04 2018] Modules linked in: bridge stp llc xt_recent
xt_set xt_connlimit nf_conncount xt_conntrack iptable_raw iptable_nat
nf_nat_ipv4 nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c
iptable_mangle ip_set_bitmap_port ip_set nfnetlink crct10dif_pclmul
crc32_pclmul ghash_clmulni_intel vmw_balloon vmxnet3 vmw_vmci
crc32c_intel vmw_pvscsi
[Mon Dec 24 16:08:04 2018] ---[ end trace a31b84c2a1d265ac ]---
[Mon Dec 24 16:08:04 2018] RIP: 0010:rb_erase+0x216/0x370
[Mon Dec 24 16:08:04 2018] Code: e9 6b fe ff ff 4d 89 48 10 e9 91 fe ff
ff c3 48 89 06 48 89 d0 48 8b 52 10 e9 b1 fe ff ff 48 8b 07 48 89 c1 48
83 e1 fc 74 53 <48> 3b 79 10 0f 84 94 00 00 00 4c 89 41 08 4d 85 c0 75
4c a8 01 0f
[Mon Dec 24 16:08:04 2018] RSP: 0018:ffffa8a840ac7d28 EFLAGS: 00010286
[Mon Dec 24 16:08:04 2018] RAX: 89a7851347204a0d RBX: ffff940e233086c0
RCX: 89a7851347204a0c
[Mon Dec 24 16:08:04 2018] RDX: 0000000000000000 RSI: ffff940e22502730
RDI: ffff940e233086c0
[Mon Dec 24 16:08:04 2018] RBP: ffff940e25942ec8 R08: 0000000000000000
R09: ffffffffc01533de
[Mon Dec 24 16:08:04 2018] R10: ffff940e24aac268 R11: 00000000000003c0
R12: ffff940e22502730
[Mon Dec 24 16:08:04 2018] R13: ffff940e22502808 R14: ffff940e22502000
R15: ffff940e233086e0
[Mon Dec 24 16:08:04 2018] FS:  00007f29173e7740(0000)
GS:ffff940e25e00000(0000) knlGS:0000000000000000
[Mon Dec 24 16:08:04 2018] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Mon Dec 24 16:08:04 2018] CR2: 000055f6ca2fab48 CR3: 000000002242e003
CR4: 00000000001606f0

Am 24.12.18 um 14:07 schrieb Reindl Harald:
> https://bugzilla.redhat.com/show_bug.cgi?id=1659706
> 
> i triggered something similar as
> https://www.spinics.net/lists/netdev/msg533254.html recently with
> 4.19.12-200.fc28.x86_64 by just call my "iptables.sh" which clears and
> sets up all sort of rules, chains and ipset - was a one-time event by
> call "iptables -t filter -P INPUT DROP" but i guess that should not
> happen and may point out a general problem explaining the random
> instability of the whole 4.19.x series
> 
> IPTABLES="/usr/sbin/iptables"
> IPTABLES_FLT="$IPTABLES -t filter"
> 
> /scripts/iptables.sh: line 617:  7874 Segmentation fault
> $IPTABLES_FLT -P INPUT DROP
> 
> [root@firewall:~]$ dmesg -c
> [Mon Dec 24 13:49:01 2018] general protection fault: 0000 [#1] SMP PTI
> [Mon Dec 24 13:49:01 2018] CPU: 0 PID: 7874 Comm: iptables Not tainted
> 4.19.12-200.fc28.x86_64 #1
> [Mon Dec 24 13:49:01 2018] Hardware name: VMware, Inc. VMware Virtual
> Platform/440BX Desktop Reference Platform, BIOS 6.00 07/03/2018
> [Mon Dec 24 13:49:01 2018] RIP: 0010:rb_erase+0x216/0x370
> [Mon Dec 24 13:49:01 2018] Code: e9 6b fe ff ff 4d 89 48 10 e9 91 fe ff
> ff c3 48 89 06 48 89 d0 48 8b 52 10 e9 b1 fe ff ff 48 8b 07 48 89 c1 48
> 83 e1 fc 74 53 <48> 3b 79 10 0f 84 94 00 00 00 4c 89 41 08 4d 85 c0 75
> 4c a8 01 0f
> [Mon Dec 24 13:49:01 2018] RSP: 0018:ffffb63fc2263d28 EFLAGS: 00010286
> [Mon Dec 24 13:49:01 2018] RAX: ffd7d18a01ee7a26 RBX: ffff9651a1b1c960
> RCX: ffd7d18a01ee7a24
> [Mon Dec 24 13:49:01 2018] RDX: 0000000000000000 RSI: ffff96519890d3e8
> RDI: ffff9651a1b1c960
> [Mon Dec 24 13:49:01 2018] RBP: ffff9651a5942c08 R08: 0000000000000000
> R09: ffffffffc02a23de
> [Mon Dec 24 13:49:01 2018] R10: ffff96519890b000 R11: 0000000000000000
> R12: ffff96519890d3e8
> [Mon Dec 24 13:49:01 2018] R13: ffff96519890d808 R14: ffff96519890d000
> R15: ffff9651a1b1c980
> [Mon Dec 24 13:49:01 2018] FS:  00007f76a1b53740(0000)
> GS:ffff9651a5e00000(0000) knlGS:0000000000000000
> [Mon Dec 24 13:49:01 2018] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [Mon Dec 24 13:49:01 2018] CR2: 0000564a46078000 CR3: 000000001951e005
> CR4: 00000000001606f0
> [Mon Dec 24 13:49:01 2018] Call Trace:
> [Mon Dec 24 13:49:01 2018]  nf_conncount_destroy+0x58/0xc0 [nf_conncount]
> [Mon Dec 24 13:49:01 2018]  cleanup_match+0x45/0x70
> [Mon Dec 24 13:49:01 2018]  cleanup_entry+0x3e/0xc0
> [Mon Dec 24 13:49:01 2018]  __do_replace+0x1ca/0x230
> [Mon Dec 24 13:49:01 2018]  do_ipt_set_ctl+0x146/0x1a2
> [Mon Dec 24 13:49:01 2018]  nf_setsockopt+0x44/0x70
> [Mon Dec 24 13:49:01 2018]  __sys_setsockopt+0x82/0xe0
> [Mon Dec 24 13:49:01 2018]  __x64_sys_setsockopt+0x20/0x30
> [Mon Dec 24 13:49:01 2018]  do_syscall_64+0x5b/0x160
> [Mon Dec 24 13:49:01 2018]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
> [Mon Dec 24 13:49:01 2018] RIP: 0033:0x7f76a0a7e4ea
> [Mon Dec 24 13:49:01 2018] Code: ff ff ff c3 48 8b 15 b5 d9 2b 00 f7 d8
> 64 89 02 48 c7 c0 ff ff ff ff eb b1 0f 1f 80 00 00 00 00 49 89 ca b8 36
> 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 86 d9 2b 00 f7 d8
> 64 89 01 48
> [Mon Dec 24 13:49:01 2018] RSP: 002b:00007ffc72e63a78 EFLAGS: 00000202
> ORIG_RAX: 0000000000000036
> [Mon Dec 24 13:49:01 2018] RAX: ffffffffffffffda RBX: 0000564a46032268
> RCX: 00007f76a0a7e4ea
> [Mon Dec 24 13:49:01 2018] RDX: 0000000000000040 RSI: 0000000000000000
> RDI: 0000000000000004
> [Mon Dec 24 13:49:01 2018] RBP: 0000564a46060060 R08: 0000000000015f80
> R09: 0000000000000000
> [Mon Dec 24 13:49:01 2018] R10: 0000564a46060060 R11: 0000000000000202
> R12: 0000564a460600c0
> [Mon Dec 24 13:49:01 2018] R13: 0000564a46032268 R14: 0000000000015f20
> R15: 0000564a46032260
> [Mon Dec 24 13:49:01 2018] Modules linked in: bridge stp llc nf_nat_ftp
> nf_log_ipv4 nf_log_common xt_LOG xt_limit xt_connlimit nf_conncount
> xt_recent nf_conntrack_ftp xt_CT xt_multiport xt_set iptable_raw xt_nat
> xt_NETMAP xt_iprange iptable_nat nf_nat_ipv4 nf_nat xt_conntrack
> nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c iptable_mangle
> ip_set_bitmap_port ip_set_hash_net ip_set nfnetlink crct10dif_pclmul
> crc32_pclmul ghash_clmulni_intel vmw_balloon vmxnet3 vmw_vmci
> crc32c_intel vmw_pvscsi
> [Mon Dec 24 13:49:01 2018] ---[ end trace b66858b9c9a97ef2 ]---
> [Mon Dec 24 13:49:01 2018] RIP: 0010:rb_erase+0x216/0x370
> [Mon Dec 24 13:49:01 2018] Code: e9 6b fe ff ff 4d 89 48 10 e9 91 fe ff
> ff c3 48 89 06 48 89 d0 48 8b 52 10 e9 b1 fe ff ff 48 8b 07 48 89 c1 48
> 83 e1 fc 74 53 <48> 3b 79 10 0f 84 94 00 00 00 4c 89 41 08 4d 85 c0 75
> 4c a8 01 0f
> [Mon Dec 24 13:49:01 2018] RSP: 0018:ffffb63fc2263d28 EFLAGS: 00010286
> [Mon Dec 24 13:49:01 2018] RAX: ffd7d18a01ee7a26 RBX: ffff9651a1b1c960
> RCX: ffd7d18a01ee7a24
> [Mon Dec 24 13:49:01 2018] RDX: 0000000000000000 RSI: ffff96519890d3e8
> RDI: ffff9651a1b1c960
> [Mon Dec 24 13:49:01 2018] RBP: ffff9651a5942c08 R08: 0000000000000000
> R09: ffffffffc02a23de
> [Mon Dec 24 13:49:01 2018] R10: ffff96519890b000 R11: 0000000000000000
> R12: ffff96519890d3e8
> [Mon Dec 24 13:49:01 2018] R13: ffff96519890d808 R14: ffff96519890d000
> R15: ffff9651a1b1c980
> [Mon Dec 24 13:49:01 2018] FS:  00007f76a1b53740(0000)
> GS:ffff9651a5e00000(0000) knlGS:0000000000000000
> [Mon Dec 24 13:49:01 2018] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [Mon Dec 24 13:49:01 2018] CR2: 0000564a46078000 CR3: 000000001951e005
> CR4: 00000000001606f0
> 
> Am 15.12.18 um 04:04 schrieb Reindl Harald:
>> am i really the only one where 4.19.x up to 4.19.9 randomly crashes?
>>
>> on my homeserver it takes some hours, a ton of virtual machines on ESXi
>> 6.5 are running stable all the time but on a NAT-Firewall guest it
>> survives just a few seconds until "kernel panic - Fatal exception in
>> interrupt" and a production webserver shortly after deploy 4.19.9 hat
>> the same yet while i though after running for days on 4.19.8/4.19.9 the
>> problem is now gone
>>
>> back to 4.18.20-100.fc27.x86_64 which had 3 weeks uptime on the same machine
>>
>> that's the first time any Fedor akernel is that unstable for years, in
>> 2014 or so there was a series which crashed at raid-check on a RAID10
>> regulary but since then until a few weeks ago every single build rock stable
>>
>> god, hopefully 4.20.x becomes stable again and rebased ASAP :-(