Massive WireGuard performance regression on Fedora 37 vs Ubuntu 18.04

[James Wynn]

I noticed a massive performance regression for WireGuard in Ubuntu 20.04 & 22.04, but
it also affects Fedora. I don't know since which version.

Should I report this as a bug or did I mess something up? 

I have fully reproducible steps to demonstrate this issue on a vanilla DigitalOcean
droplet, minimal WireGuard configuration and no firewall rules. I've also seen this
issue on other hosting providers.

Testing with `iperf3 -c XXX -P 5`:

- Unencrypted traffic on DigitalOcean's VPC = ~2Gbps
- WireGuard Ubuntu 18.04 = ~1.3Gbps
- WireGuard Fedora 37 = ~400Mbps

htop reported only 20-30% load on the vCPU core so it isn't CPU-bound. After doing
these tests, I did them all again on a different day to rule out temporary network
congestion.

Steps to reproduce below. Repeat with each OS version.

0. Create a DigitalOcean account.
1. Create two $6 droplets (eg, LON1 region) with Regular CPU & 1GB RAM each, called
test01 & test02.
2. `dnf update -y && reboot`
3. `dnf install -y wireguard-tools iperf3`

4. On test01, create `/etc/wireguard/test.conf` with these contents. Replace `YYY` with
the IP address of the eth1 interface (VPC) on test02.

--------------------
[Interface]
PrivateKey = wOEa8/RS2v065wgYGQn5k7FqOXuZJ9aC/6NDW569c3g=
Address = 192.168.200.10/24
ListenPort = 51820
SaveConfig = false

[Peer]
PublicKey = wdXOzBptLD/QMZjhG475GErrz95Vpj4S7JPEwzcDMV8=
PresharedKey = j5Oeyhu/qDag2LunpVlFqKycp/9CH+Izjza5aq2cYss=
Endpoint = YYY:51820
AllowedIPs = 192.168.200.20/32
--------------------

5. On test02, create `/etc/wireguard/test.conf` with these contents. Replace `XXX` with
the IP address of the eth1 interface (VPC) on test01.

--------------------
[Interface]
PrivateKey = kCJ/4rVDTy86HxP9N5wUmgMF1Esqjc051jQPGhrQIGw=
Address = 192.168.200.20/24
ListenPort = 51820
SaveConfig = false

[Peer]
PublicKey = s/GtXkHOtPsqcNDy0BSRoMuxXYb4hK18dsQdkZk20yQ=
PresharedKey = j5Oeyhu/qDag2LunpVlFqKycp/9CH+Izjza5aq2cYss=
Endpoint = XXX:51820
AllowedIPs = 192.168.200.10/32
--------------------

6. On both droplets, run `systemctl start wg-quick@test`
7. On test01, run `iperf3 -s -B XXX`.
8. On test02, run `iperf3 -c XXX -P 5 -t 30` and observe ~2Gbps.
9. On test01, run `iperf3 -s -B 192.168.200.10`
10. On test02, run `iperf3 -c 192.168.200.10 -P 5 -t 30` and observe ~400Mbps.

In steps 7 and 8, replace XXX with the IP address of the eth1 interface on test01.

[Barry Scott]

>  On 30 Jan 2023, at 10:23, James Wynn via users
> <users(a)lists.fedoraproject.org&gt; wrote:
>  
>  I noticed a massive performance regression for WireGuard in Ubuntu 20.04 & 22.04, but
> it also affects Fedora. I don't know since which version.
>  
>  Should I report this as a bug or did I mess something up? 
>  
>  I have fully reproducible steps to demonstrate this issue on a vanilla DigitalOcean
> droplet, minimal WireGuard configuration and no firewall rules. I've also seen this
> issue on other hosting providers.
>  
>  Testing with `iperf3 -c XXX -P 5`:
>  
>  - Unencrypted traffic on DigitalOcean's VPC = ~2Gbps
>  - WireGuard Ubuntu 18.04 = ~1.3Gbps
>  - WireGuard Fedora 37 = ~400Mbps
>  
>  htop reported only 20-30% load on the vCPU core so it isn't CPU-bound. After doing
> these tests, I did them all again on a different day to rule out temporary network
> congestion.
>  
>  Steps to reproduce below. Repeat with each OS version.
>  
>  0. Create a DigitalOcean account.
>  1. Create two $6 droplets (eg, LON1 region) with Regular CPU & 1GB RAM each, called
> test01 & test02.
>  2. `dnf update -y && reboot`
>  3. `dnf install -y wireguard-tools iperf3`
>  
>  4. On test01, create `/etc/wireguard/test.conf` with these contents. Replace `YYY` with
> the IP address of the eth1 interface (VPC) on test02.
>  
>  --------------------
>  [Interface]
>  PrivateKey = wOEa8/RS2v065wgYGQn5k7FqOXuZJ9aC/6NDW569c3g=
>  Address = 192.168.200.10/24
>  ListenPort = 51820
>  SaveConfig = false
>  
>  [Peer]
>  PublicKey = wdXOzBptLD/QMZjhG475GErrz95Vpj4S7JPEwzcDMV8=
>  PresharedKey = j5Oeyhu/qDag2LunpVlFqKycp/9CH+Izjza5aq2cYss=
>  Endpoint = YYY:51820
>  AllowedIPs = 192.168.200.20/32
>  --------------------
>  
>  5. On test02, create `/etc/wireguard/test.conf` with these contents. Replace `XXX` with
> the IP address of the eth1 interface (VPC) on test01.
>  
>  --------------------
>  [Interface]
>  PrivateKey = kCJ/4rVDTy86HxP9N5wUmgMF1Esqjc051jQPGhrQIGw=
>  Address = 192.168.200.20/24
>  ListenPort = 51820
>  SaveConfig = false
>  
>  [Peer]
>  PublicKey = s/GtXkHOtPsqcNDy0BSRoMuxXYb4hK18dsQdkZk20yQ=
>  PresharedKey = j5Oeyhu/qDag2LunpVlFqKycp/9CH+Izjza5aq2cYss=
>  Endpoint = XXX:51820
>  AllowedIPs = 192.168.200.10/32
>  --------------------
>  
>  6. On both droplets, run `systemctl start wg-quick@test`
>  7. On test01, run `iperf3 -s -B XXX`.
>  8. On test02, run `iperf3 -c XXX -P 5 -t 30` and observe ~2Gbps.
>  9. On test01, run `iperf3 -s -B 192.168.200.10`
>  10. On test02, run `iperf3 -c 192.168.200.10 -P 5 -t 30` and observe ~400Mbps.
>  
>  In steps 7 and 8, replace XXX with the IP address of the eth1 interface on test01.
You could raise this on the wireguard mailing list.
That is where the maintainers are and expert users.

Subscribe here https://lists.zx2c4.com/mailman/listinfo/wireguard

Barry


>  _______________________________________________
>  users mailing list -- users(a)lists.fedoraproject.org
>  To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue