From: Ondrej Lichtner olichtne@redhat.com
When binding ping to an IP address the kernel can choose a different source nic if it has an IP address from the same network configured, In our case all pings were leaving from the first (*_nic1) interface. The same then happens on the receiver side - the replies are sent from the default nic for that network (which is the first one again). This resulted in the switch not getting the full information about mac addresses that will later be used for the dpdk test.
Binding to the nic directly partially fixes this issue - the ICMP request messages will leave from the correct nic, the replies will still go out of the default nic for the specific network. But the switch will get all the info it needs from the request messages.
Signed-off-by: Ondrej Lichtner olichtne@redhat.com --- recipes/regression_tests/phase3/ovs-dpdk-pvp.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/recipes/regression_tests/phase3/ovs-dpdk-pvp.py b/recipes/regression_tests/phase3/ovs-dpdk-pvp.py index 5a5854e..fd58d0f 100644 --- a/recipes/regression_tests/phase3/ovs-dpdk-pvp.py +++ b/recipes/regression_tests/phase3/ovs-dpdk-pvp.py @@ -149,17 +149,17 @@ ctl.wait(5) ping_opts = {"count": 100, "interval": 0.1, "limit_rate": 20}
pings = [] -pings.append(ping((h1, h1_nic1, 0, {"scope": 0}), +pings.append(ping((h1, h1_nic1), (h2, h2_nic1, 0, {"scope": 0}), options=ping_opts, bg=True)) -pings.append(ping((h1, h1_nic2, 0, {"scope": 0}), +pings.append(ping((h1, h1_nic2), (h2, h2_nic2, 0, {"scope": 0}), options=ping_opts, bg=True))
-pings.append(ping((h2, h2_nic1, 0, {"scope": 0}), +pings.append(ping((h2, h2_nic1), (h1, h1_nic1, 0, {"scope": 0}), options=ping_opts, bg=True)) -pings.append(ping((h2, h2_nic2, 0, {"scope": 0}), +pings.append(ping((h2, h2_nic2), (h1, h1_nic2, 0, {"scope": 0}), options=ping_opts, bg=True))
From: Ondrej Lichtner olichtne@redhat.com
This cpu will end up being used for the management of the vm, this can improve the stability of the results. Setting the default value of the emulator pin alias to 9 (as in core 9) as this is the next free core in our usual setup.
Also setting the default value for guest cores to 5,6,7,8 for the same reason.
Signed-off-by: Ondrej Lichtner olichtne@redhat.com --- recipes/regression_tests/phase3/ovs-dpdk-pvp.py | 3 +++ recipes/regression_tests/phase3/ovs-dpdk-pvp.xml | 3 ++- 2 files changed, 5 insertions(+), 1 deletion(-)
diff --git a/recipes/regression_tests/phase3/ovs-dpdk-pvp.py b/recipes/regression_tests/phase3/ovs-dpdk-pvp.py index fd58d0f..10ac031 100644 --- a/recipes/regression_tests/phase3/ovs-dpdk-pvp.py +++ b/recipes/regression_tests/phase3/ovs-dpdk-pvp.py @@ -119,6 +119,7 @@ guest_hostname = ctl.get_alias("guest_hostname") guest_username = ctl.get_alias("guest_username") guest_password = ctl.get_alias("guest_password") guest_cpus = ctl.get_alias("guest_cpus") +guest_emulatorpin = ctl.get_alias("guest_emulatorpin") trex_dir = ctl.get_alias("trex_dir") pkt_size = int(ctl.get_alias("pkt_size")) test_duration = int(ctl.get_alias("test_duration")) @@ -277,6 +278,8 @@ cputune = ET.SubElement(guest_xml, "cputune") for i, cpu_id in enumerate(guest_cpus.split(',')): ET.SubElement(cputune, "vcpupin", vcpu=str(i), cpuset=str(cpu_id))
+ET.SubElement(cputune, "emulatorpin", cpuset=guest_emulatorpin) + updated_guest_xml = NamedTemporaryFile("w+b", delete=False) updated_guest_xml.write(ET.tostring(guest_xml)) updated_guest_xml.close() diff --git a/recipes/regression_tests/phase3/ovs-dpdk-pvp.xml b/recipes/regression_tests/phase3/ovs-dpdk-pvp.xml index e9edf74..d534b94 100644 --- a/recipes/regression_tests/phase3/ovs-dpdk-pvp.xml +++ b/recipes/regression_tests/phase3/ovs-dpdk-pvp.xml @@ -14,7 +14,8 @@ <alias name="guest_hostname" value="" /> <alias name="guest_username" value="" /> <alias name="guest_password" value="" /> - <alias name="guest_cpus" value="" /> + <alias name="guest_cpus" value="5,6,7,8" /> + <alias name="guest_emulatorpin" value="9"/> <alias name="pkt_size" value="64" /> <alias name="test_duration" value="60" /> <alias name="test_runs" value="5" />
From: Ondrej Lichtner olichtne@redhat.com
The first '-c <value>' argument of testpmd specifies the coremask for the whole testpmd application, this includes all pmd operations (2 dpdk nics in our case) and the management thread. For optimal performance we should therefore provide 3 cores. This will now be supplied by the guest_testpmd_cores alias (default 0x7 - cores 0,1,2).
The second (after '-- ') '--coremask' argument specifies the coremask for the pmd threads (2 in our case) where we'll specify whic 2 of the 3 cores will be used for the dpdk nics. This will be supplied by the guest_dpdk_cores alias (default 0x6 - cores 1,2).
Signed-off-by: Ondrej Lichtner olichtne@redhat.com --- recipes/regression_tests/phase3/ovs-dpdk-pvp.py | 5 ++++- recipes/regression_tests/phase3/ovs-dpdk-pvp.xml | 1 + 2 files changed, 5 insertions(+), 1 deletion(-)
diff --git a/recipes/regression_tests/phase3/ovs-dpdk-pvp.py b/recipes/regression_tests/phase3/ovs-dpdk-pvp.py index 10ac031..1586dc2 100644 --- a/recipes/regression_tests/phase3/ovs-dpdk-pvp.py +++ b/recipes/regression_tests/phase3/ovs-dpdk-pvp.py @@ -110,6 +110,7 @@ official_result = bool_it(ctl.get_alias("official_result")) pr_user_comment = ctl.get_alias("perfrepo_comment") host1_dpdk_cores = ctl.get_alias("host1_dpdk_cores") host2_dpdk_cores = ctl.get_alias("host2_dpdk_cores") +guest_testpmd_cores = ctl.get_alias("guest_testpmd_cores") guest_dpdk_cores = ctl.get_alias("guest_dpdk_cores") nr_hugepages = int(ctl.get_alias("nr_hugepages")) socket_mem = int(ctl.get_alias("socket_mem")) @@ -340,9 +341,11 @@ testpmd_shell = guest.get_transport().open_session() testpmd_cmd = ("testpmd -c {coremask} " "-w {pci1} -w {pci2} " "-n 4 --socket-mem 1024,0 -- " + "--coremask {pmd_coremask} " "-i --eth-peer=0,{hw1} --eth-peer=1,{hw2} " "--forward-mode=mac").format( - coremask=guest_dpdk_cores, + coremask=guest_testpmd_cores, + pmd_coremask=guest_dpdk_cores, pci1=g_nic1_pci, pci2=g_nic2_pci, hw1=h1_nic1.get_hwaddr(), hw2=h1_nic2.get_hwaddr())
diff --git a/recipes/regression_tests/phase3/ovs-dpdk-pvp.xml b/recipes/regression_tests/phase3/ovs-dpdk-pvp.xml index d534b94..3c2326a 100644 --- a/recipes/regression_tests/phase3/ovs-dpdk-pvp.xml +++ b/recipes/regression_tests/phase3/ovs-dpdk-pvp.xml @@ -5,6 +5,7 @@ <alias name="official_result" value="no" /> <alias name="host1_dpdk_cores" value="2,3,4" /> <alias name="host2_dpdk_cores" value="0x1E" /> + <alias name="guest_testpmd_cores" value="0x7" /> <alias name="guest_dpdk_cores" value="0x6" /> <alias name="nr_hugepages" value="2048" /> <alias name="socket_mem" value="2048" />
From: Ondrej Lichtner olichtne@redhat.com
For the guest to properly use hugepages it is necessary to configure memory backing using them.
It's also important to ensure that the host machine has enough hugepages to cover all the ram of the guest. In our setup we use a guest with 16G ram so we need to increase the number of 2M pages to at least 8000. I'm setting it to 13000 just to be sure.
Signed-off-by: Ondrej Lichtner olichtne@redhat.com --- recipes/regression_tests/phase3/ovs-dpdk-pvp.py | 4 ++++ recipes/regression_tests/phase3/ovs-dpdk-pvp.xml | 2 +- 2 files changed, 5 insertions(+), 1 deletion(-)
diff --git a/recipes/regression_tests/phase3/ovs-dpdk-pvp.py b/recipes/regression_tests/phase3/ovs-dpdk-pvp.py index 1586dc2..2bc3dff 100644 --- a/recipes/regression_tests/phase3/ovs-dpdk-pvp.py +++ b/recipes/regression_tests/phase3/ovs-dpdk-pvp.py @@ -275,6 +275,10 @@ cpu = guest_xml.find("cpu") numa = ET.SubElement(cpu, 'numa') ET.SubElement(numa, 'cell', id='0', cpus='0', memory=guest_mem_amount, unit='KiB', memAccess='shared')
+memoryBacking = ET.SubElement(guest_xml, "memoryBacking") +hugepages = ET.SubElement(memoryBacking, "hugepages") +ET.SubElement(hugepages, "page", size="2", unit="M", nodeset="0") + cputune = ET.SubElement(guest_xml, "cputune") for i, cpu_id in enumerate(guest_cpus.split(',')): ET.SubElement(cputune, "vcpupin", vcpu=str(i), cpuset=str(cpu_id)) diff --git a/recipes/regression_tests/phase3/ovs-dpdk-pvp.xml b/recipes/regression_tests/phase3/ovs-dpdk-pvp.xml index 3c2326a..8f4af69 100644 --- a/recipes/regression_tests/phase3/ovs-dpdk-pvp.xml +++ b/recipes/regression_tests/phase3/ovs-dpdk-pvp.xml @@ -7,7 +7,7 @@ <alias name="host2_dpdk_cores" value="0x1E" /> <alias name="guest_testpmd_cores" value="0x7" /> <alias name="guest_dpdk_cores" value="0x6" /> - <alias name="nr_hugepages" value="2048" /> + <alias name="nr_hugepages" value="13000" /> <alias name="socket_mem" value="2048" /> <alias name="trex_dir" value="" /> <alias name="guest_mem_amount" value="16777216" />
From: Ondrej Lichtner olichtne@redhat.com
We should also be configuring the dpdk-lcore-mask ovs option to ensure that logic operations don't get scheduled on the pmd cores. Setting the default value of the alias to 0x400 (core 10) as this is the next free core in our usual setup.
Signed-off-by: Ondrej Lichtner olichtne@redhat.com --- recipes/regression_tests/phase3/ovs-dpdk-pvp.py | 2 ++ recipes/regression_tests/phase3/ovs-dpdk-pvp.xml | 1 + 2 files changed, 3 insertions(+)
diff --git a/recipes/regression_tests/phase3/ovs-dpdk-pvp.py b/recipes/regression_tests/phase3/ovs-dpdk-pvp.py index 2bc3dff..6b16270 100644 --- a/recipes/regression_tests/phase3/ovs-dpdk-pvp.py +++ b/recipes/regression_tests/phase3/ovs-dpdk-pvp.py @@ -109,6 +109,7 @@ else: official_result = bool_it(ctl.get_alias("official_result")) pr_user_comment = ctl.get_alias("perfrepo_comment") host1_dpdk_cores = ctl.get_alias("host1_dpdk_cores") +host2_dpdk_lcores = ctl.get_alias("host2_dpdk_lcores") host2_dpdk_cores = ctl.get_alias("host2_dpdk_cores") guest_testpmd_cores = ctl.get_alias("guest_testpmd_cores") guest_dpdk_cores = ctl.get_alias("guest_dpdk_cores") @@ -193,6 +194,7 @@ h2.config("/sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages", nr_hugepages h2.enable_service("openvswitch") h2.run("ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true") h2.run("ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem=%d" % socket_mem) +h2.run("ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-lcore-mask=%s" % host2_dpdk_lcores) h2.run("ovs-vsctl --no-wait set Open_vSwitch . other_config:pmd-cpu-mask=%s" % host2_dpdk_cores) h2.restart_service("openvswitch")
diff --git a/recipes/regression_tests/phase3/ovs-dpdk-pvp.xml b/recipes/regression_tests/phase3/ovs-dpdk-pvp.xml index 8f4af69..4968f7d 100644 --- a/recipes/regression_tests/phase3/ovs-dpdk-pvp.xml +++ b/recipes/regression_tests/phase3/ovs-dpdk-pvp.xml @@ -4,6 +4,7 @@ <alias name="driver" value=""/> <alias name="official_result" value="no" /> <alias name="host1_dpdk_cores" value="2,3,4" /> + <alias name="host2_dpdk_lcores" value="0x400" /> <alias name="host2_dpdk_cores" value="0x1E" /> <alias name="guest_testpmd_cores" value="0x7" /> <alias name="guest_dpdk_cores" value="0x6" />
From: Ondrej Lichtner olichtne@redhat.com
Instead of waiting an arbitrary 60 seconds for the guest to start then trying a single ssh connection we should be repeatedly trying to ssh connect until it succeeds and set an upper limit.
For several past runs the guest startup time was much more unpredictable (maybe due to more isolated cores on the host?) so this upper time limit should be much larger. The implemented loop should try 60 times to connect with 5s waits in between tries. In the worst case scenario this should be enough of an upper bound.
Signed-off-by: Ondrej Lichtner olichtne@redhat.com --- .../regression_tests/phase3/ovs-dpdk-pvp.py | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-)
diff --git a/recipes/regression_tests/phase3/ovs-dpdk-pvp.py b/recipes/regression_tests/phase3/ovs-dpdk-pvp.py index 6b16270..c46999f 100644 --- a/recipes/regression_tests/phase3/ovs-dpdk-pvp.py +++ b/recipes/regression_tests/phase3/ovs-dpdk-pvp.py @@ -4,6 +4,7 @@ import paramiko import logging from tempfile import NamedTemporaryFile from lnst.Common.Utils import bool_it, std_deviation, dict_to_dot, Noop +from lnst.Common.Logs import log_exc_traceback from lnst.Controller.Task import ctl from lnst.Controller.PerfRepoUtils import perfrepo_baseline_to_dict from lnst.RecipeCommon.ModuleWrap import ping @@ -298,7 +299,19 @@ h2.run("virsh start %s" % guest_virtname) #============================================ # Host2 wait for guest start #============================================ -ctl.wait(60) + +guest = paramiko.SSHClient() +guest.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +tries = 60 +while tries: + logging.info("Connecting to guest, %d tries remaining" % tries) + tries -= 1 + try: + guest.connect(guest_hostname, username=guest_username, password=guest_password) + break + except paramiko.ssh_exception.NoValidConnectionsError as e: + logging.debug(str(e)) + ctl.wait(5)
#============================================ # Host2 add openvswitch flows between DPDK NICs and Guest NICs @@ -314,10 +327,6 @@ h2.run("ovs-ofctl add-flow br0 in_port=22,action=12") # Guest configure DPDK for vhostuser nics #============================================
-guest = paramiko.SSHClient() -guest.set_missing_host_key_policy(paramiko.AutoAddPolicy()) -guest.connect(guest_hostname, username=guest_username, password=guest_password) - run_ssh_command_on_guest("service irqbalance stop", guest, h2, guest_virtname) run_ssh_command_on_guest("MASK=1; for i in `ls -d /proc/irq/[0-9]*` ; do echo $MASK > ${i}/smp_affinity ; done", guest, h2, guest_virtname)
Fri, Sep 14, 2018 at 11:39:45AM CEST, olichtne@redhat.com wrote:
From: Ondrej Lichtner olichtne@redhat.com
When binding ping to an IP address the kernel can choose a different source nic if it has an IP address from the same network configured, In our case all pings were leaving from the first (*_nic1) interface. The same then happens on the receiver side - the replies are sent from the default nic for that network (which is the first one again). This resulted in the switch not getting the full information about mac addresses that will later be used for the dpdk test.
Binding to the nic directly partially fixes this issue - the ICMP request messages will leave from the correct nic, the replies will still go out of the default nic for the specific network. But the switch will get all the info it needs from the request messages.
Whole series looks ok.
Acked-by: Jan Tluka jtluka@redhat.com
On Fri, Sep 14, 2018 at 02:37:13PM +0200, Jan Tluka wrote:
Fri, Sep 14, 2018 at 11:39:45AM CEST, olichtne@redhat.com wrote:
From: Ondrej Lichtner olichtne@redhat.com
When binding ping to an IP address the kernel can choose a different source nic if it has an IP address from the same network configured, In our case all pings were leaving from the first (*_nic1) interface. The same then happens on the receiver side - the replies are sent from the default nic for that network (which is the first one again). This resulted in the switch not getting the full information about mac addresses that will later be used for the dpdk test.
Binding to the nic directly partially fixes this issue - the ICMP request messages will leave from the correct nic, the replies will still go out of the default nic for the specific network. But the switch will get all the info it needs from the request messages.
Whole series looks ok.
Acked-by: Jan Tluka jtluka@redhat.com
thanks, pushed.
-Ondrej
lnst-developers@lists.fedorahosted.org
-
Jan Tluka -
olichtne＠redhat.com -
Ondrej Lichtner