On Wed, Jul 16, 2014 at 09:32:51PM +1200, Nikola Pajkovsky wrote:
Vitezslav Samel vitezslav@samel.cz writes:
On Sun, Jul 13, 2014 at 06:02:19PM +1200, Nikola Pajkovsky wrote:
Vitezslav Samel vitezslav@samel.cz writes:
Vitezslav Samel (3): introduce packet capturing abstraction capt.c: add capturing using recvmmsg() capt.c: add capturing using mmap()ed PACKET_RX_RING memory
why do we using now recvmsg, recvmmsg and mmap? Which one is faster? Please elaborate deeply, because you don't have useful commit message.
It's all about speed:
- in recvmsg() case there are 2 syscalls per packet (poll() and then recvmsg() gives us one packet);
- in recvmmsg() case there are 2 syscalls (poll() and then recvmmsg() can give us more packets if available);
- in mmaped case there is syscall only in case there's no packet (poll())
On my workloads going from recvmsg() to mmap-style receiver lowers number of dropped packets 100x (from ten thousands to hundreds). And this is still with one thread.
The packet capturing abstraction was chosen to have modular packet receiving techniques: in our case recvmsg(), recvmmsg() and mmap. recvmsg() is available (almost) always, recvmmsg() is available only in linux-2.6.34+ and glibc-2.12+ and mmap-style receiver can be turned off in the kernel. The capturing interface tries mmap-style first, then tries recvmmsg() (if configured in) and recvmsg() is the slowest fallback.
To be honest, it took me awhile to find out, what the patches all about. Now, I have better picture, that you want to implement *zero copy* for rx.
linux/Documentation/networking/packet_mmap.txt
Since commit 889b8f964f2f ("packet: Kill CONFIG_PACKET_MMAP."), kills CONFIG_PACKET_MMAP and have enabled struct packet_ring_buffer rx_ring and tx_ring by default, the recvmmsg() becames not that interesting. I'm not
Didn't know about that.
saying, that we should not implement it, but I would rather go with mmap as default and recvmsg() as fallback. I haven't check if RHEL6 has CONFIG_PACKET_MMAP enabled, but it would not surprise me, if it has.
Since we are doing because of speeding things up, we should avoid *trying* like in recvmsg() and then do fallback to continue like in
[PATCH 2/3] capt.c: add capturing using recvmmsg()
and rather *doing* it. The trying things will became huge bottleneck and waste of time.
Trying is done only at packet capturing initialization; then only initialized capturing function is done without any trying in the fast path.
I'm still reading your code over and over and over. It's making more and more sense ;).
One way of implementing it as via weak functions, where weak functions are recv*() and strong are mmaped/whatever (choose during build).
I don't think weak functions are the best solution for us. The choice is done in compile time, but I want to make iptraf-ng versatile and to make the choice at run time.
Another option is, that we will build all of them (recvmsg(),recvmmsg(), mmap(), ...) and set one as default, which one can be override via cmd like --recv recvmsg/recvmmsg/mmap/... (or enable/disable during build via [NO_]MMAP=YesPlease)
Could be. The default one would be mmap and can be overriden to something else. But ...
Or have it like in linux kernel. Have module_init/module_exit and pkt_ops struct, which will hold pointer functions like you have, some config file with
CONFIG_MMAP=y # for enabled # CONFIG_MMAP not set
and build it according to config.
I have been wondering how modules work in linux kernel and who the heck they can call static functions from modules via module_init/module_exit.
So what do you think?
... I think my approach is best: when initializing try the best one, then try others when the best one isn't available/buggy/... with recvmsg() as sane fallback/default. Then in hot path just use what was initialized.
What could be added is override from command line. Should I add it?
Cheers,
Vita