NFLOG bugs and idiosyncrasies

In my current project I use NFLOG to capture network traffic. NFLOG queues network traffic to usermode programs following rules in iptables. This is quite powerful and performance is good.

The bad thing is that NFLOG documentation is very incomplete and, sometimes, clearly incorrect. And it has ugly bugs too.

I am doing most of my development on a Linux kernel 2.6.32, using Python 3 and CFFI. With this kernel version, every packet captured is directly transferred to usermode, independently. My programs works perfectly.

The project is being deployed on Raspberry PI, running kernel 3.10.28+ (current Raspbian). Compared with old 2.6.32, this version allows to queue several datagrams in-kernel to be transfered to usermode using a single syscall. You can configure the queue to push to usermode using a timeout and/or a packet count, whatever happens first. This is useful because Rasberry PI CPU is anemic and I need cycles for Python :).

But after sucessful intensive testing on kernel 2.6.32, my program explodes on the Raspberry PI, every single time.

If I get a single datagram per syscall, it works. But if I get a bundle of datagrams, nflog_handle_packet() returns -1. Always.

According to ancient documentation (I am unable to find any reference to nflog_handle_packet() in the official netfilter documentation, WTF!), a -1 means "internal error". After checking sourcecode and spending a few hours on it, I see that nflog_handle_packet() is ALWAYS failing with a -1 when you are processing a bundle made of more than a datagram.

Always.

Moreover, (ancient) documentation says:

0 on success, non-zero on error. Negative values are returned when internal errors occured. Otherwise the return value of the callback is returned (which must return zero on success).

That is false.

First, I am getting -1 at the end of all multi datagram bundles. The datagrams are processed just fine, the error is raised at the very end of the buffer. Looks like the library is not recognizing correctly the end of bundle mark.

Second, nflog_handle_packet() will return 0 if the callback returns any value >=0. And will return -1 if the callback returns any value <0.

So, if I am getting -1 everytime, how do I know when I am getting a real error?. What I am doing is to catch any exception inside the callback, export it outside, and check for it when nflog_handle_packet() returns -1 error code (that is, everytime I get a multi datagram bundle). Slow, ugly, hacky :-(.

How are other people managing this?. Checking tons of code around, the conclusion is simple: nobody does, nobody checks the return code. For example, check this official netfilter example code. Life sucks.

After spending a couple of days capturing packets at IP level, I was interested in getting access to the Ethernet framing too. NFLOG provides two functions for it: msg_packet_hwhdrlen() and msg_packet_hwhdr(). Regular Ethernet header is 14 bytes long. 18 bytes if using 802.1Q. That is, VLAN. But msg_packet_hwhdrlen() is returning 26. Yes, 26. First 14/18 bytes are Ethernet headers, and the rest are real data. If you create a PCAP record doing "HARDWARE HEADER + PAYLOAD", you get 12/8 bytes of duplicate data and your capture will be corrupt. You MUST ignore msg_packet_hwhdrlen() return value and parse msg_packet_hwhdr() returned array yourself by hand. Life sucks again.

It is funny that you have functions like nflog_get_packet_hw() that only provides the source ethernet address, but not the destination. It could be different than your NIC ethernet address if you are in promiscuous mode, or getting a broadcast/multicast frame.

Of course, nothing of this is documented.

I can’t believe I wasted an entire day on this - every programmer in existence. (Link)

If only it was a single day... Have I already said that life sucks?