使用lwIP的原始TCP API时的性能问题

Question

I use lwIP to add networking functionality to my system. On my platform i built a buffer that i want to send every time it is full. This can happen quite quickly. The system is directly connected to a switch in a private LAN. Initially the sending of data had a very large time gap in between of 2 seconds. Additionally the packets had a size of 720 bytes if my memory serves me correctly. The used buffer currently has capacity for about 20000 bytes and I might decide to increase this in the future. The network has 100 mbit speeds and I would like to come close to these speeds on my platform.

When searching for the cause of the slow speeds I ended up at lwIP's configuration. Prior to that I altered my sending mechanism. I use the raw lwIP API and at present I write the data as follows:

tcp_write(<pcb>, (const void*) data, <bytes>, TCP_WRITE_FLAG_COPY);
//<bytes> is at most tcp_sndbuf(<pcb>)

I know the copy flag creates a performance hit, but this is added because I don't want to overwrite data before it's actually sent. (and the flag isn't the main problem but something to polish once it's working properly) In a prior solution i omitted the flag and simply waited for all bytes to be ACK'd (after forcing data to be send after writing by calling tcp_output()) by using the callback function. (This might be worse performance wise and I dont think it's related)

I played a little with the settings in the of lwIP and that seemed to make some difference. I think the window size especially made a difference although I'm not quite sure. At the moment I increased the window size significantly and even though I get a burst of packets with about 2ms between them (instead of 2s!) this is followed with a long period of "nothing" and then a burst again. I want it to continuously send at the speed it should be capable of which should be 100 mbit at most but atleast 10 mbit is not weird to expect, right?

I loaded up wireshark to see what was happening.

192.168.1.26 is my desktop computer running windows. 192.168.1.192 is the embedded system that's using lwIP.

Initially I send a start request from the desktop to the lwIP system, letting the system know it should start sending the buffer each time it is full. In case it is relevant, this is the corresponding part of the trace:

5    2.007754    192.168.1.26    192.168.1.192    TCP    61    [TCP segment of a reassembled PDU]
6    2.008196    192.168.1.192    192.168.1.26    TCP    60    patrolview > afs3-fileserver [SYN] Seq=0 Win=65535 Len=0 MSS=1400
7    2.226238    192.168.1.192    192.168.1.26    TCP    60    afs3-fileserver > 50015 [ACK] Seq=1 Ack=8 Win=65528 Len=0
13    4.976858    192.168.1.192    192.168.1.26    TCP    60    patrolview > afs3-fileserver [ACK] Seq=1 Ack=1 Win=65535 Len=0
22    6.976572    192.168.1.192    192.168.1.26    TCP    60    [TCP segment of a reassembled PDU]
23    7.177903    192.168.1.26    192.168.1.192    TCP    54    50015 > afs3-fileserver [ACK] Seq=8 Ack=2 Win=64399 Len=0

I believe this is alright although i'm not certain. Anyway, after this the actual sending happens. The relevant trace looks as follows The start time is 207.992115 which should be considered the starting time. The difference between that and the 7.177903 is expected:

2578    207.992115    192.168.1.192    192.168.1.26    Gryphon    1422    - Invalid -
2581    208.194336    192.168.1.26    192.168.1.192    TCP    54    afs3-fileserver > patrolview [ACK] Seq=1 Ack=1369 Win=64400 Len=0
2582    208.195880    192.168.1.192    192.168.1.26    TCP    1422    [TCP segment of a reassembled PDU]
2583    208.197035    192.168.1.192    192.168.1.26    TCP    1422    [TCP segment of a reassembled PDU]
2584    208.197134    192.168.1.26    192.168.1.192    TCP    54    afs3-fileserver > patrolview [ACK] Seq=1 Ack=4105 Win=64400 Len=0
2585    208.198712    192.168.1.192    192.168.1.26    TCP    1422    [TCP segment of a reassembled PDU]
2586    208.199867    192.168.1.192    192.168.1.26    TCP    1422    [TCP segment of a reassembled PDU]
2587    208.199965    192.168.1.26    192.168.1.192    TCP    54    afs3-fileserver > patrolview [ACK] Seq=1 Ack=6841 Win=64400 Len=0
2588    208.200927    192.168.1.192    192.168.1.26    TCP    1314    [TCP segment of a reassembled PDU]
2590    208.397469    192.168.1.26    192.168.1.192    TCP    54    afs3-fileserver > patrolview [ACK] Seq=1 Ack=8101 Win=63140 Len=0

It seems that I currently send things quicker than the desktop is ACKing. The traffic after the trace above shows as black bars and looks like:

2591    208.399051    192.168.1.192    192.168.1.26    TCP    1422    [TCP Previous segment lost] [TCP segment of a reassembled PDU]
2592    208.399136    192.168.1.26    192.168.1.192    TCP    54    [TCP Dup ACK 2590#1] afs3-fileserver > patrolview [ACK] Seq=1 Ack=8101 Win=63140 Len=0
2593    208.400208    192.168.1.192    192.168.1.26    Gryphon    1422   
2594    208.400285    192.168.1.26    192.168.1.192    TCP    54    [TCP Dup ACK 2590#2] afs3-fileserver > patrolview [ACK] Seq=1 Ack=8101 Win=63140 Len=0
2595    208.401361    192.168.1.192    192.168.1.26    Gryphon    1422    - Invalid -
2596    208.401445    192.168.1.26    192.168.1.192    TCP    54    [TCP Dup ACK 2590#3] afs3-fileserver > patrolview [ACK] Seq=1 Ack=8101 Win=63140 Len=0
2597    208.402425    192.168.1.192    192.168.1.26    Gryphon    1314   
2598    208.402516    192.168.1.26    192.168.1.192    TCP    54    [TCP Dup ACK 2590#4] afs3-fileserver > patrolview [ACK] Seq=1 Ack=8101 Win=63140 Len=0
2599    208.403588    192.168.1.192    192.168.1.26    Gryphon    1422    [TCP Fast Retransmission] Command response
2600    208.403685    192.168.1.26    192.168.1.192    TCP    54    afs3-fileserver > patrolview [ACK] Seq=1 Ack=14833 Win=64400 Len=0
2605    209.992237    192.168.1.192    192.168.1.26    Gryphon    1422    - Invalid -
2607    210.200219    192.168.1.26    192.168.1.192    TCP    54    afs3-fileserver > patrolview [ACK] Seq=1 Ack=16201 Win=63032 Len=0
2608    210.201819    192.168.1.192    192.168.1.26    Gryphon    1422    [TCP Previous segment lost] - Invalid -
2609    210.201903    192.168.1.26    192.168.1.192    TCP    54    [TCP Dup ACK 2607#1] afs3-fileserver > patrolview [ACK] Seq=1 Ack=16201 Win=63032 Len=0
2609    210.201903    192.168.1.26    192.168.1.192    TCP    54    [TCP Dup ACK 2607#1] afs3-fileserver > patrolview [ACK] Seq=1 Ack=16201 Win=63032 Len=0
2611    210.203070    192.168.1.26    192.168.1.192    TCP    54    [TCP Dup ACK 2607#2] afs3-fileserver > patrolview [ACK] Seq=1 Ack=16201 Win=63032 Len=0
2955    345.001223    192.168.1.192    192.168.1.26    Gryphon    1422    [TCP Retransmission]

Now after this point there is a huge delay that I cannot explain. The next packets arrive at 345 seconds, this is a 135 second difference. (although in most cases it was a bit less, but still wayy too high) It starts as follows:

2955    345.001223    192.168.1.192    192.168.1.26    Gryphon    1422    [TCP Retransmission]
2958    345.001707    192.168.1.26    192.168.1.192    TCP    54    afs3-fileserver > patrolview [ACK] Seq=1 Ack=20305 Win=64400 Len=0
2959    345.003336    192.168.1.192    192.168.1.26    TCP    1422    [TCP segment of a reassembled PDU]
2960    345.004395    192.168.1.192    192.168.1.26    TCP    1314    [TCP segment of a reassembled PDU]
2961    345.004494    192.168.1.26    192.168.1.192    TCP    54    afs3-fileserver > patrolview [ACK] Seq=1 Ack=22933 Win=64400 Len=0

etc.

Later on a similar problems occur although the mentioned delay is shorter. My question is: How can I fix the problem of the slow sending from my platform and how should i configure my lwIP settings to expect decent/good results? I want to send the data at fast speeds. (my network is capable of 100Mbps, the closer the better) I think i currently messed up my settings entirely but I am not sure how to finetune them for my needs. Here are some (hopefully) relevant settings from my lwipopts.h

file:

#define MEM_SIZE                        65000
#define PBUF_POOL_SIZE                  1024
#define IP_FRAG_USES_STATIC_BUF         0
#define TCP_WND                         65535
#define TCP_MSS                         1400
#define TCP_SND_BUF                     65535
#define PBUF_POOL_BUFSIZE               LWIP_MEM_ALIGN_SIZE(1528)
#define LWIP_TCP_KEEPALIVE              1
#define LWIP_SO_RCVTIMEO                1

Answer 1

I ran into similar problems using beaglebone and following setting

#define MEM_SIZE                        (1024 * 1024) /* 1MiB */
#define MEMP_NUM_PBUF                   1024
#define MEMP_NUM_TCP_PCB                32
#define PBUF_POOL_SIZE                  1024
#define TCP_MSS                         1460
#define TCP_WND                         (4*TCP_MSS)
#define TCP_SND_BUF                     65535
#define TCP_OVERSIZE                    TCP_MSS
#define TCP_SND_QUEUELEN                512
#define MEMP_NUM_TCP_SEG                512

Using a tcp_sent polled function I basically checked how many bytes I have in the buffer and filled them in the polled function itself immediately with another samples. This was to check the throughout

I was quite surprised that wire shark shown bursts of packets during roughly few milliseconds and then 700ms of nothing .

Going deep into the stack I've found, that this happens exactly at the moment there are 65535 bytes sent (or roughly around this).

All that was solved by DISABLING MEMORY SANITY CHECK

Look in your lwipopts.h , whether by chance you do not define somewhere:

#define MEMP_SANITY_CHECK 1

If so, remove that line or set it to zero. As such, my packet sending performance did not increase (I'm still around 11Mbits when firing on maximum speed the data), but the total throughput increased considerably as the time between two sent packet is now constant and takes roughly 100us.

It needs to be said, that this still does not resolve the issue of having only 11MBits on 100MBit line completely dedicated only to this equipment