lwIP / TCP tx'd pbuf ACK'd?

[LCE]

Hello,

I have to admit that this is more a question concerning lwIP, but as I'm using a STM32F7...

And the search engines haven't helped yet.

I need to transmit TCP packets quickly, without waiting for TCP's ACK.

Payload for these packets come from an array of DMA buffers (sourced by SAI) which I might need for other interfaces, like USB output, not at the same time though.

Each of these buffers is given a state, like FREE, WRITING, TRANSMITTING, WAIT_ACK.

So I give a buffer to lwIP with tcp_write() (building its own pbuf) and call tcp_output() immediately.

Most examples I found also set the tcp_sent callback, and only continue transmitting after the latest data has been ACK'd by the receiver. I have no time for that, wireshark showed me that ACK can take a few milli seconds - too long.

I've been through lwIP's TCP functions, but I cannot find how to pass back the info from lwIP which pbuf / data was recently ACK'd to free the DMA buffers.

tcp_sent() length does not help, because all DMA buffers are the same size.

Any hints or ideas?

I hope there's something I have not yet understood or have overseen.

I could start fiddling within the lwIP functions, but I hope there's a more elegant way.

PS: otherwise zero-copy TX ethernet is working, as is http, PTP

[LCE]

It looks like the solution might be using the tcp_pcb, which has the member unacked, which again holds info to unacked segments including the sent pbuf:

/** the TCP protocol control block */
struct tcp_pcb
{
...
	/* These are ordered by sequence number: */
	struct tcp_seg *unsent; 		/* Unsent (queued) segments. */
	struct tcp_seg *unacked; 		/* Sent but unacknowledged segments. */
...
}
 
/* This structure represents a TCP segment on the unsent, unacked and ooseq queues */
struct tcp_seg
{
	struct tcp_seg *next;		/* used when putting segments on a queue */
	struct pbuf *p; 		/* buffer containing data + TCP header */
	u16_t len; 		/* the TCP length of this segment */
...
}

I had overseen that the segment holds a pbuf...

[Bob S]

Disable Nagel's algorithm. That is the code that buffers outgoing data until the previously sent data is ACK'd. How you do that depends on which LwIP interface you are using (raw, netconn or sockets)..

[LCE]

Raw, and yes, Nagle is disabled.

[Piranha]

> I need to transmit TCP packets quickly, without waiting for TCP's ACK.

If you don't need guaranteed delivery, you don't need TCP at all. Sounds like you should use UDP transport protocol for this purpose.

> I cannot find how to pass back the info from lwIP which pbuf / data was recently ACK'd to free the DMA buffers

Create pbuf_custom buffers and wait for the custom_free_function() to be called.

[LCE]

I need guaranteed delivery, so I need the ACK.

But until now I found that the ACK might take a few ms, and I can't afford to wait that long before sending the next packet, but have enough buffers to keep data for about 40 ms.

Thanks again, I'll have a look at the custom buffers.

[Piranha]

You misunderstood my idea. The buffer size as such doesn't matter and same size buffers are even slightly easier. And the callback most likely will be called for TCP segments, the size of which is not guaranteed to directly correspond to your buffers.

err_t tcp_sent_fn(void *arg, struct tcp_pcb *tpcb, u16_t len)
{
	static size_t nbSent_;
	
	for (nbSent_ += len; nbSent_ >= BUF_SIZE; nbSent_ -= BUF_SIZE) {
		// Free the next buffer
	}
}

[LCE]

Okay, so I have to assume and hope that packets will be acknowledged in the right order.

I had the hope that I could solve that a little smarter, in case some older packet is not ACK'd and must be sent again.

Until now, Wireshark shows that ACKs are coming quite quick and "linear".

But now I have just one device with 6.4 Mbit/s, it might go up to 4 devices (via Gb-switch) with ~ 50 Mbit/s each.

But first things first, let's get one up with 50 Mbps...

[Piranha]

As the TCP presents an ordered stream (not packets) to the application, the callback must be called sequentially. Otherwise such API doesn't make a sense at all. Also I found a confirmation here:

https://lists.gnu.org/archive/html/lwip-users/2022-02/msg00003.html

[LCE]

Update:

I had Piranha's solution implemented, counting sent/ACK'd bytes and freeing buffers.

But I found that this was not good enough, every few hours there was some late ACK (when a retransmission was needed I think), so this kind of "linear" scheme combined with linear source buffers (SAIs -> DMA) was blocking at some point, yet with other buffers free.

So I built a queue for the source buffers,

added a custom pbuf variable with a pointer to the source buffer,

copied and changed tcp_write() for that application which adds the corresponding pointer to the pbuf,

and added in pbuf_free() that if that pointer is not NULL, then the source buffer's state changes to free.

Learned a lot about lwIP...

But still feels awkward having changed some lwIP stuff.