Судя по всему nginx надо пересобирать с лимитом этих самых fastopen, только почему он тогда не ругается на то, как сейчас выставлено... Параметр в ядре пробовал и в 1, 2, 3 ставить...
tcp_fastopen - INTEGER
Enable TCP Fast Open feature (draft-ietf-tcpm-fastopen) to send data in the opening SYN packet. To use this feature, the client application must use sendmsg() or sendto() with MSG_FASTOPEN flag rather than connect() to perform a TCP handshake automatically.
The values (bitmap) are
1: Enables sending data in the opening SYN on the client.
2: Enables TCP Fast Open on the server side, i.e., allowing data in a SYN packet to be accepted and passed to the application before 3-way hand shake finishes.
4: Send data in the opening SYN regardless of cookie availability and without a cookie option.
0x100: Accept SYN data w/o validating the cookie.
0x200: Accept data-in-SYN w/o any cookie option present.
0x400/0x800: Enable Fast Open on all listeners regardless of the TCP_FASTOPEN socket option. The two different flags designate two different ways of setting max_qlen without the TCP_FASTOPEN socket option.