Track re-adding neqo to s2n-quic interop test

[johubertj]

Security issue notifications

Problem:

neqo QUIC implementation consistently fails interop tests with s2n-quic in both client and server roles. Failures occur across multiple test cases. We suspect an issue within neqo caused the interop test failure. Neqo developer acknowledged issue

Removal PR to unblock CI: #2724 (comment)

Need By Date:

No strict deadline. neqo should be re-enabled in interop tests as soon as its bug is resolved to restore full test coverage.

Solution:

• Re-enable neqo in QUIC interop tests once the upstream bug is fixed by the neqo team.

Requirements / Acceptance Criteria:

• neqo is restored in QUIC interop test configuration.
• All neqo-related interop scenarios pass in CI.

Out of scope:

[mxinden]

👋 mozilla/neqo#2807 has been fixed. The latest Docker image should be up-to-date.

Let me know in case you are still facing any issues.

[boquan-fang]

Related PRs: #2734, #2732.

This issue needs more investigations from our side.

[boquan-fang]

I am able to reproduce the interop report test failure locally using the instruction from https://github.com/aws/s2n-quic/tree/main/scripts/interop.

Documenting the error I am seeing from neqo: https://dnglbrstg7yg.cloudfront.net/latest/interop/logs/latest/neqo_s2n-quic/handshake/output.txt

server  | 1.088 DEBUG [Server 259e67bb4fd4614e] output_path send_profile SendProfile { limit: 255, pto: None, probe: EnumSet(), paced: true }
server  | 1.088 DEBUG Building Short dcid Some(CID [16]: 2a3c78be1ada6d8c78c89ea1466a3a12)
server  | 1.088 DEBUG TX blocked, profile=SendProfile { limit: 255, pto: None, probe: EnumSet(), paced: true }
server  | 1.088 DEBUG ECN probing: sent 2 probes
server  | 1.088 INFO `libc::sendmsg` failed with Input/output error (os error 5); halting segmentation offload
server  | Error: Io(Os { code: 5, kind: Uncategorized, message: "Input/output error" })

server exited with code 1

Seems like neqo tries to do do libc::sendmsg which is failing on our interop test environment. This is not failing on the main interop runner: https://interop.seemann.io/logs/2025-08-28T02:02/neqo_s2n-quic/handshake/output.txt

server  | 1.126 DEBUG [Server 04098d19c598bc80] output_path send_profile SendProfile { limit: 1232, pto: None, probe: EnumSet(), paced: false }
server  | 1.126 DEBUG Building Short dcid Some(CID [16]: 0920f53fab2baf33e9d0b13824975722)
server  | 1.126 DEBUG TX blocked, profile=SendProfile { limit: 1232, pto: None, probe: EnumSet(), paced: false }
server  | 1.126 DEBUG [Server 04098d19c598bc80] delay duration 19.815762ms
server  | 1.126 DEBUG Setting timeout of 19.815762ms
server  | 1.143 DEBUG [Server 04098d19c598bc80] pn=3 type=Short pri-path:0920f53fab2baf33e9d0b13824975722 [::]:443->[::ffff:193.167.0.100]:55236 Tos(Cs0, Ect0) len 47
server  |   -> RX ConnectionClose { error_code: Transport(0), frame_type: 0, reason_phrase: "" }
server  |   -> RX Padding { len: 15 }
server  | 1.143 INFO [Server 04098d19c598bc80] ConnectionClose received. Error code: Transport(0) frame type 0 reason 
server  | 1.143 DEBUG [Server 04098d19c598bc80] State change from Confirmed -> Draining { error: Transport(Peer(0)), timeout: Instant { tv_sec: 1156, tv_nsec: 715922727 } }
server  | 1.143 DEBUG [Server 04098d19c598bc80] output_path send_profile SendProfile { limit: 1232, pto: None, probe: EnumSet(), paced: false }
server  | 1.143 DEBUG Building Short dcid Some(CID [16]: 0920f53fab2baf33e9d0b13824975722)
server  | 1.143 DEBUG [Recvd-ap] immediate_ack at Instant { tv_sec: 1156, tv_nsec: 389297412 }
server  | 1.143 DEBUG [Server 04098d19c598bc80] pn=7 type=Short pri-path:0920f53fab2baf33e9d0b13824975722 [::]:443->[::ffff:193.167.0.100]:55236 Tos(Cs0, Ect0) len 46
server  |   TX -> Ack { largest_acknowledged: 3, ack_delay: 0, first_ack_range: 3, ack_ranges: [], ecn_count: Some(Count({NotEct: 0, Ect1: 0, Ect0: 4, Ce: 0})) }
server  |   TX -> ConnectionClose { error_code: Transport(0), frame_type: 28, reason_phrase: "" }
server  | 1.143 DEBUG ECN probing: sent 8 probes
server  | 1.143 DEBUG Setting timeout of 326.505709ms
server  | 1.143 DEBUG Setting timeout of 326.494368ms
server  | 1.209 DEBUG [Server ...] Dropped received packet: State Draining { error: Transport(Peer(0)), timeout: Instant { tv_sec: 1156, tv_nsec: 715922727 } }; Total: 1
server  | 1.209 DEBUG Setting timeout of 260.71903ms
server  | 1.209 DEBUG Setting timeout of 260.698752ms
server  | 1.470 DEBUG [Server 04098d19c598bc80] State change from Draining { error: Transport(Peer(0)), timeout: Instant { tv_sec: 1156, tv_nsec: 715922727 } } -> Closed(Transport(Peer(0)))
server  | 1.470 INFO Closing timer expired

We are using a forked of the network simulator:

s2n-quic/.github/interop/runner.patch

Lines 19 to 22 in d2c0794

	services:
	sim:
	- image: martenseemann/quic-network-simulator
	+ image: wesleyrosenblum/quic-network-simulator

I am suspecting that something go wrong there. Needs more investigation into this matter.

[boquan-fang]

I have tried to update the QUIC Interop runner to the latest version, but Neqo still doesn't work. I will wait for @WesleyRosenblum to update the quic network simulator and update the docker image https://hub.docker.com/r/wesleyrosenblum/quic-network-simulator as another attempt.