Sysex packets not arriving

[ClintH]

Hi,

I have a problem where sysex packets intermittently don't bubble up through the API.

Background
I'm using the library on a ESP32 connecting to rtpMIDI v1.1.11(244) running on Windows 10. The Windows machine hosts a RTP session which the ESP32 joins by IP address. I'm sending control surface sysex from Ableton, which can get quite large. It seems to be these ones that go missing more often.

Troubleshooting
I turned on DEBUG and APPLEMIDI_DEBUG_VERBOSE flags in the library, and it's not giving any errors or warnings. The "Incoming buffer of ... bytes" message never shows for packets that seem lost. The only error that shows up is the following, but that's not correlated time-wise to when the large packets are lost:

Incoming buffer of 36 bytes: ff ff 43 4b 78 8a 91 76 1 0 0 0 0 0 0 0 0 8 80 a4 0 0 0 0 8 ea 88 eb 0 0 0 0 0 0 0 0 
dissect_rtp_midi 5005, packetSize is 36
start: dissect_rtp
current: 3
unsupported version 3
Unprocessed packet (No valid rtp midi content).

Using Wireshark, it seems the packet is being emitted by rtpMIDI as it should. Here's an example of something that never arrived (the first column is a location reference added by Wireshark). This was sent as a single packet of 245 bytes.

0000   80 7d 3a b9 c3 44 74 d0 2b 2a fc 6a 08 00 45 00
0010   00 e7 ab 5c 00 00 80 11 0b 4b c0 a8 01 02 c0 a8
0020   01 0c 13 8d 13 8d 00 d3 87 80 80 61 01 fb 07 f1
0030   85 70 78 8a 91 76 80 bd f0 00 20 29 02 0a 01 02
0040   08 01 00 42 73 63 52 63 6b 44 6c 43 00 00 01 00
0050   41 6d 70 20 47 61 69 6e 00 01 01 00 41 6d 70 50
0060   72 65 73 6e 63 00 02 01 00 53 74 65 72 65 57 64
0070   74 68 00 03 01 00 52 65 76 72 62 41 6d 6e 74 00
0080   04 01 00 43 6d 70 54 68 72 65 73 68 00 05 01 00
0090   53 74 72 74 72 41 6d 6e 74 00 06 01 00 52 63 6b
00a0   44 72 79 2f 57 74 00 07 01 00 52 61 63 6b 56 6c
00b0   75 6d 65 00 00 01 01 35 39 00 01 01 01 34 33 00
00c0   02 01 01 36 35 00 03 01 01 33 30 20 25 00 04 01
00d0   01 2d 30 2e 39 39 20 64 42 00 05 01 01 31 30 00
00e0   06 01 01 31 31 37 00 07 01 01 2d 33 2e 34 34 37
00f0   20 64 42 00 f7

In AppleMidi_Settings.h, I've changed the following:

#define PACKET_MAX_SIZE 512 // 350
static const unsigned SysExMaxSize = 512; // 128;

I'd be grateful for tips for getting to the bottom of this!

[lathoub]

Issue #57 indicated that Sysex did got thru. Strange that you get it thru.
I can't test it right away, but will look at it later

[ClintH]

Thanks, lathoub!

It fails pretty often on packets of varying sizes over 200 bytes. Unlike #57 there's not an exact packet size when it fails. The smaller ones always seem to get through. Sending the same packet data from the same machine to a UDP receiver on my phone (on the same wifi network as the ESP) works fine, so it's not my AP.

I can't rule out this being an ESP software stack bug. I've tried spelunking through arduino-esp32 and esp32-idf but I can't find this specific issue out there. The closest issue I could find is this one, but the symptoms are different, and I tried the patches that are supposed to fix it to no avail.

[lathoub]

Hi @ClintH

I just tried on an Arduino Ethernet and it worked without a problem. I have a ESP32 arriving next week and will try on it as well. Did you get it resolved already?
Beware: UDP does not guarantee delivery of the packet (albeit I have not seen cases where it was actually dropped)

Lathout

[lathoub]

Other note: in the 245 bytes example above, the message does not start with SysEx start byte 0xf0 (but does end with the SysEx end byte 0xf7

[ClintH]

Hi @lathoub, I gave up and have been using MIDI over USB. UDP is not that unreliable that packets are consistently lost on a home network. Besides, sending large packets through other means seems to work. I think the dump is prefixed by some other messages, all sent in the one packet.

I look forward to how your ESP32 testing goes! I suspect the issue is on that side rather than your library.