Skip to content

Failed assertion in parser.c (ts_parser__external_scanner_serialize) #134

@domodwyer

Description

@domodwyer

Hi there!

I have recently wrote a TLA formatter using the rust bindings (tlafmt), and as part of the project I have been fuzzing the internal library end-to-end - it has caught a failed assertion in the tree-sitter code:

Assertion failed: (length <= 1024), function ts_parser__external_scanner_serialize, file parser.c, line 408.

For the following input (2456 chars long):

~-e*d*\//\-\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-d\/C*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-d*\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f-d*\/-qeA*\//\-d*\//\-d*\/-\A�TJe-f'T@*d*\//\-Je-f'T@*d*\//\-\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-d*\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-/\A�TJe-f'T@*d*\//\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-d*\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-@*d*\//\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-d*\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f-d*\/-qeA*\//\-d*\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-Je-f'T@*d*\//\-\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-d*\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-/\A�TJe-f'T@*d*\//\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-d*\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-Je-f'T@*d*\//\-\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-d*\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-/\A�TJe-f'T@*d*\//\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-d*\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-e-f'T@*d*\//\-/\A�TJe-f'T@*d*\//\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-d*\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-Je-f'T@*d*\//\-\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-d*\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-/\A�TJe-f'T@*d*\//\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-d*\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-
Here it is as a byte array too.

The string is a bit unwieldy - here's a byte array:

const CRASH: [u8; 2456] = [
    0x7e, 0x2d, 0x65, 0x2a, 0x64, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d, 0x5c, 0x2d, 0x64, 0x2a, 0x5c,
    0x2f, 0x2d, 0x71, 0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27,
    0x54, 0x40, 0x2a, 0x64, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71,
    0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a,
    0x64, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x43, 0x2a,
    0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e,
    0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64, 0x65, 0x43, 0x2a, 0x2f, 0x5c, 0x2d,
    0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65,
    0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d, 0x64, 0x5c, 0x2f,
    0x43, 0x2a, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x41, 0x2a, 0x2f, 0x5c,
    0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64, 0x2a, 0x5c, 0x2f, 0x2f,
    0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65,
    0x43, 0x2a, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x41, 0x2a, 0x2f, 0x5c,
    0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64, 0x65, 0x43, 0x2a, 0x2f,
    0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e, 0x54,
    0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d, 0x2f,
    0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x43, 0x2a, 0x2f, 0x5c, 0x2d, 0x64,
    0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d,
    0x66, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x41, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d,
    0x64, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x5c, 0x41, 0x0e, 0x54,
    0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d, 0x4a,
    0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d, 0x5c, 0x2f,
    0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x43, 0x2a, 0x2f, 0x5c, 0x2d, 0x64,
    0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d,
    0x66, 0x27, 0x54, 0x40, 0x2a, 0x64, 0x65, 0x43, 0x2a, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f,
    0x2d, 0x71, 0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54,
    0x40, 0x2a, 0x64, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d,
    0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x43, 0x2a, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f,
    0x2d, 0x71, 0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54,
    0x40, 0x2a, 0x64, 0x65, 0x43, 0x2a, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65,
    0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64,
    0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d, 0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27,
    0x54, 0x40, 0x2a, 0x64, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71,
    0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a,
    0x64, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x43, 0x2a,
    0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e,
    0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64, 0x65, 0x43, 0x2a, 0x2f, 0x5c, 0x2d,
    0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65,
    0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c,
    0x2f, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x43, 0x2a, 0x2f, 0x5c, 0x2d,
    0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65,
    0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64, 0x65, 0x43, 0x2a, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c,
    0x2f, 0x2d, 0x71, 0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27,
    0x54, 0x40, 0x2a, 0x64, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d, 0x40, 0x2a, 0x64, 0x2a, 0x5c, 0x2f,
    0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e,
    0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d,
    0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x43, 0x2a, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f,
    0x2d, 0x71, 0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54,
    0x40, 0x2a, 0x64, 0x65, 0x43, 0x2a, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65,
    0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64,
    0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c,
    0x2f, 0x2d, 0x71, 0x65, 0x43, 0x2a, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65,
    0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64,
    0x65, 0x43, 0x2a, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x41, 0x2a, 0x2f,
    0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64, 0x2a, 0x5c, 0x2f,
    0x2f, 0x5c, 0x2d, 0x2f, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x43, 0x2a,
    0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e,
    0x54, 0x4a, 0x65, 0x2d, 0x66, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x41, 0x2a, 0x5c,
    0x2f, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d,
    0x71, 0x65, 0x43, 0x2a, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x41, 0x2a,
    0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64, 0x65, 0x43,
    0x2a, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41,
    0x0e, 0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c,
    0x2d, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d,
    0x5c, 0x2f, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x43, 0x2a, 0x2f, 0x5c,
    0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a,
    0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64, 0x65, 0x43, 0x2a, 0x2f, 0x5c, 0x2d, 0x64, 0x2a,
    0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d, 0x66,
    0x27, 0x54, 0x40, 0x2a, 0x64, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2f,
    0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x43, 0x2a, 0x2f, 0x5c, 0x2d, 0x64, 0x2a,
    0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d, 0x66,
    0x27, 0x54, 0x40, 0x2a, 0x64, 0x65, 0x43, 0x2a, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d,
    0x71, 0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54, 0x40,
    0x2a, 0x64, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d, 0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d,
    0x66, 0x27, 0x54, 0x40, 0x2a, 0x64, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f,
    0x2d, 0x71, 0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54,
    0x40, 0x2a, 0x64, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65,
    0x43, 0x2a, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x41, 0x2a, 0x2f, 0x5c,
    0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64, 0x65, 0x43, 0x2a, 0x2f,
    0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e, 0x54,
    0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d, 0x64,
    0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x43, 0x2a, 0x2f,
    0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e, 0x54,
    0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64, 0x65, 0x43, 0x2a, 0x2f, 0x5c, 0x2d, 0x64,
    0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d,
    0x66, 0x27, 0x54, 0x40, 0x2a, 0x64, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d, 0x2f, 0x2f, 0x5c, 0x2d,
    0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x43, 0x2a, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f,
    0x2d, 0x71, 0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54,
    0x40, 0x2a, 0x64, 0x65, 0x43, 0x2a, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65,
    0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64,
    0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64, 0x2a,
    0x5c, 0x2f, 0x2f, 0x5c, 0x2d, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71,
    0x65, 0x43, 0x2a, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x41, 0x2a, 0x2f,
    0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64, 0x65, 0x43, 0x2a,
    0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e,
    0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d,
    0x64, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x43, 0x2a,
    0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e,
    0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64, 0x65, 0x43, 0x2a, 0x2f, 0x5c, 0x2d,
    0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65,
    0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d, 0x2f, 0x5c, 0x41,
    0x0e, 0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c,
    0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a,
    0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d, 0x64, 0x2a,
    0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x43, 0x2a, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71,
    0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a,
    0x64, 0x65, 0x43, 0x2a, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x41, 0x2a,
    0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64, 0x2a, 0x5c,
    0x2f, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d,
    0x71, 0x65, 0x43, 0x2a, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x41, 0x2a,
    0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64, 0x65, 0x43,
    0x2a, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41,
    0x0e, 0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c,
    0x2d, 0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d, 0x2f,
    0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64, 0x2a, 0x5c, 0x2f,
    0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e,
    0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d,
    0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x43, 0x2a, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f,
    0x2d, 0x71, 0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54,
    0x40, 0x2a, 0x64, 0x65, 0x43, 0x2a, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65,
    0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64,
    0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c,
    0x2f, 0x2d, 0x71, 0x65, 0x43, 0x2a, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65,
    0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64,
    0x65, 0x43, 0x2a, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x41, 0x2a, 0x2f,
    0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64, 0x2a, 0x5c, 0x2f,
    0x2f, 0x5c, 0x2d, 0x2f, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x43, 0x2a,
    0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e,
    0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64, 0x65, 0x43, 0x2a, 0x2f, 0x5c, 0x2d,
    0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65,
    0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d, 0x4a, 0x65, 0x2d,
    0x66, 0x27, 0x54, 0x40, 0x2a, 0x64, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d, 0x5c, 0x2f, 0x2f, 0x5c,
    0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x43, 0x2a, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c,
    0x2f, 0x2d, 0x71, 0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27,
    0x54, 0x40, 0x2a, 0x64, 0x65, 0x43, 0x2a, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71,
    0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a,
    0x64, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d, 0x64, 0x2a,
    0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x43, 0x2a, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71,
    0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a,
    0x64, 0x65, 0x43, 0x2a, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x41, 0x2a,
    0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64, 0x2a, 0x5c,
    0x2f, 0x2f, 0x5c, 0x2d, 0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54, 0x40,
    0x2a, 0x64, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x41,
    0x2a, 0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64, 0x2a,
    0x5c, 0x2f, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x43, 0x2a, 0x2f, 0x5c,
    0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a,
    0x65, 0x2d, 0x66, 0x27, 0x54, 0x40, 0x2a, 0x64, 0x65, 0x43, 0x2a, 0x2f, 0x5c, 0x2d, 0x64, 0x2a,
    0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d, 0x66,
    0x27, 0x54, 0x40, 0x2a, 0x64, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2f,
    0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x43, 0x2a, 0x2f, 0x5c, 0x2d, 0x64, 0x2a,
    0x5c, 0x2f, 0x2d, 0x71, 0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d, 0x66,
    0x27, 0x54, 0x40, 0x2a, 0x64, 0x65, 0x43, 0x2a, 0x2f, 0x5c, 0x2d, 0x64, 0x2a, 0x5c, 0x2f, 0x2d,
    0x71, 0x65, 0x41, 0x2a, 0x2f, 0x5c, 0x41, 0x0e, 0x54, 0x4a, 0x65, 0x2d, 0x66, 0x27, 0x54, 0x40,
    0x2a, 0x64, 0x2a, 0x5c, 0x2f, 0x2f, 0x5c, 0x2d,
];

I can reproduce this by:

  1. Cloning this repo (current HEAD: d711f24)
  2. Modifying the test/consumers/rust/src/main.rs file:
diff --git old - test/consumers/rust/src/main.rs new - test/consumers/rust/src/main.rs
index 04129f9..5765157 100644
--- old - test/consumers/rust/src/main.rs	
+++ new - test/consumers/rust/src/main.rs	
@@ -1,26 +1,13 @@
-use tree_sitter::{Parser, Query, QueryCursor};
 use streaming_iterator::StreamingIterator;
+use tree_sitter::{Parser, Query, QueryCursor};
 
 fn main() {
     let mut parser = Parser::new();
     parser
         .set_language(&tree_sitter_tlaplus::LANGUAGE.into())
         .expect("Error loading TLA+ grammar");
-    let source_code = r#"
-        ---- MODULE Test ----
-        op ≜ ∀ n ∈ ℕ : n ≥ 0
-        ===="#;
-    let tree = parser.parse(source_code, None).unwrap();
-    println!("{}", tree.root_node().to_sexp());
 
-    let query = Query::new(
-        &tree_sitter_tlaplus::LANGUAGE.into(),
-        "(def_eq \"≜\") @capture",
-    )
-    .unwrap();
-    let mut cursor = QueryCursor::new();
-    let mut captures = cursor.matches(&query, tree.root_node(), "".as_bytes());
-    while let Some(capture) = captures.next() {
-        println!("{:?}", capture);
-    }
+    let s = r#"~-e*d*\//\-\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-d\/C*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-d*\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f-d*\/-qeA*\//\-d*\//\-d*\/-\A�TJe-f'T@*d*\//\-Je-f'T@*d*\//\-\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-d*\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-/\A�TJe-f'T@*d*\//\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-d*\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-@*d*\//\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-d*\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f-d*\/-qeA*\//\-d*\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-Je-f'T@*d*\//\-\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-d*\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-/\A�TJe-f'T@*d*\//\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-d*\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-Je-f'T@*d*\//\-\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-d*\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-/\A�TJe-f'T@*d*\//\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-d*\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-e-f'T@*d*\//\-/\A�TJe-f'T@*d*\//\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-d*\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-Je-f'T@*d*\//\-\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-d*\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-/\A�TJe-f'T@*d*\//\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-d*\//\-d*\/-qeC*/\-d*\/-qeA*/\A�TJe-f'T@*deC*/\-d*\/-qeA*/\A�TJe-f'T@*d*\//\-"#;
+
+    let _tree = parser.parse(s, None).unwrap();
 }
  1. Run cargo run and you'll get an assertion failure.

The sample consumer uses tree-sitter 0.24.3, but I have confirmed the same crash using the latest 0.25.2 too. I did notice a similar issue in the tree-sitter repo that was closed as "this is the downstream scanner", which is why I'm opening the ticket here.

I tried to reproduce this in the python consumer too, but I ran into dependency conflicts so never got it working, but I did manage to get the tla-web trace explorer to error with an assertion failure in the console too so it seems likely to be the shared core rather than the rust specific binding.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions