BUG: to_json segfaults when exception occurs in UTF8 encoding of string

[lithomas1]

• closes BUG: strange string causes segmentation fault on df.to_json #50307 (Replace xxxx with the GitHub issue number)
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

[WillAyd]

Very nice

[gfyoung]

LGTM pending CI

[WillAyd]

Awesome thanks @lithomas1 would take an audit of any of these to support #49756

[phofl]

it looks like this causes the 32bit built to time out on main. Any ideas why this is happening?

[lithomas1]

Not sure, do you know the exact test that is timing out?

[MarcoGorelli]

Pretty sure it's pandas/tests/io/json/test_ujson.py

CI passes if I skip it #50381

[lithomas1]

I can confirm that it hangs on the test added in this PR. I'm going to debug more, but feel free to revert this PR in the meantime.

[WillAyd]

Are there any warnings being generated in the logs? I would also maybe check what happens if we get rid of some of the casts - generally we do this too much in the C code

[lithomas1]

I've pulled the 32 bit docker container onto my machine, so hopefully I'll be able to debug with gdb soon.

[WillAyd]

My other guess is that enc->errorMsg = "Encoding failed."; is problematic. I think the rhs of this is a char[] with auto storage, and it is possible that the memory for it has been reused after the function exists and by the time ultrajson actually evaluates it. This could cause undefined behavior

You could try heap allocating that string and see if it allows tests to pass (though it will cause a memory leak)

[WillAyd]

Looks like I was wrong about the duration of the string literal

https://stackoverflow.com/questions/9970295/life-time-of-a-string-literal-in-c

[lithomas1]

Looks like its getting stuck in Buffer_ReAlloc

#0  Buffer_Realloc (enc=0xffa2ac0c, cbNeeded=3079809510)
    at pandas/_libs/src/ujson/lib/ultrajsonenc.c:373
#1  0xf3ede5f8 in encode (obj=0xf1ace110, enc=0xffa2ac0c, name=0x0, cbName=0)
    at pandas/_libs/src/ujson/lib/ultrajsonenc.c:1083
#2  0xf3ede240 in encode (obj=0xf1a55828, enc=0xffa2ac0c, name=0x0, cbName=0)
    at pandas/_libs/src/ujson/lib/ultrajsonenc.c:993
#3  0xf3edea7d in JSON_EncodeObject (obj=0xf1a55828, enc=0xffa2ac0c, 
    _buffer=0xffa2acc4 "[", _cbBuffer=65536)

Maybe I need to set the length of the string to zero.

[lithomas1]

Looks like that did it. PR incoming.