BUG: `Series.to_json()` wraps `np.uint64` values ≥ `2**63` when stored in `object` dtype

[Ethan0456]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import numpy as np
import pandas as pd

pd.Series([np.uint64(2**63)], dtype=object).to_json()
# '{"0":-9223372036854775808}'

Issue Description

Expected Output for Reproducible Example

'{"0":9223372036854775808}'

Observed Behavior

It appears that np.uint64 scalars stored in object-dtype Series are being serialized through a signed int64 path, causing values ≥ 2**63 to wrap to negative numbers.

>>> pd.Series([np.uint64(2**63)], dtype="uint64").to_json()
'{"0":9223372036854775808}'

>>> pd.Series([np.uint64(2**63)], dtype=object).to_json()
'{"0":-9223372036854775808}'

Existing tests and documentation

• ujson.ujson_dumps explicitly tests Python integers up to 18446744073709551615 (2**64 - 1) and expects exact JSON output. [ref]
• DataFrame.to_json has a test for large uint64-looking values like 13342205958987758245, and expects them to serialize correctly. [ref]
• The docs for to_json/read_json do not appear to say "ujson only encodes up to signed int64." [read_json ref] [to_json ref]

There is one internal test comment in pandas/tests/io/json/test_ujson.py:718:

pandas/pandas/tests/io/json/test_ujson.py

Lines 712 to 726 in f7246dc

	def test_int_max(self, any_int_numpy_dtype):
	if any_int_numpy_dtype in ("int64", "uint64") and not IS64:
	pytest.skip("Cannot test 64-bit integer on 32-bit platform")
	klass = np.dtype(any_int_numpy_dtype).type
	# uint64 max will always overflow,
	# as it's encoded to signed.
	if any_int_numpy_dtype == "uint64":
	num = np.iinfo("int64").max
	else:
	num = np.iinfo(any_int_numpy_dtype).max
	assert klass(ujson.ujson_loads(ujson.ujson_dumps(num))) == num

But that test is only avoiding np.uint64.max in a numpy-scalar roundtrip test. This suggests the object-scalar path may already have a known limitation around np.uint64 values.

Debugging Hints

From debugging, it appears the object-scalar serialization path may be converting np.uint64 values using the signed integer conversion (NPY_INT64) rather than the unsigned path (NPY_UINT64).

pandas/pandas/_libs/src/vendored/ujson/python/objToJSON.c

Lines 1627 to 1631 in f7246dc

	} else if (PyArray_IsScalar(obj, Integer)) {
	tc->type = JT_LONG;
	PyArray_CastScalarToCtype(obj, &(pc->longValue),
	PyArray_DescrFromType(NPY_INT64));

If this is confirmed as a bug, I’d be happy to help work on a fix and open a pull request.

Expected Behavior

Series.to_json() should preserve the value of np.uint64 scalars regardless of whether they are stored in a uint64 Series or an object Series.

Installed Versions

Details

pandas==3.1.0.dev0+1194.gf7246dccfa

[Pavan19047]

Hey @AlexKirko,
Can i work on this issue?

[Ethan0456]

Hi @Pavan19047, thanks for your interest! I had mentioned earlier that I'd work on this if it was confirmed to be a bug, and I already have a working fix. I'll be opening a PR shortly, so I'd appreciate it if you could let me continue with this one. Thanks!

[Pavan19047]

Got it @Ethan0456 , thanks for confirming that you're already working on this

[anzinmhd]

@Ethan0456 Apologies for the overlap, I missed your comment before opening the PR. Happy to close mine if you're ready to submit yours. If it helps, feel free to take a look at my approach in #66148.

[Ethan0456]

Thanks for the PR and for the thoughtful approach in #66148. I’ve also opened #66149 with my fix, so we now have two implementations. Happy to let the maintainers decide on the best direction or merge ideas if useful.