to_dict doesn't convert np.int64 to python integers (second try)

[0anton]

Code Sample, a copy-pastable example if possible

# Your code here
import pandas as pd

data = [{'id': 11, 'text': 'Osv1wbZoL'},
        {'id': 0, 'text': 'KQpPReW3S9nZOS3'},
        {'id': 0, 'text': 'cbqLhjrb0B2Ah6E'},
        {'id': 3, 'text': 'qu1Jlnyba'},
        {'id': 14, 'text': 'aJUv5DBjbcGc3'},
        {'id': 12, 'text': 'Yobf9'},
        {'id': 4, 'text': 'awzZCV'},
        {'id': 4, 'text': '3NvBAVL'},
        {'id': 11, 'text': '80sPCxIf9s5wmEZ1'},
        {'id': 5, 'text': 'afrPD0X6mIzFK'}]
df = pd.DataFrame(data)

df.dtypes
id       int64
text    object
dtype: object
type(df[['id', 'text']].to_dict(orient='records')[0]['id'])
<class 'int'>
type(df[['id']].to_dict(orient='records')[0]['id'])
<class 'numpy.int64'>

Problem description

Here I am just repeating here the description given my @Tomasz-Kluczkowski in #16048, which I believe is erroneously closed, because the issue still there in the latest stable version.

depending on the count of output columns, numpy integers getting converted to python integers
afterwards both json.dumps and ujson.dumps fails to encode

#13258 promises the solution for 0.21.0, but as seen above issue is still there in 0.23.4

Als workaround helps casting all types to strings bevor calling to_dict():

type(df[['id']].astype(object).to_dict(orient='records')[0]['id'])
<class 'int'>

Opening this as a new issue, because I cannot re-open the #16048.

Expected Output

type(df[['id']].to_dict(orient='records')[0]['id'])
<class 'int'>

Output of pd.show_versions()

pd.show_versions()
INSTALLED VERSIONS
------------------
commit: None
python: 3.7.2.final.0
python-bits: 32
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 94 Stepping 3, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None
pandas: 0.23.4
pytest: 4.1.1
pip: 18.1
setuptools: 40.6.3
Cython: None
numpy: 1.16.0
scipy: None
pyarrow: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.7.5
pytz: 2018.9
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: 1.2.16
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

[chris-b1]

This is fixed in soon to be release 0.24, or you can install the rc now. pip install --pre --upgrade pandas

[chris-b1]

Note that actual closing PR is #23921

[andrey999333]

i'm still having this issue in version 0.25.1

[jreback]

@andrey999333 pls open a new issue with a reproducible example

[selik]

I also am getting this issue, but having trouble reproducing in a small example. In case it matters, I used astype(int) before to_dict.

[alexanderwhatley]

I'm getting this issue as well in the latest pandas version (1.1.4). Haven't been able to create a small reproducible example either. I should mention that my code worked fine on 0.25.1.