Handle int64 columns with missing data in SQL Lab

[betodealmeida]

CATEGORY

Choose one

• Bug Fix
• Enhancement (new features, refinement)
• Refactor
• Add tests
• Build / Development Environment
• Documentation

This PR fixes #8225.

SUMMARY

When a column has int64 integers and missing data, Pandas will cast it to float64, resulting in loss of precision and possibly returning incorrect numbers.

This PR fixes the bug by adding a method to the DB engine specs that returns a dtype based on the cursor description, currently implemented in Presto only. With the dtype, we can create a Pandas Series for each column, and create a DataFrame that has the proper types.

Note that in order to represent the column correctly we need to use a nullable data type, introduced in Pandas 0.240. Unfortunately, PyArrow is unable to serialize the resulting data frame, so msgpack has to be disabled.

TEST PLAN

Added unit test.

ADDITIONAL INFORMATION

• Has associated issue: Pandas casting int64 to float64, misrepresenting value #8225
• Changes UI
• Requires DB Migration.
• Confirm DB Migration upgrade and downgrade tested.
• Introduces new feature or API
• Removes existing feature or API

REVIEWERS

@villebro @robdiciuccio

[betodealmeida]

Actually, this method is Pandas specific. I'll have to use this and the previous one together.

[mistercrunch]

@robdiciuccio ^^^

[robdiciuccio]

The dataframe implementation looks good, as does the Presto engine dtype fix, but does this fully address #8225 if only Presto is handled? Are there other databases this fix should be implemented for (even in a separate PR)?

Also curious about your thoughts on the feasibility of the PyArrow workaround here.

[betodealmeida]

The dataframe implementation looks good, as does the Presto engine dtype fix, but does this fully address #8225 if only Presto is handled? Are there other databases this fix should be implemented for (even in a separate PR)?

You're right, this only fixes Presto. I'll do a separate PR addressing the other DBs.

Also curious about your thoughts on the feasibility of the PyArrow workaround here.

Looks like it would solve our problem, but I don't know if it would be better to monkey patch PyArrow (or if it can be done), or if we should create a light wrapper around it.