Skip to content

Read numpy nans/datetimes#741

Merged
texodus merged 10 commits intomasterfrom
dataframe-fix
Sep 25, 2019
Merged

Read numpy nans/datetimes#741
texodus merged 10 commits intomasterfrom
dataframe-fix

Conversation

@sc1f
Copy link
Contributor

@sc1f sc1f commented Sep 24, 2019

This PR improves the Python API's ability to parse Pandas DataFrames, ensuring compatibility with numpy.nan and numpy.datetime64.

  • Columns that begin with numpy.nan will now have their types inferred as float, as numpy.nan is a float type. If the column contains a value (i.e. [nan, nan, 1]) later on, and the value is not of float or integer type, then the column will be promoted to a string, all numpy.nans will be converted to None, and subsequent writes will be converted to string.
  • numpy.nan serializes to None when exported from Perspective.
  • Timestamp generation has been moved to the date parser, and numpy.datetime64 correctly returns Unix timestamps for use within the C++ engine.

@timkpaine timkpaine changed the title Correctly parse dataframes, streamline datetime processing Read numpy nans Sep 24, 2019
@timkpaine timkpaine changed the title Read numpy nans Read numpy nans/datetimes Sep 24, 2019
@codecov-io
Copy link

codecov-io commented Sep 24, 2019

Codecov Report

Merging #741 into master will increase coverage by 1.06%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #741      +/-   ##
==========================================
+ Coverage    92.3%   93.36%   +1.06%     
==========================================
  Files          61       61              
  Lines        2703     2970     +267     
==========================================
+ Hits         2495     2773     +278     
+ Misses        208      197      -11
Impacted Files Coverage Δ
perspective/table/_date_validator.py 86.84% <0%> (-0.66%) ⬇️
perspective/tests/node/test_node.py 31.25% <0%> (ø) ⬆️
perspective/table/view.py 80.95% <0%> (ø) ⬆️
perspective/tests/table/test_table_pandas.py 100% <0%> (ø) ⬆️
perspective/tests/table/test_view.py 98.55% <0%> (+0.09%) ⬆️
perspective/table/_accessor.py 92.13% <0%> (+1.99%) ⬆️
perspective/table/table.py 94.94% <0%> (+3.94%) ⬆️
perspective/tests/table/test_update.py 100% <0%> (+11.23%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9481128...159af03. Read the comment docs.

@texodus
Copy link
Member

texodus commented Sep 25, 2019

Thanks for the PR!

@texodus texodus merged commit 82155a7 into master Sep 25, 2019
@texodus texodus deleted the dataframe-fix branch September 25, 2019 19:48
@texodus texodus added the bug Concrete, reproducible bugs label Jan 3, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Concrete, reproducible bugs

Development

Successfully merging this pull request may close these issues.

5 participants