Skip to content

Dataframe/Numpy Array Loader#791

Merged
texodus merged 10 commits intomasterfrom
numpy-fast-working
Nov 4, 2019
Merged

Dataframe/Numpy Array Loader#791
texodus merged 10 commits intomasterfrom
numpy-fast-working

Conversation

@sc1f
Copy link
Contributor

@sc1f sc1f commented Nov 4, 2019

This PR adds a separate load path for Pandas dataframe and numpy arrays (dictionaries of numpy arrays, numpy structured arrays, and numpy record arrays.

  • It will try to copy all arrays of integral/float/bool type, and fill other types through iteration.
  • Arrays of dtype=object will always be iteratively filled using the marshal method, which coerces data into the correct format.
  • Support for mixed dictionaries of numpy arrays and lists has been deprecated: datasets should be fully composed of numpy.ndarray or list, but not both.
  • Adds support for numpy structured arrays.
  • Adds and fully tests support for all numpy datetime types from nanosecond (datetime64[ns]) to year (datetime64[Y]).
  • Adds support for loading timestamps (in seconds or milliseconds since epoch) into columns already typed as datetime. To load timestamps as datetimes, either construct the table with a schema of {"name": datetime}, or with a valid datetime/Timestamp/datetime64 object.
  • Adds and tests support for timedelta64 arrays, which are cast into string to preserve their units.
  • Adds and tests support for pandas.DateTimeIndex and pandas.PeriodIndex.

@sc1f sc1f force-pushed the numpy-fast-working branch from 45272ae to 82b0fc4 Compare November 4, 2019 17:29
Copy link
Member

@texodus texodus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@texodus
Copy link
Member

texodus commented Nov 4, 2019

Thanks for the PR!

@texodus texodus merged commit 83a67a7 into master Nov 4, 2019
@texodus texodus deleted the numpy-fast-working branch November 4, 2019 20:46
@texodus texodus added enhancement Feature requests or improvements Python labels Jan 3, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement Feature requests or improvements Python

Development

Successfully merging this pull request may close these issues.

4 participants