ARROW-3903: [Python] Random array generator for Arrow conversion and Parquet testing #3301
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kszucs
commented
Jan 30, 2019
| if isinstance(type, st.SearchStrategy): | ||
| type = draw(type) | ||
|
|
||
| # TODO(kszucs): remove it, field metadata is not kept |
There was a problem hiding this comment.
The type equality check fails at https://github.com/apache/arrow/blob/master/python/pyarrow/table.pxi#L297
We should probably use .equals(check_metadata=False) and find out why the two metadata are different.
I didn't file a jira issue because I couldn't create a reproducible example - the metadata is not displayed. However commenting out the assume line reproduces the issue.
xhochy
approved these changes
Feb 1, 2019
| @@ -1155,9 +1155,9 @@ cdef class Table(_PandasConvertible): | |||
|
|
|||
| Parameters | |||
| ---------- | |||
| arrays: list of pyarrow.Array or pyarrow.Column | |||
| arrays : list of pyarrow.Array or pyarrow.Column | |||
There was a problem hiding this comment.
This is actually not needed in the latest numpydoc spec. But for docs improvement, we could probably build one day on pandas' work: pandas-dev/pandas#22408
| @@ -32,6 +34,7 @@ | |||
| pickle5 = None | |||
|
|
|||
| import pyarrow as pa | |||
| import pyarrow.tests.strategies as past | |||
xhochy
pushed a commit
that referenced
this pull request
Feb 8, 2019
…Parquet testing Generate random schemas, arrays, chunked_arrays, columns, record_batches and tables. Slow, but makes quiet easy to isolate corner cases (already created jira issues). In follow up PRs We should use these strategies to increase the coverage. It'll enable us to reduce the issues, We could even use it for generate benchmark datasets periodically (only if We persist somewhere). Example usage: Run 10 samples (dev profile): `pytest -sv pyarrow/tests/test_strategies.py::test_tables --enable-hypothesis --hypothesis-show-statistics --hypothesis-profile=dev` Print the generated examples (debug): `pytest -sv pyarrow/tests/test_strategies.py::test_schemas --enable-hypothesis --hypothesis-show-statistics --hypothesis-profile=debug` Author: Krisztián Szűcs <[email protected]> Closes #3301 from kszucs/ARROW-3903 and squashes the following commits: ff6654c <Krisztián Szűcs> finalize 8b5e7ea <Krisztián Szűcs> rat 61fe01d <Krisztián Szűcs> strategies for chunked_arrays, columns, record batches; test the strategies themselves bdb63df <Krisztián Szűcs> hypothesis array strategy
trxcllnt
pushed a commit
to trxcllnt/arrow
that referenced
this pull request
Feb 12, 2019
…Parquet testing Generate random schemas, arrays, chunked_arrays, columns, record_batches and tables. Slow, but makes quiet easy to isolate corner cases (already created jira issues). In follow up PRs We should use these strategies to increase the coverage. It'll enable us to reduce the issues, We could even use it for generate benchmark datasets periodically (only if We persist somewhere). Example usage: Run 10 samples (dev profile): `pytest -sv pyarrow/tests/test_strategies.py::test_tables --enable-hypothesis --hypothesis-show-statistics --hypothesis-profile=dev` Print the generated examples (debug): `pytest -sv pyarrow/tests/test_strategies.py::test_schemas --enable-hypothesis --hypothesis-show-statistics --hypothesis-profile=debug` Author: Krisztián Szűcs <[email protected]> Closes apache#3301 from kszucs/ARROW-3903 and squashes the following commits: ff6654c <Krisztián Szűcs> finalize 8b5e7ea <Krisztián Szűcs> rat 61fe01d <Krisztián Szűcs> strategies for chunked_arrays, columns, record batches; test the strategies themselves bdb63df <Krisztián Szűcs> hypothesis array strategy
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Generate random schemas, arrays, chunked_arrays, columns, record_batches and tables.
Slow, but makes quiet easy to isolate corner cases (already created jira issues). In follow up PRs We should use these strategies to increase the coverage. It'll enable us to reduce the issues, We could even use it for generate benchmark datasets periodically (only if We persist somewhere).
Example usage:
Run 10 samples (dev profile):
pytest -sv pyarrow/tests/test_strategies.py::test_tables --enable-hypothesis --hypothesis-show-statistics --hypothesis-profile=devPrint the generated examples (debug):
pytest -sv pyarrow/tests/test_strategies.py::test_schemas --enable-hypothesis --hypothesis-show-statistics --hypothesis-profile=debug