Skip to content

Conversation

@ueshin
Copy link
Collaborator

@ueshin ueshin commented Apr 7, 2020

I have sometimes seen misuse of a keyword argument options in read_xxx/to_xxx functions. E.g.,

kdf = ks.read_csv(..., options={ ... })

In this case, the argument options is actually {'options': { ... }}, which is not what the user wants to do.

We can help those cases by getting the 'options' value.

@ueshin ueshin requested a review from HyukjinKwon April 7, 2020 21:32
@itholic
Copy link
Contributor

itholic commented Apr 7, 2020

This way is not currently supported in pandas, right?

>>> pd.read_csv('test.csv', options={'engine': None})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: parser_f() got an unexpected keyword argument 'options'

But It's LGTM though since I think It will not not harm the experience of the existing pandas users.

@HyukjinKwon
Copy link
Member

It's not supported in pandas because options here means PySpark specific options.

@HyukjinKwon HyukjinKwon merged commit fab9f6f into databricks:master Apr 8, 2020
@ueshin ueshin deleted the options branch April 8, 2020 02:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants