Skip to content

Conversation

@HyukjinKwon
Copy link
Member

This PR adds the support for schema inference support at DataFrame.transform.

Now, DataFrame.transform can be used as below:

import databricks.koalas as ks

ks.range(10).transform(lambda x: x ** 2)

if should_infer_schema:
# Here we execute with the first 1000 to get the return type.
# If the records were less than 1000, it uses pandas API directly for a shortcut.
limit = 1000
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should add this as a configuration as well later.

@HyukjinKwon
Copy link
Member Author

cc @RainFung FYI

@softagram-bot
Copy link

Softagram Impact Report for pull/732 (head commit: 688a54a)

⭐ Change Overview

Showing the changed files, dependency changes and the impact - click for full size
(Open in Softagram Desktop for full details)

📄 Full report

Impact Report explained. Give feedback on this report to [email protected]

@HyukjinKwon HyukjinKwon requested a review from ueshin September 2, 2019 03:57
@codecov-io
Copy link

codecov-io commented Sep 2, 2019

Codecov Report

Merging #732 into master will increase coverage by 0.02%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #732      +/-   ##
==========================================
+ Coverage    94.3%   94.33%   +0.02%     
==========================================
  Files          32       32              
  Lines        5569     5580      +11     
==========================================
+ Hits         5252     5264      +12     
+ Misses        317      316       -1
Impacted Files Coverage Δ
databricks/koalas/frame.py 94.97% <100%> (+0.11%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5a6bbb9...688a54a. Read the comment docs.

@HyukjinKwon
Copy link
Member Author

Merged.

@HyukjinKwon HyukjinKwon merged commit 800f00d into databricks:master Sep 4, 2019
@HyukjinKwon HyukjinKwon deleted the transform-infer branch November 6, 2019 02:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants