Skip to content

Conversation

@ueshin
Copy link
Collaborator

@ueshin ueshin commented Mar 29, 2021

Fix groupby-apply and transform to support additional dtypes.

After this, additional dtypes can be specified in the return type annotation of the UDFs for groupby-apply and transform.

>>> kdf = ks.DataFrame(
...     {
...         "a": pd.Categorical([1, 2, 3, 1, 2, 3]),
...         "b": pd.Categorical(
...             ["b", "a", "c", "c", "b", "a"], categories=["c", "b", "d", "a"]
...         ),
...     },
... )
>>> def identity(df) -> ks.DataFrame[zip(kdf.columns, kdf.dtypes)]:
...     return df
...
>>> applied = kdf.groupby("a").apply(identity)
>>> applied
   a  b
0  2  a
1  2  b
2  3  c
3  3  a
4  1  b
5  1  c
>>> applied.dtypes
a    category
b    category
dtype: object

FYI: without the fix:

>>> applied
   a  b
0  1  3
1  1  1
2  2  0
3  2  3
4  0  1
5  0  0
>>> applied.dtypes
a    int64
b    int64
dtype: object

@ueshin ueshin requested a review from xinrong-meng March 29, 2021 22:53
return_schema,
retain_index=False,
)
# Otherwise, it loses index.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I may not understand # Otherwise, in this context. I was thinking transform would always lose the index.

class CategoricalTest(ReusedSQLTestCase, TestUtils):
def test_categorical_frame(self):
pdf = pd.DataFrame(
@property
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice refactoring!

@xinrong-meng
Copy link
Contributor

Looks great! Thank you!

@xinrong-meng xinrong-meng self-requested a review March 30, 2021 18:07
@ueshin
Copy link
Collaborator Author

ueshin commented Mar 30, 2021

Thanks! merging.

@ueshin ueshin merged commit fdda825 into databricks:master Mar 30, 2021
@ueshin ueshin deleted the groupby branch March 30, 2021 19:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants