API: concatting DataFrames does not skip empty objects #39035
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
so if we revert this, then we actually need to change real logic here and use the combined dtypes of the empties, which is fine. |
|
I'm generally OK with this.
If we do, we should do the same for Index setops, which has special-casing for empty indexes. |
ok fair enought. let's revert then i guess try to unify and see what breaks. |
3 tasks
|
Opened a follow-up issue for this at #39122 |
luckyvs1
pushed a commit
to luckyvs1/pandas
that referenced
this pull request
Jan 20, 2021
) * Revert "BUG: casting on concat with empties (pandas-dev#38907)" This reverts commit 04282c7. * Revert "BUG: inconsistent concat casting EA vs non-EA (pandas-dev#38843)" This reverts commit 2362df9.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See discussion in #38843
To repeat the main argument: when not skipping empty objects, the resulting dtype of a concat-operation only depends on the input dtypes, and not on the exact content (the exact values, how many values (shape)). In general we want to get rid of value-dependent behaviour. In the past we discussed this in the context of the certain values (eg presence of NaNs or not), but I think also the shape should not matter (eg when slicing dataframes before contatting, you can get empties or not depending on values).
This starts with reverting the change for DataFrame. If we opt for this behaviour, in addition we should also discuss how to change the Series behaviour.
cc @jbrockmendel