Skip to content

speed up the repr for big MultiIndex objects #4846

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dcherian merged 6 commits into from
Jan 29, 2021
Merged

speed up the repr for big MultiIndex objects #4846

Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
9e3ebf2
print the repr of a multiindex using only a subset of the coordinate …
keewis Jan 26, 2021
1138487
don't index if we have less items than available width
keewis Jan 26, 2021
5e719b1
don't try to shorten arrays which are way too short
keewis Jan 26, 2021
f061dc8
col_width seems to be the maximum number of elements, not characters
keewis Jan 26, 2021
e04152f
add a asv benchmark
keewis Jan 26, 2021
8f20f1f
Apply suggestions from code review
keewis Jan 29, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions xarray/core/formatting.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -300,11 +300,18 @@ def _summarize_coord_multiindex(coord, col_width, marker):


def _summarize_coord_levels(coord, col_width, marker="-"):
if col_width < len(coord):
n_values = col_width // 4
indices = list(range(0, n_values)) + list(range(-n_values, 0))
subset = coord[indices]
else:
subset = coord
Copy link
Collaborator Author

@keewis keewis Jan 26, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this could probably use some optimization: how big does the MultiIndex have to be so indexing+get_level_variable is faster than just get_level_variable?

Copy link
Collaborator

@max-sixty max-sixty Jan 26, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, though it's so fast already relative to how often repr is called...

Could also defer to pandas, which seem to do this (though a different orientation)

More than fine to leave as a TODO imo

Copy link
Collaborator Author

@keewis keewis Jan 26, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assumed that for everything below 100 elements get_level_variable is faster than indexing first, which also means that I don't have to worry about the case where the index does not have enough elements to be truncated.

Could also defer to pandas

how would I do that?

Copy link
Collaborator

@max-sixty max-sixty Jan 27, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how would I do that?

I had meant — they have a repr for multiindex which is fast — so could we use theirs somehow, despite the different orientation. On reflection — our code is simple, I agree with your impulse.


return "\n".join(
summarize_variable(
lname, coord.get_level_variable(lname), col_width, marker=marker
lname, subset.get_level_variable(lname), col_width, marker=marker
)
for lname in coord.level_names
for lname in subset.level_names
)


Expand Down