pd.DataFrame.describe percentile string precision

When using pd.DataFrame.describe, if your percentiles are different only at the 4th decimal place, a ValueError is thrown because the the percentiles that vary at the 4th decimal place become the same value.

In [1]: s = Series(np.random.randn(10))

In [2]: s.describe()
Out[2]: 
count    10.000000
mean      0.291571
std       1.057143
min      -1.453547
25%      -0.614614
50%       0.637435
75%       0.968905
max       1.823964
dtype: float64

In [3]: s.describe(percentiles=[0.0001, 0.0005, 0.001, 0.999, 0.9995, 0.9999])
Out[3]: 
count     10.000000
mean       0.291571
std        1.057143
min       -1.453547
0.0%      -1.453107
0.1%      -1.451348
0.1%      -1.449149
50%        0.637435
99.9%      1.817201
100.0%     1.820583
100.0%     1.823288
max        1.823964
dtype: float64

@bla1089 can you add the example which raises.

You would have to determine the first significant decimal place that
provides unique string identifiers for each percentile. I'm not trained as
a programmer, but I would be surprised if there isn't a standard way to do
this.

On Tue, May 10, 2016, 09:23 teaspreader notifications@github.com wrote:

It seems like by design someone coded in to only show 1 decimal place.
def pretty_name(x):
x *= 100
if x == int(x):
return '%.0f%%' % x
else:
return '%.1f%%' % x

What would be an appropriate fix? Change decimal place if input has more
decimals in place?

First timer here. Hope I'm not stepping bounds...just wanted to see how I
can start contributing to the project and learn about python.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#13104 (comment)

In fact, it might be done by taking np.ediff1d(np.log10([100_percentiles]))
or replacing 100_percentiles with the decimals that remain after taking
100*percentiles. Round that result to the left on the number line, and take
the most negative of the numbers. That would be the denial place precision
necessary to get a unique index. Though this could be way too complicated a
formulation.

On Tue, May 10, 2016, 09:45 Ben Alterman alterman.ben@gmail.com wrote:

You would have to determine the first significant decimal place that
provides unique string identifiers for each percentile. I'm not trained as
a programmer, but I would be surprised if there isn't a standard way to do
this.

On Tue, May 10, 2016, 09:23 teaspreader notifications@github.com wrote:

It seems like by design someone coded in to only show 1 decimal place.
def pretty_name(x):
x *= 100
if x == int(x):
return '%.0f%%' % x
else:
return '%.1f%%' % x

What would be an appropriate fix? Change decimal place if input has more
decimals in place?

First timer here. Hope I'm not stepping bounds...just wanted to see how I
can start contributing to the project and learn about python.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#13104 (comment)

Test it. Does it work for cases in which you have a different number of
decimal places in the percentiles input? How anesthetically clean are the
results?

On Tue, May 10, 2016, 09:58 teaspreader notifications@github.com wrote:

How about just...?
def pretty_name(x):
decimal_place = str(x)[::-1].find('.')-2

x *= 100
if x == int(x):
return '%.0f%%' % x
else:

return '%.*f%%' % (decimal_place, x)

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#13104 (comment)

I cannot seem to reproduce this issue on my machine.

>>> s.describe( percentiles=[0.0001, 0.0005, 0.001, 0.999, 0.9995, 0.9999])
count     10.000000
mean      -0.225519
std        0.841984
min       -1.910093
0.0%      -1.909577
0.1%      -1.907514
0.1%      -1.904935
50%        0.018802
99.9%      0.752442
100.0%     0.753580
100.0%     0.754491
max        0.754719
dtype: float64
>>> pd.show_version()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'show_version'
>>> pd.show_versions()

INSTALLED VERSIONS
------------------
commit: 2e975b6f39e3ab3840097ec0aa07c44ac13c4467
python: 2.7.11.final.0
python-bits: 32
OS: Linux
OS-release: 3.13.0-71-generic
machine: i686
processor: i686
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: 0.18.1+5.g2e975b6
nose: 1.3.7
pip: 8.1.1
setuptools: 20.7.0
Cython: 0.24
numpy: 1.11.0
scipy: None
statsmodels: None
xarray: None
IPython: 4.2.0
sphinx: 1.4.1
patsy: None
dateutil: 2.5.2
pytz: 2016.3
blosc: None
bottleneck: None
tables: None
numexpr: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.8
boto: None
pandas_datareader: None

I'm actually not in favor of the warning because (1) it is only going to be
raised once; (2) we don't know how the user will use the percentiles (e.g.do
they iterate through them in the describe output?); and (3) it permits
sloppy coding that could cause someone problems later.

If you want the ability for duplicate percentiles, then how about a kwarg
like warn_on_dupe so that the default is an error and someone has to
explicitly request duplicates?

I was only proposing the warning for completeness. That being said, the
percentiles should be sorted so that the index is monotonic.

On Tue, May 24, 2016, 11:11 Jeff Reback notifications@github.com wrote:

sorting is fine. I am not really in favor of duplicates. yes they are
handled generally, but not particularly useful IMHO.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#13104 (comment)

Yes, this sounds reasonable. Exceptions then.

Slightly off-topic: if I keep finding other bugs and can fix them, should I open a new issue to discuss the code changes I propose? Or is it enough to discuss it in a pull request comments?

As an illustration, another bug with describe, this time unrelated to percentiles:

df = pd.DataFrame({'A': list("BCDE"), 0: [1,2,3,4]})
df.describe()
# long traceback listing several internal functions
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'

I've found the source of the error and have a seemingly reasonable solution (it still needs testing, though).

This seems reasonably distinct and should probably be handled on its own.

On Wed, May 25, 2016, 10:06 pijucha notifications@github.com wrote:

Yes, this sounds reasonable. Exceptions then.

Slightly off-topic: if I keep finding other bugs and can fix them, should
I open a new issue to discuss the code changes I propose? Or is it enough
to discuss it in a pull request comments?

As an illustration, another bug with describe, this time unrelated to
percentiles:

df = pd.DataFrame({'A': list("BCDE"), 0: [1,2,3,4]})
df.describe()# long traceback listing several internal functionsValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'

I've found the source of the error and have a seemingly reasonable
solution (it still needs testing, though).

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#13104 (comment)

Yes it's distinct and I don't want to discuss it here. But I was going to fix it in the same commit because the code changes will probably be contained within the function describe.

My question was essentially: does any code change require opening a new issue? Wiki says not necessarily but mentions only insignificant changes.

generally we like to have issues to go with PR's.

Ok, thanks. I'll open a new one then.

BUG: Fix describe(): percentiles (#13104), col index (#13288)

Navigation Menu

Repository navigation

pd.DataFrame.describe percentile string precision #13104

22 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

pd.DataFrame.describe percentile string precision #13104

Description

Activity

jreback commented on May 6, 2016

blalterman commented on May 10, 2016

blalterman commented on May 10, 2016

blalterman commented on May 10, 2016

srib commented on May 10, 2016

22 remaining items

blalterman commented on May 24, 2016

pijucha commented on May 25, 2016

blalterman commented on May 25, 2016

pijucha commented on May 25, 2016

jreback commented on May 25, 2016

pijucha commented on May 25, 2016

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions