pd.DataFrame.describe percentile string precision

[blalterman]

When using pd.DataFrame.describe, if your percentiles are different only at the 4th decimal place, a ValueError is thrown because the the percentiles that vary at the 4th decimal place become the same value.

In [1]: s = Series(np.random.randn(10))

In [2]: s.describe()
Out[2]: 
count    10.000000
mean      0.291571
std       1.057143
min      -1.453547
25%      -0.614614
50%       0.637435
75%       0.968905
max       1.823964
dtype: float64

In [3]: s.describe(percentiles=[0.0001, 0.0005, 0.001, 0.999, 0.9995, 0.9999])
Out[3]: 
count     10.000000
mean       0.291571
std        1.057143
min       -1.453547
0.0%      -1.453107
0.1%      -1.451348
0.1%      -1.449149
50%        0.637435
99.9%      1.817201
100.0%     1.820583
100.0%     1.823288
max        1.823964
dtype: float64

[jreback]

@bla1089 can you add the example which raises.

It seems like by design someone coded in to only show 1 decimal place.
def pretty_name(x): x *= 100 if x == int(x): return '%.0f%%' % x else: return '%.1f%%' % x

What would be an appropriate fix? Change decimal place if input has more decimals in place?

First timer here. Hope I'm not stepping bounds...just wanted to see how I can start contributing to the project and learn about python.

[blalterman]

You would have to determine the first significant decimal place that
provides unique string identifiers for each percentile. I'm not trained as
a programmer, but I would be surprised if there isn't a standard way to do
this.

On Tue, May 10, 2016, 09:23 teaspreader notifications@github.com wrote:

It seems like by design someone coded in to only show 1 decimal place.
def pretty_name(x):
x *= 100
if x == int(x):
return '%.0f%%' % x
else:
return '%.1f%%' % x

What would be an appropriate fix? Change decimal place if input has more
decimals in place?

First timer here. Hope I'm not stepping bounds...just wanted to see how I
can start contributing to the project and learn about python.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#13104 (comment)

Makes sense. Thanks.

[blalterman]

In fact, it might be done by taking np.ediff1d(np.log10([100_percentiles]))
or replacing 100_percentiles with the decimals that remain after taking
100*percentiles. Round that result to the left on the number line, and take
the most negative of the numbers. That would be the denial place precision
necessary to get a unique index. Though this could be way too complicated a
formulation.

On Tue, May 10, 2016, 09:45 Ben Alterman alterman.ben@gmail.com wrote:

You would have to determine the first significant decimal place that
provides unique string identifiers for each percentile. I'm not trained as
a programmer, but I would be surprised if there isn't a standard way to do
this.

On Tue, May 10, 2016, 09:23 teaspreader notifications@github.com wrote:

It seems like by design someone coded in to only show 1 decimal place.
def pretty_name(x):
x *= 100
if x == int(x):
return '%.0f%%' % x
else:
return '%.1f%%' % x

What would be an appropriate fix? Change decimal place if input has more
decimals in place?

First timer here. Hope I'm not stepping bounds...just wanted to see how I
can start contributing to the project and learn about python.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#13104 (comment)

How about just return string decimal format based on the input?

def pretty_name(x): decimal_place = str(x)[::-1].find('.')-2 x *= 100 if x == int(x): return '%.0f%%' % x else: return '%.*f%%' % (decimal_place, x)

[blalterman]

Test it. Does it work for cases in which you have a different number of
decimal places in the percentiles input? How anesthetically clean are the
results?

On Tue, May 10, 2016, 09:58 teaspreader notifications@github.com wrote:

How about just...?
def pretty_name(x):
decimal_place = str(x)[::-1].find('.')-2

x *= 100
if x == int(x):
return '%.0f%%' % x
else:

return '%.*f%%' % (decimal_place, x)

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#13104 (comment)

[srib]

I cannot seem to reproduce this issue on my machine.

>>> s.describe( percentiles=[0.0001, 0.0005, 0.001, 0.999, 0.9995, 0.9999])
count     10.000000
mean      -0.225519
std        0.841984
min       -1.910093
0.0%      -1.909577
0.1%      -1.907514
0.1%      -1.904935
50%        0.018802
99.9%      0.752442
100.0%     0.753580
100.0%     0.754491
max        0.754719
dtype: float64
>>> pd.show_version()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'show_version'
>>> pd.show_versions()

INSTALLED VERSIONS
------------------
commit: 2e975b6f39e3ab3840097ec0aa07c44ac13c4467
python: 2.7.11.final.0
python-bits: 32
OS: Linux
OS-release: 3.13.0-71-generic
machine: i686
processor: i686
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: 0.18.1+5.g2e975b6
nose: 1.3.7
pip: 8.1.1
setuptools: 20.7.0
Cython: 0.24
numpy: 1.11.0
scipy: None
statsmodels: None
xarray: None
IPython: 4.2.0
sphinx: 1.4.1
patsy: None
dateutil: 2.5.2
pytz: 2016.3
blosc: None
bottleneck: None
tables: None
numexpr: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.8
boto: None
pandas_datareader: None