BUG: rank with +-inf, #6945 #17903

peterpanmj · 2017-10-17T11:00:01Z

The bug is caused by replacing nan with positive infinite or negative infinite values. As a result, when keep_na is true, both real nan and positive/negative infinite will be assigned nan as the rank. The solution is to keep track of the real nan value when generate the nan mask.

closes Series/DataFrame.rank() doesn't handle np.inf properly #6945
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

jreback · 2017-10-17T11:07:54Z

tests!

codecov · 2017-10-17T11:58:40Z

Codecov Report

Merging #17903 into master will increase coverage by 0.11%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #17903      +/-   ##
==========================================
+ Coverage   91.46%   91.58%   +0.11%     
==========================================
  Files         157      155       -2     
  Lines       51439    51255     -184     
==========================================
- Hits        47051    46941     -110     
+ Misses       4388     4314      -74

Flag	Coverage Δ
#multiple	`89.44% <ø> (+0.12%)`	⬆️
#single	`40.67% <ø> (-0.05%)`	⬇️

Impacted Files	Coverage Δ
pandas/io/gbq.py	`25% <0%> (-58.34%)`	⬇️
pandas/util/testing.py	`82.01% <0%> (-0.17%)`	⬇️
pandas/core/frame.py	`97.81% <0%> (-0.1%)`	⬇️
pandas/io/json/normalize.py	`96.87% <0%> (-0.07%)`	⬇️
pandas/core/tools/datetimes.py	`84.43% <0%> (-0.06%)`	⬇️
pandas/io/parsers.py	`95.55% <0%> (-0.05%)`	⬇️
pandas/core/internals.py	`94.44% <0%> (-0.04%)`	⬇️
pandas/core/indexes/datetimes.py	`95.68% <0%> (-0.01%)`	⬇️
pandas/compat/pickle_compat.py	`75.6% <0%> (ø)`	⬆️
pandas/core/sparse/api.py	`100% <0%> (ø)`	⬆️
... and 9 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0e16818...987fb9d. Read the comment docs.

jreback · 2017-10-20T13:32:18Z

pandas/tests/series/test_rank.py

@@ -196,9 +196,6 @@ def test_rank_signature(self):
        pytest.raises(ValueError, s.rank, 'average')

    def test_rank_inf(self):
-        pytest.skip('DataFrame.rank does not currently rank '


can you add some nans in here as well. is this enough of a test?

Yes, I can add nans there.

this is going to need a battery of tests for types accepting nan
iow float32/float64

also test non nan types like uint64 (yes they cannot have nan but may not be tested fully); since u r adding paths that exclude these need to beef up testing a bit

jreback · 2017-10-20T13:34:27Z

pandas/_libs/algos_rank_helper.pxi.in

@@ -44,6 +44,10 @@ def rank_1d_{{dtype}}(object in_arr, ties_method='average', ascending=True,

        {{if dtype == 'object'}}
        ndarray sorted_data, values


you can put the sorted_nanmask for all dtypes

Yes , you are right. I should add for all dtypes.

jreback · 2017-10-20T13:34:52Z

pandas/_libs/algos_rank_helper.pxi.in

@@ -54,7 +58,7 @@ def rank_1d_{{dtype}}(object in_arr, ties_method='average', ascending=True,
        {{if dtype == 'uint64'}}
        {{ctype}} val
        {{else}}
-        {{ctype}} val, nan_value
+        {{ctype}} val, nan_value, isnan


isnan is a very confusing name here as this is an actual value

should I just call it masked ?

actually if you fix as below this is ok

but isnan should be same type as sorted_mask and mask iow uint_8

jreback · 2017-10-20T13:35:36Z

pandas/_libs/algos_rank_helper.pxi.in

+    # need to distinguish between pos/neg nan and real nan when keep_na is true
+    {{if dtype != 'uint64'}}
+    sorted_namask = mask.take(_as)
+    sorted_namask = sorted_namask.astype(np.bool)


this is a very odd thing to do, are you just computing the mask?

No, I am sorting the mask to match the sorted_data. When I loop through the sorted_data when computing ranks, I need to find the corresponding indicator for nan value. The current version is comparing the each element against nan_value, which is causing the problem

if (val is nan_value) and keep_na: ranks[argsorted[i]] = nan

Because nan_value is either pos or neg infinite value.

if ascending ^ (na_option == 'top'): nan_value = {{pos_nan_value}} else: nan_value = {{neg_nan_value}}

As a result when the underlying value is truely +/- infinite, the rank will be just nan for both. That's why ranking with inf values are failing when keep_na is true. Can you elaborate more on why this is odd ? Do you mean the sorted_namask.astype(np.bool) is redundant ?

call this sorted_mask to be consistent

you should need the astype at all; mask is already a uint_8 (which in cython is bool)

jreback

can you add a note for 0.22 in bug fixes / conversion

jreback · 2017-10-22T15:05:33Z

pandas/_libs/algos_rank_helper.pxi.in

+    # need to distinguish between pos/neg nan and real nan when keep_na is true
+    {{if dtype != 'uint64'}}
+    sorted_namask = mask.take(_as)
+    sorted_namask = sorted_namask.astype(np.bool)


call this sorted_mask to be consistent

you should need the astype at all; mask is already a uint_8 (which in cython is bool)

jreback · 2017-10-22T15:06:52Z

pandas/_libs/algos_rank_helper.pxi.in

@@ -54,7 +58,7 @@ def rank_1d_{{dtype}}(object in_arr, ties_method='average', ascending=True,
        {{if dtype == 'uint64'}}
        {{ctype}} val
        {{else}}
-        {{ctype}} val, nan_value
+        {{ctype}} val, nan_value, isnan


actually if you fix as below this is ok

but isnan should be same type as sorted_mask and mask iow uint_8

jreback · 2017-10-22T15:09:01Z

pandas/tests/series/test_rank.py

@@ -196,9 +196,6 @@ def test_rank_signature(self):
        pytest.raises(ValueError, s.rank, 'average')

    def test_rank_inf(self):
-        pytest.skip('DataFrame.rank does not currently rank '


this is going to need a battery of tests for types accepting nan
iow float32/float64

also test non nan types like uint64 (yes they cannot have nan but may not be tested fully); since u r adding paths that exclude these need to beef up testing a bit

jreback · 2017-11-07T18:03:14Z

pandas/_libs/algos_rank_helper.pxi.in

    {{endif}}

    n = len(values)
    ranks = np.empty(n, dtype='f8')

    {{if dtype == 'object'}}
    try:
-        _as = values.argsort()
+        # lexsort on object array will raise TypeError for numpy version 
+        # earlier than 1.11.0


why are you adding all of this?

I want to keep nan values at the every end or at the every beginning. Otherwise sum_ranks and dups will be messed up when real +/- infs are intertwined with replaced infs (originally Nans )

if tiebreak == TIEBREAK_AVERAGE: for j in range(i - dups + 1, i + 1): ranks[argsorted[j]] = sum_ranks / dups

It is hard to let j skip the index corresponding to nan value, and to subtract extra counts from sum_ranks and dups at the same time. So I try to separate nans when I calculate _as.
For float and int dtype, it is easy to achieve this by a double sort.

_as = np.lexsort(keys=(mask, values))

But numpy does not support lexsort on object array before version 1.11.0.
numpy/numpy#6312
Then I tried argsort with an order argument

_dt = [('values', 'object'), ('mask', 'bool')] _values = np.asarray(list(zip(mask, values)), dtype=_dt) _as = np.argsort(_values, kind='mergesort', order=('values', 'mask'))

Somehow, it is not working properly on Infinity, NegInfinity object. As a result, I sort it by mask first. The downside is I need to specify the condition on whether to reverse the sign of mask so as to control whether put Nans at the beginning or the end.

ok can you make this a try/except/else to isolate the thing which is actually raising

Infinity and NegInfinity might work if you add missing methods (might be a comparison operator eg dunder lt)

Do you mean nesting the try/except clause?

{{if dtype == 'object'}} try: # lexsort on object array will raise TypeError for numpy version # earlier than 1.11.0 _as = np.lexsort(keys=(mask, values)) except TypeError: try: _dt = [('values', 'O'), ('mask', '?')] _values = np.asarray(list(zip(values, mask)), dtype=_dt) _as = np.argsort(_values, kind='mergesort', order=('values', 'mask')) except TypeError: if not retry: raise valid_locs = (~mask).nonzero()[0] ranks.put(valid_locs, rank_1d_object(values.take(valid_locs), 0, ties_method=ties_method, ascending=ascending)) np.putmask(ranks, mask, np.nan) return ranks {{else}}

no I mean try/except/else. you should put minimal amounts in the try portion; else is evaluation if no exception is raised. nested try/excepts are way harder to read.

Somehow, it is not working properly on Infinity, NegInfinity object. As a result, I sort it by mask first. The downside is I need to specify the condition on whether to reverse the sign of mask so as to control whether put Nans at the beginning or the end.

looks like you fixed Infinity, doesn't this original strategy work (and is across versions)?

Sorry, I still don't know how to do it. (I get it , what try/except/else does)
When I am trying,

_as = np.lexsort(keys=(mask, values))

I will be expecting two kinds of errors when sorting an object array. One is due to lexsort is buggy in certain numpy versions, which can be fixed by argsort with order argument. Another is due to incomparable types in the array like comparing a str with an int. But they all raise same TypeError. I don't know how to isolate them.
Moreover, if I use argsort with order argument to handle the version issue, it still might fail due to incomparable types within the array and raise another TypeError
What is the Pythonic way to fix it other they nested try/except ?

jreback · 2017-11-07T18:04:57Z

pandas/tests/series/test_rank.py

+            [np.iinfo(np.uint8).min, 1, 2, 100, np.iinfo(np.uint8).max],
+            # -inf is eqivatlent to na value for int64 dtype becasue iNaT is
+            # the same as np.iinfo(np.int64).min. The expected rank for iNaT is
+            # nan if keep_na is true. Pending GH issue #16674


you need to make this the correct result and just xfail it pending that issue

jreback · 2017-11-07T18:05:13Z

pandas/tests/series/test_rank.py

+             1e10, np.iinfo(np.int64).max],
+            [NegInfinity(), '1', 'A', 'BA', 'Ba', 'C', Infinity()]
+        )
+        dtypes = ('float64', 'float32', 'uint8', 'int64', 'object')


parametrize this test

jreback · 2017-11-09T13:00:13Z

pandas/_libs/algos_rank_helper.pxi.in

    {{endif}}

    n = len(values)
    ranks = np.empty(n, dtype='f8')

    {{if dtype == 'object'}}
    try:
-        _as = values.argsort()
+        # lexsort on object array will raise TypeError for numpy version 
+        # earlier than 1.11.0


no I mean try/except/else. you should put minimal amounts in the try portion; else is evaluation if no exception is raised. nested try/excepts are way harder to read.

jreback · 2017-11-09T13:02:41Z

pandas/_libs/algos_rank_helper.pxi.in

    {{endif}}

    n = len(values)
    ranks = np.empty(n, dtype='f8')

    {{if dtype == 'object'}}
    try:
-        _as = values.argsort()
+        # lexsort on object array will raise TypeError for numpy version 
+        # earlier than 1.11.0


Somehow, it is not working properly on Infinity, NegInfinity object. As a result, I sort it by mask first. The downside is I need to specify the condition on whether to reverse the sign of mask so as to control whether put Nans at the beginning or the end.

looks like you fixed Infinity, doesn't this original strategy work (and is across versions)?

jreback · 2017-11-09T13:03:43Z

pandas/_libs/algos.pyx

-    __eq__ = lambda self, other: self is other
-    __ne__ = lambda self, other: self is not other
-    __gt__ = lambda self, other: self is not other
+    __le__ = lambda self, other: isinstance(other, Infinity)


can you add some tests for Infinity/NegInfitiy (some in test_algos already). esp how they react with NaN

Yes, I can add some tests.
Is it necessary to take the NaN into consideration, since Infinity/NegInfitiy is used to replace all NaN values ? There are no NaNs when comparing elements against each other.

np.putmask(values, mask, nan_value)

(nan_value is eiterh Infinity or NegInfitiy)

pep8speaks · 2017-11-10T08:54:25Z

Hello @peterpanmj! Thanks for updating the PR.

Cheers ! There are no PEP8 issues in this Pull Request. 🍻

Comment last updated on December 04, 2017 at 15:42 Hours UTC

jreback · 2017-11-12T15:01:53Z

pandas/tests/series/test_rank.py

+        values = np.array(contents, dtype=dtype)
+        nan_indices = np.random.choice(range(len(values)), 3)
+        exp_order = np.array(range(len(values)), dtype='float64') + 1.0
+        if dtype in dtype_na_map:


put a comment why you are doing this.

jreback · 2017-11-12T15:04:07Z

pandas/tests/test_algos.py

+    assert libalgos.NegInfinity() == libalgos.NegInfinity()
+    assert not libalgos.NegInfinity() != libalgos.NegInfinity()
+
+    assert Inf > np.nan and Inf >= np.nan and Inf != np.nan


put the nan comparison in a separate test / section (and comment that you are testing vs nan).

These should be consistent, where nothing matches nan.

In [1]: np.inf > np.nan Out[1]: False In [2]: np.inf < np.nan Out[2]: False In [3]: np.inf == np.nan Out[3]: False In [4]: np.inf != np.nan Out[4]: True

jreback · 2017-11-12T15:05:36Z

pandas/_libs/algos_rank_helper.pxi.in

+            _dt = [('values', 'O'), ('mask', '?')]
+            _values = np.asarray(list(zip(values, mask)), dtype=_dt)
+            _as = np.argsort(_values, kind='mergesort', order=('values', 'mask'))
+        except TypeError:


liberally put some comments on what is going on here.

what is retry?

jreback

lgtm. small changes. can you add note in the bug fix numeric section. if you think its worth a sub-section ok with that as well (as this is a numerically impactful change and should be highlited a bit).

jreback · 2017-11-22T02:24:47Z

pandas/_libs/algos_rank_helper.pxi.in

@@ -90,56 +86,69 @@ def rank_1d_{{dtype}}(object in_arr, ties_method='average', ascending=True,
    mask = values == iNaT
    {{endif}}

+    # doulbe sort first by mask and then by values to ensure nan values are


jreback · 2017-11-22T02:25:00Z

pandas/_libs/algos_rank_helper.pxi.in

@@ -27,7 +27,7 @@ dtypes = [('object', 'object', 'Infinity()', 'NegInfinity()'),
 {{if dtype == 'object'}}


-def rank_1d_{{dtype}}(object in_arr, bint retry=1, ties_method='average',


very odd that we had this arg here in the first place

lgtm. small changes. can you add note in the bug fix numeric section. if you think its worth a sub-section ok with that as well (as this is a numerically impactful change and should be highlited a bit).

Btw, where should I put the note ? Which file ?

v0.22.0.txt, add a small sub-section in other enhancements with a mini-example.

jreback · 2017-11-23T15:41:51Z

rebase as well as the missing value support was moved around a bit (e.g. changes to infinity w/o nan)

jreback

merge in master, you are picking up other changes as well.

jreback · 2017-11-26T15:22:17Z

doc/source/whatsnew/v0.22.0.txt

- Better support for ``Dataframe.style.to_excel()`` output with the ``xlsxwriter`` engine. (:issue:`16149`)
-
-
+``Series.rank`` ``DataFrame.rank`` now can handle ``inf`` values and missing values properly


add a ref above this, you can just say
.rank will now handle inf values when NaN are present (or somethign like this)

jreback · 2017-11-26T15:22:36Z

doc/source/whatsnew/v0.22.0.txt

+
+.. code-block:: ipython
+
+    In [17]: pd.Series([-np.inf, 0, 1, np.inf]).rank()


add in a NaN in your example

jreback · 2017-11-26T15:22:48Z

doc/source/whatsnew/v0.22.0.txt

+    3    4.0
+    dtype: float64
+
+Furthermore, in previous versions, missing values were not distinguished from infinit values in the calculation.


infinit -> infinity

jreback · 2017-11-26T15:23:28Z

doc/source/whatsnew/v0.22.0.txt

+.. code-block:: ipython
+
+    In [3]:  pd.Series([None, None, None, 'A', 'B']).rank(na_option='top')
+    Out[3]:


this doesn't need an example (for the object dtype)

jreback · 2017-11-26T15:23:49Z

doc/source/whatsnew/v0.22.0.txt

+
+.. code-block:: ipython
+
+    In [15]: pd.Series([np.nan, np.nan, -np.inf, -np.inf]).rank(na_option='top')


would think about combining this with the above example (ok if you can't / better like this)

…with NaNs (pandas-dev#6945)

jreback

looking good! some doc changes. can you also verify that we have asv's (perf) for this case? if not can you add (and show the results); I expect it to be slightly worse, but that's ok, just want to quantify it.

jreback · 2017-12-03T16:27:03Z

doc/source/whatsnew/v0.22.0.txt

@@ -76,6 +76,67 @@ Other Enhancements
 - Improved wording of ``ValueError`` raised in :func:`to_datetime` when ``unit=`` is passed with a non-convertible value (:issue:`14350`)
 - :func:`Series.fillna` now accepts a Series or a dict as a ``value`` for a categorical dtype (:issue:`17033`)
 - :func:`pandas.read_clipboard` updated to use qtpy, falling back to PyQt5 and then PyQt4, adding compatibility with Python3 and multiple python-qt bindings (:issue:`17722`)
+- :func:`Series.rank` and :func:`DataFrame.rank` now can handle ``inf`` values properly when ``NaN`` are present (:issue:`6945`)
+
+handle ``inf`` values properly when ``NaN`` are present


move this above Other Enhancements, add a ref as well, e.g. something like

.. _whatsnew_0220.enhancements.rank_inf:

jreback · 2017-12-03T16:27:25Z

doc/source/whatsnew/v0.22.0.txt

+handle ``inf`` values properly when ``NaN`` are present
+"""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+In previous versions, ``inf`` elements were assigned ``NaN`` as their ranks. Now ranks are calculated properly.


add the issue number here

jreback · 2017-12-03T16:27:36Z

doc/source/whatsnew/v0.22.0.txt

@@ -76,6 +76,67 @@ Other Enhancements
 - Improved wording of ``ValueError`` raised in :func:`to_datetime` when ``unit=`` is passed with a non-convertible value (:issue:`14350`)
 - :func:`Series.fillna` now accepts a Series or a dict as a ``value`` for a categorical dtype (:issue:`17033`)
 - :func:`pandas.read_clipboard` updated to use qtpy, falling back to PyQt5 and then PyQt4, adding compatibility with Python3 and multiple python-qt bindings (:issue:`17722`)
+- :func:`Series.rank` and :func:`DataFrame.rank` now can handle ``inf`` values properly when ``NaN`` are present (:issue:`6945`)


this is not needed (as the new section covers)

jreback · 2017-12-03T16:28:12Z

doc/source/whatsnew/v0.22.0.txt

+
+Current Behavior
+
+.. ipython:: python


when you use the ipython directive, it executes the code, so you just need to

s.rank()

jreback · 2017-12-03T16:28:46Z

doc/source/whatsnew/v0.22.0.txt

+In previous versions, ``inf`` elements were assigned ``NaN`` as their ranks. Now ranks are calculated properly.
+
+Previous Behavior:
+


before previous behavior, use an ipython directive to define and how s

.. ipython:: python s = pd.Series([.....]) s

jreback · 2017-12-03T16:28:57Z

doc/source/whatsnew/v0.22.0.txt

+
+.. ipython:: python
+
+    In [2]: import numpy as np


impot not needed

jreback · 2017-12-03T16:29:37Z

doc/source/whatsnew/v0.22.0.txt

+
+.. code-block:: ipython
+
+    In [6]: pd.Series([-np.inf, 0, 1, np.nan, np.inf]).rank()


s.rank() (as its now defined above), but you actually should show the execution adn result of this code (e.g. just copy-past from an ipython session using 0.21.0)

jreback · 2017-12-03T16:30:04Z

doc/source/whatsnew/v0.22.0.txt

+
+Furthermore, previously if you rank ``inf`` or ``-inf`` values together with ``NaN`` values, the calculation won't distinguish ``NaN`` from infinity when using 'top' or 'bottom' argument.
+
+Previously Behavior:


same as above, define and show s

jreback · 2017-12-03T16:30:18Z

doc/source/whatsnew/v0.22.0.txt

@@ -227,7 +288,7 @@ Reshaping
 Numeric
 ^^^^^^^

-
+- Bug in :func:`Series.rank` and :func:`DataFrame.rank` could not properly rank infinity values when ``NaN`` values are present (:issue:`6945`)


you can remove this as well (covered by above)

jreback · 2017-12-06T11:29:34Z

thanks @peterpanmj ! nice patch & very responsive!

jreback changed the title ~~BUG: GH6945~~ BUG: rank with +-inf, #6945 Oct 17, 2017

peterpanmj force-pushed the bug-6945 branch from 4dd6080 to 3ec0442 Compare October 20, 2017 11:30

jreback requested changes Oct 20, 2017

View reviewed changes

jreback added Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate Numeric Operations Arithmetic, Comparison, and Logical operations Reshaping Concat, Merge/Join, Stack/Unstack, Explode labels Oct 20, 2017

jreback requested changes Oct 28, 2017

View reviewed changes

jreback added the Bug label Oct 28, 2017

peterpanmj force-pushed the bug-6945 branch from 18f4fc9 to bf6453d Compare November 6, 2017 03:13

jreback requested changes Nov 7, 2017

View reviewed changes

jreback requested changes Nov 9, 2017

View reviewed changes

jreback requested changes Nov 12, 2017

View reviewed changes

jreback requested changes Nov 22, 2017

View reviewed changes

jreback requested changes Nov 26, 2017

View reviewed changes

BUG: fix issues on rank calculations when +/- inf values are present …

c3c92af

…with NaNs (pandas-dev#6945)

peterpanmj force-pushed the bug-6945 branch from 887c5ae to c3c92af Compare December 3, 2017 13:37

jreback requested changes Dec 3, 2017

View reviewed changes

change whatsnew, make rank_inf a seperate session

987fb9d

jreback added this to the 0.22.0 milestone Dec 6, 2017

jreback approved these changes Dec 6, 2017

View reviewed changes

jreback merged commit 330fb57 into pandas-dev:master Dec 6, 2017

peterpanmj deleted the bug-6945 branch January 11, 2018 05:24

peterpanmj restored the bug-6945 branch January 12, 2018 01:52

jreback mentioned this pull request Feb 6, 2018

Rank Mixes np.nan with np.inf values #19538

Closed

peterpanmj deleted the bug-6945 branch April 20, 2018 06:16

		@@ -44,6 +44,10 @@ def rank_1d_{{dtype}}(object in_arr, ties_method='average', ascending=True,

		{{if dtype == 'object'}}
		ndarray sorted_data, values

		@@ -27,7 +27,7 @@ dtypes = [('object', 'object', 'Infinity()', 'NegInfinity()'),
		{{if dtype == 'object'}}


		def rank_1d_{{dtype}}(object in_arr, bint retry=1, ties_method='average',


		.. code-block:: ipython

		In [17]: pd.Series([-np.inf, 0, 1, np.inf]).rank()


		.. code-block:: ipython

		In [15]: pd.Series([np.nan, np.nan, -np.inf, -np.inf]).rank(na_option='top')

		In previous versions, ``inf`` elements were assigned ``NaN`` as their ranks. Now ranks are calculated properly.

		Previous Behavior:


		.. code-block:: ipython

		In [6]: pd.Series([-np.inf, 0, 1, np.nan, np.inf]).rank()


		Furthermore, previously if you rank ``inf`` or ``-inf`` values together with ``NaN`` values, the calculation won't distinguish ``NaN`` from infinity when using 'top' or 'bottom' argument.

		Previously Behavior:

BUG: rank with +-inf, #6945 #17903

BUG: rank with +-inf, #6945 #17903

Conversation

peterpanmj commented Oct 17, 2017 • edited Loading

jreback commented Oct 17, 2017

codecov bot commented Oct 17, 2017 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

peterpanmj Nov 10, 2017 • edited Loading

Choose a reason for hiding this comment

pep8speaks commented Nov 10, 2017 • edited Loading

Comment last updated on December 04, 2017 at 15:42 Hours UTC

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Nov 23, 2017

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Dec 6, 2017

peterpanmj commented Oct 17, 2017 •

edited

Loading

codecov bot commented Oct 17, 2017 •

edited

Loading

peterpanmj Nov 10, 2017 •

edited

Loading

pep8speaks commented Nov 10, 2017 •

edited

Loading