BUG: wrong parsing of microseconds with format arg (#4152)

[jorisvandenbossche]

Closes #4152.

fraction was already in microseconds, so no need to multiply it with 1000000, which caused faulty parsing of dates with microseconds.

With this commit I get:

pandas version: 0.12.0.dev
Value to parse: 01-Apr-2011 00:00:01.978
datetime.strptime        : 2011-04-01 00:00:01.978000
to_datetime, w/out format: 2011-04-01 00:00:01.978000
to_datetime, w/ format   : 2011-04-01 00:00:01.978000

for

import datetime as dt
import pandas as pd
val = '01-Apr-2011 00:00:01.978'
print 'pandas version:',pd.__version__
print 'Value to parse:',val
format = '%d-%b-%Y %H:%M:%S.%f'
print 'datetime.strptime        :',dt.datetime.strptime(val, format)
print 'to_datetime, w/out format:',pd.to_datetime(val)
print 'to_datetime, w/ format   :', pd.to_datetime(val, format=format)

[jorisvandenbossche]

I added a test based on the reported bug in the issue, but I don't have experience with writing tests. Is it OK like this?

[hayd]

Nearly, not sure this is going to work since now to_datetime returns a Timestamp object rather than a Datatime...

Maybe better to do (will also give nicer error message):

self.assertEquals(str(result),  '2011-04-01 00:00:01.978000')

[jreback]

that is fine

there is another issue in processing to_datetime
thanks for this catch

[jreback]

a Timestamp will compare to a datetime ok

[hayd]

@jreback my bad!

[jorisvandenbossche]

OK!

@jreback What do you mean with another issue?

[jreback]

to_datetime had some convoluted logic and I came up with a case where the format was being interpreted correctly; the existing test was not good enough

but unrelated to the bug you are fixing.....

going to merge yours and then fix the other issue

[jreback]

you can reproduce if you pass a list of values (rather than a single value)

it will ignore the format

[jreback]

@jorisvandenbossche I take that back....looks like your PR fixes this....

can you add a release notes mention? (then squash together if you can)...thanks

[jorisvandenbossche]

OK, I added something to the release notes. With 'squaqh together', do you mean putting it together in one commit?

I looked at the PR #4167 you mentioned, but I doubt that this PR fixes the test you added there ... (this one only changed something that had to do with microseconds)

[jorisvandenbossche]

@jreback OK, I also put it in one commit.

[hayd]

lovely! Excellent work!

[jreback]

@jorisvandenbossche 2nd that!

this really does fix the error, my PR just makes the test more consistent (and is indepdent of what you did)

thanks again

[cpcloud]

this introduced another bug it seems, see #4179

[jorisvandenbossche]

@cpcloud Strange, it is exactly the test that I introduced in this PR and which should be fixed by this PR that is failing. Did you rebuild the cython files?

[cpcloud]

oh gosh duh. sorry. closing the issue