Bad value in mongoObjectToParseObject when query order by Date descending

[andj207]

Issue Description &

Steps to reproduce

I use a self host ParseServer and my database currently grows more than 80k objects in a collection. I use parse-dashboard to view and manage them. The problem happens when I browse that collection sorted by "publishedDate" field descending. The error is below:
ParseError { code: 1, message: 'bad value in mongoObjectToParseObject' }

It only happens with that field ordered by descending, works well by ascending. Other fields in that collection and other collections work well too. I've checked on my mobile apps, the data cannot be loaded because of that error.
I've tried many ways to overcome the error:
(1) set all null values to a Date value;
(2) copied all that field's values to a new field and copied them back;
(3) converted that field's date values to Number and stored in other field and converted them back (using mongo language)

object.time = object.publishedDate.getTime(); 

object.publishedDate = new ISODate(object.time); // this works but the value is wrong (for example Jan 31, 9998 while expected is Sep 3, 2001) or

object.publishedDate = new ISODate(new Date(object.time).toISOString()) //or

object.publishedDate = new Date(object.time);

None of them help to solve the error. My local database work well with the same data.

Environment Setup

• Server

  • parse-server version: 2.8.1
  • Operating System: Ubuntu 16.04
  • Hardware: 1GB Ram, 25GB Disk space.
  • Localhost or remote server?: Digital Ocean

• Database

  • MongoDB version: 3.0.8
  • Storage engine: MongoRocks
  • Hardware: 1GB Ram, 25GB Disk space.
  • Localhost or remote server?: Digital Ocean

Logs/Trace

Error generating response. ParseError { code: 1, message: 'bad value in mongoObjectToParseObject'}

[flovilmart]

Do you have a sample of the objects that don’t work? It’s possible that a single bad value may cause the error.

If you have the objects as stored in the DB (right from mongo) and one after transformation, that would help.

[andj207]

@flovilmart , I'm trying to find what causes the problem, I still cannot find the error objects in 80k objects data yet. It just cannot be sorted descending, only works with ascending.

[dplewis]

Quick question does it not work for all date fields or just that one. Like if you copied to another field could you sort?

[andj207]

createdAt and updatedAt work well. It does not work with the new Date type field that the data was copied to. When I converted to Number using getTime() function, the number type field could not be sorted descending too. It must there be some sort of error value or data overflow or something in the field and I cannot find it yet. I wished the error can report more info when I turned on logLevel to verbose but the error was just a message ParseError { code: 1, message: 'bad value in mongoObjectToParseObject'}.

[dplewis]

Have you tried inspect element / debugging the dashboard / server?

Are your fields indexes properly 80k sounds like a lot?

[andj207]

The dashboard work will with other collections. In the browser console, dashboard's requests run through except when I sort that field and http response code is 500 Internal Server Error. Both the dashboard and my mobile apps throw the same error message above. The field is not index.

[dplewis]

Well we know it’s coming from here

https://github.com/parse-community/parse-server/blob/master/src/Adapters/Storage/Mongo/MongoTransform.js#L1144

You can try to set a breakpoint or console.log to see if you can find the object.

[dplewis]

If you find it can you post it here so we can write a test for it?

A mongo object or verbose log would be appreciated.

[andj207]

I logged the object at the point (actually it's at nestedMongoObjectToNestedParseObject function) console.log("I found you", mongoObject);, and it's an undefined object. I rechecked my database with db.mycollection.count({publishedDate: {$exists: false}}), it returns 0. The dashboard I filter with "does not exist" query returns empty collection too. But it still can not be sorted descending.

[dplewis]

Oh I didn’t realize it was thrown in two places. Need to update that message.

At least you are on the right track. You can follow the call stack.

[andj207]

Found it

{ _id: 'vQHyinCW1l',
publishedDate: 2018-08-09T17:00:00.000Z,
urls:
{ firstUrl: ‘<valid url>’,
	secondUrl: undefined },
uid: 'fd9ad3cfe364a9b09a535f3ba2bab8b6',
title: ‘a title’,
eNumber: 19,
numberId: 1,
parentUid: '58d9be13dd717e4b063fc4e23e2abc25',
image: ‘<image path>’,
_created_at: 2018-08-12T05:03:03.540Z,
_updated_at: 2018-08-12T05:03:03.540Z,
lastComment: '' }

The error is thrown when urls nested object (Object type value) is mapping and because of secondUrl is undefined. In this case, do I have to set secondUrl to empty string? Why can't it be null/undefined?

[dplewis]

Thanks for finding the object. Once I saw nested I knew it had to be a different field.

Try setting it to an empty string and see if that works. You might have more objects and you know what to look for now.

That error has been there since the beginning Parse Server 2.0 and there are no test cases for it.

I don’t know what the logic was behind it. But I don’t see anything wrong with returning undefined / null. @flovilmart Thoughts?

[andj207]

Thank you @dplewis for your great help.

[dplewis]

@andj207 no problem. Although I still have a ton of questions about the behavior you were having from this. I’ll play around with it and see if I can replicate it

[flovilmart]

There’s no reason to crash there, If we can work around it, i’d Happily release a 2.8.5 sooner