Firestore slow queries

[jenjia]

[REQUIRED] Step 2: Describe your environment

• Xcode version: Version 9.0.1 (9A1004)
• Firebase SDK version: 4.4.0
• Firebase Component: Firestore
• Component version: 0.9.0

[REQUIRED] Step 3: Describe the problem

Goal: mobile (testing on iOS) offline data access.

As the title says queries in Firestore are really slow, memory hungry and CPU.
My user case is a collection of 35.000 documents, each document has ~50 fields, 7 of them being of type Object with 6 of them having one child and 1 having 2 child's
A simple query (eg .whereField("name", isEqualTo: "xxxxxxxx")) takes 30 sec, 100% cpu, up to 1GB memory usage.
(Here is a gif showing the usage over time in xCode: https://giphy.com/gifs/3ov9jGSQVNmwUrFYaY)

Steps to reproduce:

Use the example "Friendly Eats" provided by Firebase.
Change the populate function to populate 20.000 examples (not more, or you will cap the write ops - if you have a free account) wait for all the data to be uploaded to Firebase cloud. Then you can start filtering using the example and the slowness will also be there.

[wilhuff]

This kind of use case is outside the intended performance envelope of the beta SDKs. Right now the iOS SDK does a linear scan of documents in the offline cache with the expectation that the number of documents people actually use is small.

The Javascript SDK is slightly better in that the query is restricted by parent path but this wouldn't really help in this case where you're hoping to get 35k documents in a single collection.

We intend to implement indexing for GA, though I can't promise what that timeline will be.

[fleitz]

Can you add to the documentation that Firestore isn't actually intended for use?

Like maybe, wording such as if you have more than 5 documents in a collection, don't use this as it will be slow as molasses. I have 10 documents in a DB and the queries are taking 2-5 seconds to return.

It would have been highly helpful if I had known that it was not intended for use with more than a document or two.

[morganchen12]

@fleitz that's a bug. Do you have any sample data or code that reproduces this issue?

[hoangvuanhdevelopervn]

Me too. Firestore very very very slow. I just call 10 documents each time but it's take 5, 6 minutes to finish. My netwok is strong, i also using another back-end server and it's fast, firestore is too slow, worst back-end i ever use.

[lonelydatum]

I'm experiencing the same thing. It takes 3 seconds to get the data. Please improve the speed.

[slopSurfer]

Im using ADMIN SDK in node.js AWS lambda function. RDS Postgres will return 280 rows in ~300 millseconds (from a table that contains millions of rows), Dynamodb takes about 1100 ms for same query, Firestore is taking on average 3- 4.5 seconds to return 2 documents! The collection only contains 4 documents. In an Android app the performance is similar. Im in Sydney Australia. Surely there must be some explanation beyond distance? I like Firestore but cant use it the moment because of this issue. I have been experimenting with it for a couple of weeks

[jdimond]

@slopSurfer some of this latency may be attributed to slow-start problems: It takes some time to setup the correct connections between the backends from Australia. For backend/admin operations it's usually recommended to co-locate the code with where your data lives. Are you able to execute the lambda functions in us-central1? This should drastically improve latency.

[slopSurfer]

@jdimond You are correct. The issue appears to be slow initial database connection time. After some brief investigation it appears about 75% of the 'query' time is attributed to the connection process. After the connection latter queries are significantly faster. I looked at a connection and query to Firebase and it was about 30% faster then Firestore, but once again the initial connection was the cause of the slow initial query. I will do the experiment to move the lambda's to us-central1 and get some metrics. Thanks for the feedback.

[jenjia]

We intend to implement indexing for GA, though I can't promise what that timeline will be.

@wilhuff sorry to bother you, a year as passed, and I cannot find if this is already implemented. Can you tell me? or guide me to where I can find the doc of implementation of features likes this?

Thank you.

[wilhuff]

So there are two answers to this: we've addressed some performance problems within the current implementation, mostly to do with large batches. We've also changed the implementation to match the web SDK, where we now restrict by path.

We don't (and can't) publish a roadmap. The definitive list of what we've released is in the Firebase iOS release notes. The Firestore portion of those is generally drawn from the Firestore changelog.

One thing that's just around the corner is document garbage collection. While this doesn't directly address performance problems when querying large collections, it will help keep collections small if you treat current implementation as an offline cache rather than as a local replica of a complete database. You can try this pre-release feature out by building from source using the gsoltis/lru_gc_disabled branch.

We have not yet implemented client-side indexing but still plan to do so.