FR: Add something like batchGet to Firestore SDK

[IchordeDionysos]

• Xcode version: 10.0 (10A255)
• Firebase SDK version: 5.7.0
• Firebase Component: Firestore
• Component version: 0.13.2

Sometimes one comes in the situation where you get back multiple document from a query and each document has a reference saved in the field.
Now you want to get those (and only those documents), want you end up doing is either:

• loop over the array of references and get each individually and try to somehow keep track if every document was already fetched and then call another function
• or if you know that every reference is in one collection to query every document in this collection (and probably query too much data) and after you got all documents you match the queried documents with the reference array.

Both are not the most ideal ways so I looked for an alternative, I knew that there were batches, but they are only for writing, so I dig deeper and ended up at this Firestore REST API page:
There is a function called batchGet https://firebase.google.com/docs/firestore/reference/rest/v1beta1/projects.databases.documents/batchGet
As far as I know this function is not yet implemented in the Firestore SDKs and there is no open issue for that.

Is there any issue with the offline functionality of Firestore and because of that this is not implemented?

[wilhuff]

Thanks for taking the time to write in. We're well aware of this function :-).

The Firestore mobile and web clients use the streaming API to do all their work (see Listen in the protocol). This API forms the foundation of our synchronization mechanism: when you run a query the client supplies a resume token that describes the last time the query was run and the server can send us only the changes.

Additionally, the Listen protocol allows for multiple queries to be active at once, integrating them into a single consistent snapshot with each update. This means that when a document changes if it matches multiple active queries the change is only sent once. The protocol also provides a means of verifying cache coherence.

The batch get RPC doesn't fit this model:

• It's not a part of the stream, which would force the client to reason about versions outside the stream
• Any results we get have to be integrated into the cache and other active listens, but this forces those query results to become inconsistent
• It doesn't accept a resume token, which means it would transfer every result on every request

There are two paths forward on this:

1. Add a new kind of query to the Listen stream that allows a multi-lookup, where each key gets its own resume token or version. This is the right choice from the point of view of consistency, minimizing data transmission, and ease of use. It's also a non-trivial amount of work and hard to prioritize higher than any of the other problems that don't have straightforward workarounds.

2. Add some kind of online-only subset of the API that signals that the results you're getting are strictly coming from the server, don't participate in the cache, may not be in sync with other snapshot results, and wouldn't reflect any locally pending writes. This is much easier to build but far less useful. We've resisted these kinds of things in the past because network latency from mobile devices is perceptible and not being able to see things you just wrote is frustrating.

At this point we're planning on (1). It's on our backlog, but other features are prioritized higher.

For now my advice is to just do the loop.

[saheljalal]

This seems really useful as I have data models that store identifiers for other documents and cannot find a simple way to query or fetch multiple records at once unless I'm missing something.

[wilhuff]

You're not missing anything, but at some point realize that Firestore isn't a traditional database, and structuring your data in a way that requires a join is cutting across the grain. If you referenced data is relatively static you may find it easier to denormalize some or all of it into the referring document.

In any case, as I said above, we recognize the value of this operation, but can't prioritize this higher than other work.

[saheljalal]

Yeah I would have liked to do that but the data as you say may not be static and updates would need to touch many records possibly. Will be great when this gets in.

[funct7]

you may find it easier to denormalize some or all of it into the referring document

Not the greatest solution, but at least I can say "Well, Google told me to do it." about the denormalized document. 😄Thanks! (Not being sarcastic.)

[eminisrafil]

@wilhuff Any update on this? Around priority or timing?

[morganchen12]

@eminisrafil the Firestore SDK team's recent work has largely been focused on adding support for C++/Unity, and this will likely continue until those platforms' support reaches GA. The priority of this issue has not been increased and thus far no significant progress has been made.