Add support for hint option to aggregation. [DATAMONGO-1836]

[spring-projects-issues]

Christoph Strobl opened DATAMONGO-1836 and commented

---

Attachments:

• AggregationOptions.java (5.28 kB)

Issue Links:

• DATAMONGO-2588 Ability to add hint to an aggregation pipeline
  ("is duplicated by")

Referenced from: pull request #878

5 votes, 5 watchers

[spring-projects-issues]

Tristan Bezotte commented

I think this addition would be pretty easy to put in.  We would basically add a hint and handle this in the applyAndReturnPotentiallyChangedCommand(DBObject command),  toDbObject() and the Builder methods. Is there anything else that needs to be taken into consideration? I attached a pared down file with the changes, just to get a feel of it

[dmak]

Dear team,
Could you please give me an idea how to create a hint which specifies the index name explicitly:

{ hint: "index_name" }

As you see I want to pass a BSON literal, not a Document.

[christophstrobl]

@dmak thanks for bringing this up! Good catch - I opened #3747 and JAVA-4262 to tackle this issue.

[dmak]

@christophstrobl Maybe you can help me with the following issue. If you wish I can post it to StackOverflow, but I really think that I do something simply wrong.
I execute the following query using Robo3D

db.definitions.explain().aggregate([ ... ],{ hint : { "definitionId" : 1 } })

It does not matter what aggregate is, the point is that I pass a hint as a document, the hint is correctly interpreted as "use any index of field definitionId" and the winning plan suggests using index definitionId_1 as expected:

{
  "$cursor": {
    "queryPlanner": {
      "winningPlan": {
        "stage": "FETCH",
        "filter": { ... },
        "inputStage": {
          "stage": "IXSCAN",
          "keyPattern": {
            "definitionId": 1.0
          },
          "indexName": "definitionId_1",
          "isMultiKey": false,
          "multiKeyPaths": {
            "definitionId": [ ]
          },
          "isUnique": false,
          "isSparse": false,
          "isPartial": false,
          "indexVersion": 2,
          "direction": "forward",
          "indexBounds": {
            "definitionId": [
              "[ObjectId('5ebacfafac02286c585f8963'), ObjectId('5ebacfafac02286c585f8963')]"
            ]
          }
        }
      },
      "rejectedPlans": [ ]
    }
  }
}

and the query works just fine if I remove .explain(). Now I execute the same query using Spring MongoDB:

CloseableIterator<Document> iter = mongoTemplate.aggregateStream(Aggregation.newAggregation(...)
    .withOptions(AggregationOptions.builder().hint(new Document().append("definitionId", Integer.valueOf(1))).build()), "definitions", Document.class));

which should be equivalent to the above. However it fails this way:

Caused by: com.mongodb.MongoCommandException: Command failed with error 2 (BadValue): 'Failed to determine whether query system can provide a covered projection :: caused by :: error processing query: ns=MAIN.definitionsTree:
    $and ...
Sort: {}
Proj: {}
 planner returned error :: caused by :: hint provided does not correspond to an existing index' on server localhost:10008. The full response is {"operationTime": {"$timestamp": {"t": 1627042099, "i": 1}}, "ok": 0.0, "errmsg": "Failed to determine whether query system can provide a covered projection :: caused by :: error processing query: ns=MAIN.definitionsTree: $and\n  ...\nSort: {}\nProj: {}\n planner returned error :: caused by :: hint provided does not correspond to an existing index", "code": 2, "codeName": "BadValue", "$clusterTime": {"clusterTime": {"$timestamp": {"t": 1627042099, "i": 1}}, "signature": {"hash": {"$binary": {"base64": "PQEZ2jfGn2uXWcyAZqAKjSP17Z8=", "subType": "00"}}, "keyId": 6972219745323974658}}}
	at com.mongodb.internal.connection.ProtocolHelper.getCommandFailureException(ProtocolHelper.java:195) ~[mongodb-driver-core-4.3.0.jar:na]
	at com.mongodb.internal.connection.InternalStreamConnection.receiveCommandMessageResponse(InternalStreamConnection.java:398) ~[mongodb-driver-core-4.3.0.jar:na]
	at com.mongodb.internal.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:319) ~[mongodb-driver-core-4.3.0.jar:na]
	at com.mongodb.internal.connection.CommandProtocolImpl.execute(CommandProtocolImpl.java:81) ~[mongodb-driver-core-4.3.0.jar:na]
	at com.mongodb.internal.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:249) ~[mongodb-driver-core-4.3.0.jar:na]
	at com.mongodb.internal.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:214) ~[mongodb-driver-core-4.3.0.jar:na]
	at com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:123) ~[mongodb-driver-core-4.3.0.jar:na]
	at com.mongodb.internal.operation.CommandOperationHelper.executeCommand(CommandOperationHelper.java:318) ~[mongodb-driver-core-4.3.0.jar:na]
	at com.mongodb.internal.operation.CommandOperationHelper.executeCommandWithConnection(CommandOperationHelper.java:201) ~[mongodb-driver-core-4.3.0.jar:na]
	at com.mongodb.internal.operation.CommandOperationHelper.lambda$executeCommand$4(CommandOperationHelper.java:189) ~[mongodb-driver-core-4.3.0.jar:na]
	at com.mongodb.internal.operation.OperationHelper.withReadConnectionSource(OperationHelper.java:583) ~[mongodb-driver-core-4.3.0.jar:na]
	at com.mongodb.internal.operation.CommandOperationHelper.executeCommand(CommandOperationHelper.java:189) ~[mongodb-driver-core-4.3.0.jar:na]
	at com.mongodb.internal.operation.AggregateOperationImpl.execute(AggregateOperationImpl.java:195) ~[mongodb-driver-core-4.3.0.jar:na]
	at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:184) ~[mongodb-driver-sync-4.3.0.jar:na]
	at com.mongodb.client.internal.MongoIterableImpl.iterator(MongoIterableImpl.java:92) ~[mongodb-driver-sync-4.3.0.jar:na]
	at org.springframework.data.mongodb.core.MongoTemplate$CloseableIterableCursorAdapter.<init>(MongoTemplate.java:3412) ~[spring-data-mongodb-3.2.2.jar:3.2.2]

It seems to be an error in MongoDB driver (I am using the latest v4.3.0) but it is very strange... I have put a breakpoint to com.mongodb.internal.operation.CommandOperationHelper#executeCommand():189 and I see that AggregateOperationImpl#hint is org.bson.BsonDocumentWrapper with value {"annotationDefinitionId": 1}, so everything looks fine. So either there is a trick in a client (Robo3D, CLI, ...) or something is wrong in a driver, or Spring does not cook a hint properly before passing it to the driver, or I just messed up everything and using wrong database / collection (I've checked everything twice)...