3.11.0 regression: documents inserted in a transaction expire immediately

[ianprime0509]

As of Java SDK 3.11.0, it seems that documents inserted in a transaction are set with an expiry of the current time, making them expire immediately and quickly get removed from the collection. Here's a simple example program to demonstrate:

package com.example;

import com.couchbase.client.java.Cluster;
import com.couchbase.client.java.json.JsonObject;
import com.couchbase.client.java.kv.GetOptions;
import org.testcontainers.couchbase.BucketDefinition;
import org.testcontainers.couchbase.CouchbaseContainer;
import org.testcontainers.utility.DockerImageName;

public class TransactionalInsert {
    private static final DockerImageName IMAGE = DockerImageName
            .parse("couchbase:enterprise-7.6.5@sha256:f25b85b64066ad479c18ea5d1b4fb24e921979652f9a2cc50049a9dda04a40a8")
            .asCompatibleSubstituteFor("couchbase/server");

    public static void main(String[] args) throws Exception {
        var container = new CouchbaseContainer(IMAGE).withBucket(new BucketDefinition("MyBucket"));
        container.start();
        var cluster = Cluster.connect(container.getConnectionString(), container.getUsername(), container.getPassword());
        var bucket = cluster.bucket("MyBucket");
        var collection = bucket.defaultCollection();
        cluster.transactions().run(ctx ->
                ctx.insert(collection, "test-document", JsonObject.create().put("key", "value")));

        var document = collection.get("test-document", GetOptions.getOptions().withExpiry(true));
        System.out.println("Document immediately after insert: " + document);

        Thread.sleep(60_000);

        var laterDocument = collection.get("test-document", GetOptions.getOptions().withExpiry(true));
        System.out.println("Document after waiting for expiry cleanup: " + laterDocument);
    }
}

Dependencies

<dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>java-client</artifactId>
    <version>3.11.0</version>
</dependency>
<dependency>
    <groupId>ch.qos.logback</groupId>
    <artifactId>logback-classic</artifactId>
    <version>1.5.28</version>
</dependency>
<dependency>
    <groupId>org.testcontainers</groupId>
    <artifactId>testcontainers-couchbase</artifactId>
    <version>2.0.3</version>
</dependency>

Using SDK version 3.11.0 as above results in this output (only relevant parts included):

Document immediately after insert: GetResult{content={"key":"value"}, flags=0x2000000, cas=0x1892acef6da70000, expiry=Optional[2026-02-09T20:09:07Z]}
Exception in thread "main" com.couchbase.client.core.error.DocumentNotFoundException: Document with the given id not found {"completed":true,"coreId":"0xe3f6717000000001","idempotent":true,"lastChannelId":"E3F6717000000001/0000000088CFCAE1","lastDispatchedFrom":"127.0.0.1:50586","lastDispatchedTo":"localhost:50490","requestId":1068,"requestType":"SubdocGetRequest","retried":0,"serverDuration":6,"service":{"bucket":"MyBucket","collection":"_default","documentId":"test-document","errorCode":{"description":"Not Found","name":"KEY_ENOENT"},"opaque":"0x446","scope":"_default","type":"kv","vbucket":863},"status":"NOT_FOUND","timeoutMs":2500,"timings":{"dispatchMicros":841,"totalDispatchMicros":841,"totalServerMicros":0,"totalMicros":1443,"serverMicros":0}}

Using SDK version 3.10.1 results in the expected output:

Document immediately after insert: GetResult{content={"key":"value"}, flags=0x2000000, cas=0x1892ad5bfc3f0000, expiry=Optional.empty}
Document after waiting for expiry cleanup: GetResult{content={"key":"value"}, flags=0x2000000, cas=0x1892ad5bfc3f0000, expiry=Optional.empty}

[yeikel]

cc @dnault This is quite the regression. This forced us to revert the upgrade

Could you please comment?

[dnault]

@ianprime0509 Thank you for the comprehensive bug report.

@yeikel You have a gift for understatement. Tracking as JVMCBC-1712.

We are investigating the root cause of the bug, and plan to release 3.11.1 with the fix as soon as we can.

More importantly, we are investigating how an issue of this severity made it past our release testing.

We'll use this GitHub issue to share what we learn, including whether it's possible to feel even more sorry and embarrassed than I do right now.

[yeikel]

We'll use this GitHub issue to share what we learn, including whether it's possible to feel even more sorry and embarrassed than I do right now.

Oh man, don’t beat yourself up like that. These things happen. As long as we can learn and improve from here, that’s all we can ask.

Thanks, as always, for your quick support and issue resolution.

[dnault]

Root cause

We think it's an interaction between "create as deleted" (which is how the transactions implementation stages changes prior to commit -- in a "deleted" tombstone document) and "preserve expiry" -- which is something we started using in transactions in 3.11.0.

When the server creates a deleted document, it sets its expiry to the current time to help with its internal bookkeeping. When that expiry is preserved during transaction commit (instead of being reset like it used to), the reified document still has the expiry of the tombstone, and so expires immediately.

Process failure post mortem

One of the integration tests did flag the issue, but we did not notice the failing test. The test is in a suite that we have not historically kept a close eye on during release testing, partly due to flaky tests in this suite, and partly because it runs separately from the jobs dedicated to release testing.

The plan

• We're reverting the change that introduced the regression. This means 3.11.1 will not support setting document expiry inside a transaction.
• In future, we are including the test suite that caught the bug in our pre-release checklist.
• We are adding a regression test for this issue to the release test suite.
• We will fix the flaky tests, so we will be alerted to legitimate test failures sooner.
• We will release 3.11.1 as soon as we're confident that reverting this change has not disrupted anything else. My best guess is that the release will happen some time next week.

[dnault]

We have completed the remediation actions mentioned above.

Version 3.11.1 is now available in Maven Central, and the release notes are on the Couchbase documentation site.

We apologize again for this regression. Thank you for your patience and understanding.