GODRIVER-3340 Add a test for goroutine leaks. #1874

matthewdale · 2024-10-23T17:52:01Z

Summary

Add a new submodule internal/test/goleak that tests for goroutine leaks.
Add a new build variant "Goroutine Leaks Test" that runs the new test.

Background & Motivation

Our usage of the zstd.Decoder pooled with a sync.Pool created the possibility of a gorutine leak when using the "zstd" compressor. We didn't catch it because we currently don't have any way to detect goroutine leaks. Add a test that uses github.com/uber-go/goleak to check for goroutine leaks.

mongodb-drivers-pr-bot · 2024-10-23T17:55:38Z

API Change Report

No changes found!

prestonvasquez · 2024-11-05T22:46:11Z

internal/test/goleak/goleak_test.go

+	}{
+		{
+			desc: "base",
+			opts: options.Client().ApplyURI(uri),


ApplyURI is used in every test and if it's empty will default to localhost:27017. Suggest applying this value in the test runner:

opts := tc.opts if u := os.Getenv("MONGODB_URI"); u != "" { opts.ApplyURI(u) } client, err := mongo.Connect(context.Background(), opts) require.NoError(t, err)

My concern with that approach is that ApplyURI will overwrite the explicitly configured values if it's called after. However, I believe I can get the same effect using something like

base := options.Client() if u := os.Getenv("MONGODB_URI"); u != "" { base.ApplyURI(u) } client, err := mongo.Connect(context.Background(), base, t.opts)

prestonvasquez · 2024-11-05T22:50:07Z

internal/test/goleak/goleak_test.go

+		// These can't be run in parallel because goleak currently can't filter
+		// out goroutines from other parallel subtests.
+		t.Run(tc.desc, func(t *testing.T) {
+			defer goleak.VerifyNone(t)


The uber-go/goleak documentation implies that we don't have to make this check both in the subtest and at the TestMain() level: https://github.com/uber-go/goleak#quick-start . If that's true, suggest removing TestMain().

Good idea. I added it to catch goroutine leaks that I didn't know I was looking for, but it's never caught a goroutine leak that goleak.VerifyNone didn't catch, so I'm fine removing it.

We should definitely prioritize using goleak.VerifyNone in specific test cases because it both indicates which scenario(s) caused a goroutine leak and the output shows up clearly in the Evergreen test logs. goleak.VerifyTestMain does neither.

Removed TestMain.

prestonvasquez · 2024-11-05T22:54:46Z

internal/test/goleak/goleak_test.go

+			_, err = coll.InsertOne(context.Background(), bson.M{"x": 123})
+			require.NoError(t, err)
+
+			for i := 0; i < 20; i++ {


What is the significance of 20? Should we constantize this value to give it context?

20 is "enough to cause a noticeable problem", so it doesn't have any special significance. However, note that we've only detected one type of goroutine leak with this test so far, so it's possible that we'll need to change the test scenario, duration, etc to catch future goroutine leaks. I'll leave a comment about the significance and limitations of the overall test loop related to detecting goroutine leaks. Ideally, we should actually be testing for goroutine leaks in our existing test suites, but that has proved somewhat difficult to implement.

I ended up removing the sleep, increased the iterations to 50, moved the InsertOne into the loop, added a ChangeStream simulated workload, and added some operations that intentionally time out. I added a comment describing the intent of the different operations.

prestonvasquez · 2024-11-05T22:55:06Z

internal/test/goleak/goleak_test.go

+				var res bson.D
+				err = coll.FindOne(context.Background(), bson.D{}).Decode(&res)
+				require.NoError(t, err)
+				time.Sleep(50 * time.Millisecond)


Could we add a comment to describe what we are awaiting?

For some reason, adding the sleep increases the number of goroutines that leak when using "zstd" compression. I will add a comment describing the current purpose for sleeping there.

…comments.

matthewdale · 2024-11-12T19:03:52Z

drivers-pr-bot please backport to master

mongodb-drivers-pr-bot · 2024-11-12T19:16:39Z

Sorry, unable to cherry-pick to master, please backport manually. Here are approximate instructions:

Checkout backport branch and update it.

git checkout -b cherry-pick-master-c3448e685952c728a60ac002bd8c3df36c8b899b master

git fetch origin c3448e685952c728a60ac002bd8c3df36c8b899b

Cherry pick the first parent branch of the this PR on top of the older branch:

git cherry-pick -x -m1 c3448e685952c728a60ac002bd8c3df36c8b899b

You will likely have some merge/cherry-pick conflicts here, fix them and commit:

git commit -am {message}

Push to a named branch:

git push origin cherry-pick-master-c3448e685952c728a60ac002bd8c3df36c8b899b

Create a PR against branch master. I would have named this PR:

"GODRIVER-3340 Add a test for goroutine leaks. (#1874) [master]"

(cherry picked from commit c3448e6)

* commit '0dc2e05e': (184 commits) GODRIVER-3448 Limit GOMAXPROCS for fuzz tests (#1939) [v1] (#1943) GODRIVER-3470 Correct BSON unmarshaling logic for null values (#1924) GODRIVER-3370 Add bypassEmptyTsReplacement option. (#1927) BUMP v1.17.2 GODRIVER-3436 Avoid initializing null data given custom decoder (#1902) GODRIVER-3340 Add a test for goroutine leaks. (#1874) Update reviewers.txt (#1855) [v1] (#1883) Fix data race in 'discard connections' pool test. [v1] (#1877) Bump golangci-lint for 1.23 compatibility [v1] (#1875) GODRIVER-3340 Bump github.com/klauspost/compress from 1.13.6 to 1.16.7 [v1] (#1869) GODRIVER-3374 Add ReadCompressedCompressedMessage back to wiremessage API (#1870) GODRIVER-3156 Detect and discard closed idle connections. (#1815) GODRIVER-3358 Do not override authSource from TXT record (#1830) DEVPROD-10453 Use assume_role for s3 uploads [v1] (#1824) GODRIVER-2589 Clarify `*Cursor.All()` behavior in comment. (#1804) GODRIVER-3313 Skip CSOT spec tests on Windows and macOS. (#1818) BUMP v1.17.0 GODRIVER-3302 Handle malformatted message length properly. (#1758) GODRIVER-3312 Use remaining test secrets from the vault [v1] (#1811) Remove GCP from supplied callback example (#1809) ...

…41094483929 * release/2.0: (203 commits) Use different credentials for merge-up PRs (#1968) Add GitHub Actions workflow for merge ups (#1962) BUMP v1.17.3 GODRIVER-3448 Limit GOMAXPROCS for fuzz tests (#1939) [v1] (#1943) [release/1.17] (#1957) Update reviewers.txt (#1855) [v1] (#1883) [release/1.17] (#1958) Cherry pick 1.17.2 work to release/1.17 (#1956) GODRIVER-3470 Correct BSON unmarshaling logic for null values (#1924) (#1955) GODRIVER-3370 Add bypassEmptyTsReplacement option. (#1927) [release/1.17] (#1954) GODRIVER-3448 Limit GOMAXPROCS for fuzz tests (#1939) [v1] (#1943) GODRIVER-3470 Correct BSON unmarshaling logic for null values (#1924) GODRIVER-3370 Add bypassEmptyTsReplacement option. (#1927) BUMP v1.17.2 GODRIVER-3436 Avoid initializing null data given custom decoder (#1902) GODRIVER-3340 Add a test for goroutine leaks. (#1874) Update reviewers.txt (#1855) [v1] (#1883) GODRIVER-3340 Bump github.com/klauspost/compress from 1.13.6 to 1.16.7 [release/1.17] (#1880) GODRIVER-3374 Add ReadCompressedCompressedMessage back to wiremessage API (#1870) Fix data race in 'discard connections' pool test. [v1] (#1877) Bump golangci-lint for 1.23 compatibility [v1] (#1875) Fix data race in 'discard connections' pool test. [v1] (#1877) ...

mongodb-drivers-pr-bot bot added dependencies Pull requests that update a dependency file priority-3-low Low Priority PR for Review labels Oct 23, 2024

matthewdale force-pushed the experiment-detect-goroutines branch 5 times, most recently from c8d11ec to e81225e Compare October 25, 2024 18:52

matthewdale marked this pull request as ready for review October 25, 2024 18:56

GODRIVER-3340 Add a test for goroutine leaks.

e339231

matthewdale force-pushed the experiment-detect-goroutines branch from e81225e to e339231 Compare October 25, 2024 19:07

mongodb-drivers-pr-bot bot requested a review from blink1073 October 25, 2024 19:12

blink1073 requested review from prestonvasquez and removed request for blink1073 October 28, 2024 09:55

prestonvasquez reviewed Nov 5, 2024

View reviewed changes

Run goleak tests on replica set and sharded. Run change streams. Add …

45fba77

…comments.

matthewdale requested a review from prestonvasquez November 11, 2024 20:18

prestonvasquez approved these changes Nov 12, 2024

View reviewed changes

matthewdale merged commit c3448e6 into mongodb:v1 Nov 12, 2024
34 checks passed

matthewdale added a commit to matthewdale/mongo-go-driver that referenced this pull request Nov 13, 2024

GODRIVER-3340 Add a test for goroutine leaks. (mongodb#1874)

b30096f

(cherry picked from commit c3448e6)

matthewdale mentioned this pull request Nov 13, 2024

GODRIVER-3340 Add a test for goroutine leaks. (#1874) [master] #1890

Merged

matthewdale added a commit that referenced this pull request Nov 19, 2024

GODRIVER-3340 Add a test for goroutine leaks. (#1874) [master] (#1890)

4f4f715

prestonvasquez pushed a commit to prestonvasquez/mongo-go-driver that referenced this pull request Feb 21, 2025

GODRIVER-3340 Add a test for goroutine leaks. (mongodb#1874)

993ea0e

mongodb-drivers-pr-bot bot mentioned this pull request Mar 4, 2025

Merge v1 into release/2.0 #1971

Merged

mongodb-drivers-pr-bot bot mentioned this pull request Mar 4, 2025

Merge release/2.0 into release/2.1 #1972

Merged

mongodb-drivers-pr-bot bot mentioned this pull request Mar 4, 2025

Merge release/2.1 into master #1973

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GODRIVER-3340 Add a test for goroutine leaks. #1874

GODRIVER-3340 Add a test for goroutine leaks. #1874

matthewdale commented Oct 23, 2024 •

edited

Loading

mongodb-drivers-pr-bot bot commented Oct 23, 2024 •

edited

Loading

prestonvasquez Nov 5, 2024 •

edited

Loading

matthewdale Nov 11, 2024

matthewdale Nov 11, 2024

prestonvasquez Nov 5, 2024

matthewdale Nov 11, 2024

matthewdale Nov 11, 2024

prestonvasquez Nov 5, 2024

matthewdale Nov 11, 2024

matthewdale Nov 11, 2024

prestonvasquez Nov 5, 2024

matthewdale Nov 11, 2024

matthewdale commented Nov 12, 2024

mongodb-drivers-pr-bot bot commented Nov 12, 2024

GODRIVER-3340 Add a test for goroutine leaks. #1874

GODRIVER-3340 Add a test for goroutine leaks. #1874

Conversation

matthewdale commented Oct 23, 2024 • edited Loading

Summary

Background & Motivation

mongodb-drivers-pr-bot bot commented Oct 23, 2024 • edited Loading

API Change Report

prestonvasquez Nov 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matthewdale commented Nov 12, 2024

mongodb-drivers-pr-bot bot commented Nov 12, 2024

matthewdale commented Oct 23, 2024 •

edited

Loading

mongodb-drivers-pr-bot bot commented Oct 23, 2024 •

edited

Loading

prestonvasquez Nov 5, 2024 •

edited

Loading