-
Notifications
You must be signed in to change notification settings - Fork 476
chore(DSM): fix test flake for dsm processor shutdown #15821
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
chore(DSM): fix test flake for dsm processor shutdown #15821
Conversation
Codeowners resolved as |
dubloom
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Performance SLOsComparing candidate rob.carlan/test-dsm-processor-flake-fix-simple (c3749d9) with baseline main (3ad335e) 📈 Performance Regressions (3 suites)📈 iastaspects - 118/118✅ add_aspectTime: ✅ 17.940µs (SLO: <20.000µs 📉 -10.3%) vs baseline: 📈 +21.0% Memory: ✅ 42.585MB (SLO: <43.250MB 🟡 -1.5%) vs baseline: +4.8% ✅ add_inplace_aspectTime: ✅ 14.955µs (SLO: <20.000µs 📉 -25.2%) vs baseline: ~same Memory: ✅ 42.566MB (SLO: <43.250MB 🟡 -1.6%) vs baseline: +4.8% ✅ add_inplace_noaspectTime: ✅ 0.343µs (SLO: <10.000µs 📉 -96.6%) vs baseline: +1.5% Memory: ✅ 42.664MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +5.1% ✅ add_noaspectTime: ✅ 0.549µs (SLO: <10.000µs 📉 -94.5%) vs baseline: +0.8% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +5.2% ✅ bytearray_aspectTime: ✅ 17.956µs (SLO: <30.000µs 📉 -40.1%) vs baseline: -1.3% Memory: ✅ 42.585MB (SLO: <43.500MB -2.1%) vs baseline: +4.7% ✅ bytearray_extend_aspectTime: ✅ 23.977µs (SLO: <30.000µs 📉 -20.1%) vs baseline: +0.6% Memory: ✅ 42.546MB (SLO: <43.500MB -2.2%) vs baseline: +4.6% ✅ bytearray_extend_noaspectTime: ✅ 2.744µs (SLO: <10.000µs 📉 -72.6%) vs baseline: -0.7% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +5.1% ✅ bytearray_noaspectTime: ✅ 1.488µs (SLO: <10.000µs 📉 -85.1%) vs baseline: +0.1% Memory: ✅ 42.487MB (SLO: <43.500MB -2.3%) vs baseline: +4.4% ✅ bytes_aspectTime: ✅ 16.656µs (SLO: <20.000µs 📉 -16.7%) vs baseline: +0.1% Memory: ✅ 42.664MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +4.9% ✅ bytes_noaspectTime: ✅ 1.417µs (SLO: <10.000µs 📉 -85.8%) vs baseline: -1.3% Memory: ✅ 42.487MB (SLO: <43.500MB -2.3%) vs baseline: +4.8% ✅ bytesio_aspectTime: ✅ 55.471µs (SLO: <70.000µs 📉 -20.8%) vs baseline: -0.3% Memory: ✅ 42.546MB (SLO: <43.500MB -2.2%) vs baseline: +4.8% ✅ bytesio_noaspectTime: ✅ 3.302µs (SLO: <10.000µs 📉 -67.0%) vs baseline: +1.0% Memory: ✅ 42.487MB (SLO: <43.500MB -2.3%) vs baseline: +4.8% ✅ capitalize_aspectTime: ✅ 14.676µs (SLO: <20.000µs 📉 -26.6%) vs baseline: ~same Memory: ✅ 42.605MB (SLO: <43.500MB -2.1%) vs baseline: +5.0% ✅ capitalize_noaspectTime: ✅ 2.569µs (SLO: <10.000µs 📉 -74.3%) vs baseline: -1.0% Memory: ✅ 42.585MB (SLO: <43.500MB -2.1%) vs baseline: +4.8% ✅ casefold_aspectTime: ✅ 14.706µs (SLO: <20.000µs 📉 -26.5%) vs baseline: -0.3% Memory: ✅ 42.664MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +5.0% ✅ casefold_noaspectTime: ✅ 3.168µs (SLO: <10.000µs 📉 -68.3%) vs baseline: -0.2% Memory: ✅ 42.664MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +5.0% ✅ decode_aspectTime: ✅ 15.594µs (SLO: <30.000µs 📉 -48.0%) vs baseline: -0.5% Memory: ✅ 42.566MB (SLO: <43.500MB -2.1%) vs baseline: +4.6% ✅ decode_noaspectTime: ✅ 1.599µs (SLO: <10.000µs 📉 -84.0%) vs baseline: -1.0% Memory: ✅ 42.585MB (SLO: <43.500MB -2.1%) vs baseline: +4.9% ✅ encode_aspectTime: ✅ 14.766µs (SLO: <30.000µs 📉 -50.8%) vs baseline: ~same Memory: ✅ 42.585MB (SLO: <43.500MB -2.1%) vs baseline: +5.0% ✅ encode_noaspectTime: ✅ 1.662µs (SLO: <10.000µs 📉 -83.4%) vs baseline: 📈 +11.1% Memory: ✅ 42.526MB (SLO: <43.500MB -2.2%) vs baseline: +4.8% ✅ format_aspectTime: ✅ 170.664µs (SLO: <200.000µs 📉 -14.7%) vs baseline: -0.5% Memory: ✅ 42.723MB (SLO: <43.250MB 🟡 -1.2%) vs baseline: +5.0% ✅ format_map_aspectTime: ✅ 191.307µs (SLO: <200.000µs -4.3%) vs baseline: ~same Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +4.8% ✅ format_map_noaspectTime: ✅ 3.793µs (SLO: <10.000µs 📉 -62.1%) vs baseline: +0.6% Memory: ✅ 42.625MB (SLO: <43.250MB 🟡 -1.4%) vs baseline: +4.9% ✅ format_noaspectTime: ✅ 3.169µs (SLO: <10.000µs 📉 -68.3%) vs baseline: ~same Memory: ✅ 42.605MB (SLO: <43.250MB 🟡 -1.5%) vs baseline: +5.0% ✅ index_aspectTime: ✅ 15.266µs (SLO: <20.000µs 📉 -23.7%) vs baseline: -0.4% Memory: ✅ 42.644MB (SLO: <43.250MB 🟡 -1.4%) vs baseline: +5.0% ✅ index_noaspectTime: ✅ 0.461µs (SLO: <10.000µs 📉 -95.4%) vs baseline: ~same Memory: ✅ 42.664MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +5.1% ✅ join_aspectTime: ✅ 17.112µs (SLO: <20.000µs 📉 -14.4%) vs baseline: ~same Memory: ✅ 42.566MB (SLO: <43.500MB -2.1%) vs baseline: +4.9% ✅ join_noaspectTime: ✅ 1.551µs (SLO: <10.000µs 📉 -84.5%) vs baseline: -1.0% Memory: ✅ 42.625MB (SLO: <43.250MB 🟡 -1.4%) vs baseline: +5.0% ✅ ljust_aspectTime: ✅ 20.852µs (SLO: <30.000µs 📉 -30.5%) vs baseline: +0.3% Memory: ✅ 42.684MB (SLO: <43.250MB 🟡 -1.3%) vs baseline: +5.2% ✅ ljust_noaspectTime: ✅ 2.717µs (SLO: <10.000µs 📉 -72.8%) vs baseline: -1.0% Memory: ✅ 42.546MB (SLO: <43.250MB 🟡 -1.6%) vs baseline: +4.6% ✅ lower_aspectTime: ✅ 21.659µs (SLO: <30.000µs 📉 -27.8%) vs baseline: 📈 +20.9% Memory: ✅ 42.605MB (SLO: <43.500MB -2.1%) vs baseline: +4.9% ✅ lower_noaspectTime: ✅ 2.440µs (SLO: <10.000µs 📉 -75.6%) vs baseline: -0.2% Memory: ✅ 42.605MB (SLO: <43.250MB 🟡 -1.5%) vs baseline: +4.9% ✅ lstrip_aspectTime: ✅ 17.734µs (SLO: <30.000µs 📉 -40.9%) vs baseline: +0.4% Memory: ✅ 42.546MB (SLO: <43.250MB 🟡 -1.6%) vs baseline: +4.8% ✅ lstrip_noaspectTime: ✅ 1.847µs (SLO: <10.000µs 📉 -81.5%) vs baseline: -1.4% Memory: ✅ 42.644MB (SLO: <43.500MB 🟡 -2.0%) vs baseline: +5.1% ✅ modulo_aspectTime: ✅ 165.867µs (SLO: <200.000µs 📉 -17.1%) vs baseline: -0.3% Memory: ✅ 42.684MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +5.1% ✅ modulo_aspect_for_bytearray_bytearrayTime: ✅ 174.053µs (SLO: <200.000µs 📉 -13.0%) vs baseline: -0.2% Memory: ✅ 42.566MB (SLO: <43.500MB -2.1%) vs baseline: +4.5% ✅ modulo_aspect_for_bytesTime: ✅ 169.069µs (SLO: <200.000µs 📉 -15.5%) vs baseline: ~same Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +4.7% ✅ modulo_aspect_for_bytes_bytearrayTime: ✅ 172.065µs (SLO: <200.000µs 📉 -14.0%) vs baseline: +0.3% Memory: ✅ 42.605MB (SLO: <43.500MB -2.1%) vs baseline: +4.8% ✅ modulo_noaspectTime: ✅ 3.672µs (SLO: <10.000µs 📉 -63.3%) vs baseline: -0.3% Memory: ✅ 42.546MB (SLO: <43.500MB -2.2%) vs baseline: +5.0% ✅ replace_aspectTime: ✅ 211.964µs (SLO: <300.000µs 📉 -29.3%) vs baseline: -0.2% Memory: ✅ 42.703MB (SLO: <44.000MB -2.9%) vs baseline: +5.2% ✅ replace_noaspectTime: ✅ 2.893µs (SLO: <10.000µs 📉 -71.1%) vs baseline: -1.0% Memory: ✅ 42.566MB (SLO: <43.500MB -2.1%) vs baseline: +4.8% ✅ repr_aspectTime: ✅ 1.418µs (SLO: <10.000µs 📉 -85.8%) vs baseline: ~same Memory: ✅ 42.644MB (SLO: <43.500MB 🟡 -2.0%) vs baseline: +5.0% ✅ repr_noaspectTime: ✅ 0.527µs (SLO: <10.000µs 📉 -94.7%) vs baseline: +0.8% Memory: ✅ 42.546MB (SLO: <43.500MB -2.2%) vs baseline: +4.8% ✅ rstrip_aspectTime: ✅ 19.048µs (SLO: <30.000µs 📉 -36.5%) vs baseline: +0.2% Memory: ✅ 42.546MB (SLO: <43.500MB -2.2%) vs baseline: +4.8% ✅ rstrip_noaspectTime: ✅ 1.951µs (SLO: <10.000µs 📉 -80.5%) vs baseline: +0.7% Memory: ✅ 42.526MB (SLO: <43.500MB -2.2%) vs baseline: +4.5% ✅ slice_aspectTime: ✅ 15.880µs (SLO: <20.000µs 📉 -20.6%) vs baseline: -0.5% Memory: ✅ 42.605MB (SLO: <43.500MB -2.1%) vs baseline: +5.0% ✅ slice_noaspectTime: ✅ 0.600µs (SLO: <10.000µs 📉 -94.0%) vs baseline: -0.4% Memory: ✅ 42.605MB (SLO: <43.500MB -2.1%) vs baseline: +5.0% ✅ stringio_aspectTime: ✅ 54.199µs (SLO: <80.000µs 📉 -32.3%) vs baseline: -0.7% Memory: ✅ 42.585MB (SLO: <43.500MB -2.1%) vs baseline: +5.1% ✅ stringio_noaspectTime: ✅ 3.636µs (SLO: <10.000µs 📉 -63.6%) vs baseline: -0.4% Memory: ✅ 42.566MB (SLO: <43.500MB -2.1%) vs baseline: +4.8% ✅ strip_aspectTime: ✅ 17.579µs (SLO: <20.000µs 📉 -12.1%) vs baseline: -0.3% Memory: ✅ 42.526MB (SLO: <43.500MB -2.2%) vs baseline: +4.9% ✅ strip_noaspectTime: ✅ 1.861µs (SLO: <10.000µs 📉 -81.4%) vs baseline: -1.2% Memory: ✅ 42.605MB (SLO: <43.500MB -2.1%) vs baseline: +5.0% ✅ swapcase_aspectTime: ✅ 18.549µs (SLO: <30.000µs 📉 -38.2%) vs baseline: -0.2% Memory: ✅ 42.507MB (SLO: <43.500MB -2.3%) vs baseline: +4.7% ✅ swapcase_noaspectTime: ✅ 2.805µs (SLO: <10.000µs 📉 -72.0%) vs baseline: -0.6% Memory: ✅ 42.585MB (SLO: <43.500MB -2.1%) vs baseline: +5.1% ✅ title_aspectTime: ✅ 18.289µs (SLO: <30.000µs 📉 -39.0%) vs baseline: -0.2% Memory: ✅ 42.566MB (SLO: <43.000MB 🟡 -1.0%) vs baseline: +4.8% ✅ title_noaspectTime: ✅ 2.672µs (SLO: <10.000µs 📉 -73.3%) vs baseline: -0.2% Memory: ✅ 42.585MB (SLO: <43.500MB -2.1%) vs baseline: +4.9% ✅ translate_aspectTime: ✅ 20.559µs (SLO: <30.000µs 📉 -31.5%) vs baseline: -0.9% Memory: ✅ 42.585MB (SLO: <43.500MB -2.1%) vs baseline: +4.9% ✅ translate_noaspectTime: ✅ 4.328µs (SLO: <10.000µs 📉 -56.7%) vs baseline: -0.1% Memory: ✅ 42.507MB (SLO: <43.500MB -2.3%) vs baseline: +4.9% ✅ upper_aspectTime: ✅ 17.961µs (SLO: <30.000µs 📉 -40.1%) vs baseline: -1.0% Memory: ✅ 42.526MB (SLO: <43.500MB -2.2%) vs baseline: +4.6% ✅ upper_noaspectTime: ✅ 2.438µs (SLO: <10.000µs 📉 -75.6%) vs baseline: -0.1% Memory: ✅ 42.566MB (SLO: <43.500MB -2.1%) vs baseline: +4.9% 📈 iastaspectsospath - 24/24✅ ospathbasename_aspectTime: ✅ 5.162µs (SLO: <10.000µs 📉 -48.4%) vs baseline: 📈 +21.3% Memory: ✅ 42.487MB (SLO: <43.500MB -2.3%) vs baseline: +4.5% ✅ ospathbasename_noaspectTime: ✅ 4.275µs (SLO: <10.000µs 📉 -57.2%) vs baseline: -0.6% Memory: ✅ 42.526MB (SLO: <43.500MB -2.2%) vs baseline: +4.8% ✅ ospathjoin_aspectTime: ✅ 6.214µs (SLO: <10.000µs 📉 -37.9%) vs baseline: -0.7% Memory: ✅ 42.664MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +5.0% ✅ ospathjoin_noaspectTime: ✅ 6.324µs (SLO: <10.000µs 📉 -36.8%) vs baseline: ~same Memory: ✅ 42.428MB (SLO: <43.500MB -2.5%) vs baseline: +4.3% ✅ ospathnormcase_aspectTime: ✅ 3.565µs (SLO: <10.000µs 📉 -64.4%) vs baseline: -0.7% Memory: ✅ 42.585MB (SLO: <43.500MB -2.1%) vs baseline: +4.9% ✅ ospathnormcase_noaspectTime: ✅ 3.591µs (SLO: <10.000µs 📉 -64.1%) vs baseline: -0.8% Memory: ✅ 42.526MB (SLO: <43.500MB -2.2%) vs baseline: +4.5% ✅ ospathsplit_aspectTime: ✅ 4.893µs (SLO: <10.000µs 📉 -51.1%) vs baseline: -0.8% Memory: ✅ 42.487MB (SLO: <43.500MB -2.3%) vs baseline: +4.9% ✅ ospathsplit_noaspectTime: ✅ 5.045µs (SLO: <10.000µs 📉 -49.6%) vs baseline: +0.6% Memory: ✅ 42.546MB (SLO: <43.500MB -2.2%) vs baseline: +4.9% ✅ ospathsplitdrive_aspectTime: ✅ 3.760µs (SLO: <10.000µs 📉 -62.4%) vs baseline: -0.1% Memory: ✅ 42.566MB (SLO: <43.500MB -2.1%) vs baseline: +4.5% ✅ ospathsplitdrive_noaspectTime: ✅ 0.746µs (SLO: <10.000µs 📉 -92.5%) vs baseline: -0.4% Memory: ✅ 42.467MB (SLO: <43.500MB -2.4%) vs baseline: +4.9% ✅ ospathsplitext_aspectTime: ✅ 4.581µs (SLO: <10.000µs 📉 -54.2%) vs baseline: -1.0% Memory: ✅ 42.566MB (SLO: <43.500MB -2.1%) vs baseline: +4.9% ✅ ospathsplitext_noaspectTime: ✅ 4.633µs (SLO: <10.000µs 📉 -53.7%) vs baseline: +0.5% Memory: ✅ 42.566MB (SLO: <43.500MB -2.1%) vs baseline: +5.0% 📈 telemetryaddmetric - 30/30✅ 1-count-metric-1-timesTime: ✅ 3.355µs (SLO: <20.000µs 📉 -83.2%) vs baseline: 📈 +12.4% Memory: ✅ 34.859MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +4.6% ✅ 1-count-metrics-100-timesTime: ✅ 197.642µs (SLO: <220.000µs 📉 -10.2%) vs baseline: -1.5% Memory: ✅ 34.977MB (SLO: <35.500MB 🟡 -1.5%) vs baseline: +5.2% ✅ 1-distribution-metric-1-timesTime: ✅ 3.279µs (SLO: <20.000µs 📉 -83.6%) vs baseline: -1.3% Memory: ✅ 34.859MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +4.9% ✅ 1-distribution-metrics-100-timesTime: ✅ 216.141µs (SLO: <230.000µs -6.0%) vs baseline: +0.9% Memory: ✅ 34.918MB (SLO: <35.500MB 🟡 -1.6%) vs baseline: +5.2% ✅ 1-gauge-metric-1-timesTime: ✅ 2.226µs (SLO: <20.000µs 📉 -88.9%) vs baseline: +1.6% Memory: ✅ 34.859MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +5.1% ✅ 1-gauge-metrics-100-timesTime: ✅ 135.859µs (SLO: <150.000µs -9.4%) vs baseline: -1.0% Memory: ✅ 34.859MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +5.0% ✅ 1-rate-metric-1-timesTime: ✅ 3.132µs (SLO: <20.000µs 📉 -84.3%) vs baseline: +0.5% Memory: ✅ 34.800MB (SLO: <35.500MB 🟡 -2.0%) vs baseline: +4.6% ✅ 1-rate-metrics-100-timesTime: ✅ 212.882µs (SLO: <250.000µs 📉 -14.8%) vs baseline: +0.2% Memory: ✅ 34.898MB (SLO: <35.500MB 🟡 -1.7%) vs baseline: +5.1% ✅ 100-count-metrics-100-timesTime: ✅ 20.084ms (SLO: <22.000ms -8.7%) vs baseline: -0.1% Memory: ✅ 34.839MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +4.7% ✅ 100-distribution-metrics-100-timesTime: ✅ 2.242ms (SLO: <2.550ms 📉 -12.1%) vs baseline: +0.4% Memory: ✅ 34.800MB (SLO: <35.500MB 🟡 -2.0%) vs baseline: +3.9% ✅ 100-gauge-metrics-100-timesTime: ✅ 1.428ms (SLO: <1.550ms -7.9%) vs baseline: +0.7% Memory: ✅ 34.898MB (SLO: <35.500MB 🟡 -1.7%) vs baseline: +5.0% ✅ 100-rate-metrics-100-timesTime: ✅ 2.217ms (SLO: <2.550ms 📉 -13.1%) vs baseline: +1.4% Memory: ✅ 34.800MB (SLO: <35.500MB 🟡 -2.0%) vs baseline: +4.4% ✅ flush-1-metricTime: ✅ 4.549µs (SLO: <20.000µs 📉 -77.3%) vs baseline: +0.3% Memory: ✅ 35.173MB (SLO: <35.500MB 🟡 -0.9%) vs baseline: +4.9% ✅ flush-100-metricsTime: ✅ 174.725µs (SLO: <250.000µs 📉 -30.1%) vs baseline: +0.1% Memory: ✅ 35.193MB (SLO: <35.500MB 🟡 -0.9%) vs baseline: +4.9% ✅ flush-1000-metricsTime: ✅ 2.176ms (SLO: <2.500ms 📉 -12.9%) vs baseline: -0.4% Memory: ✅ 36.038MB (SLO: <36.500MB 🟡 -1.3%) vs baseline: +5.0% 🟡 Near SLO Breach (17 suites)🟡 coreapiscenario - 10/10 (1 unstable)
|
johannbotha
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you also clean up the description to make it clear why these changes are being made and a link or screenshot (since retention will remove) on the types of errors we're trying to fix. Something like
Tests were flaky due to:
Race during teardown where processor.shutdown() could be called twice (once by the test fixture and once by global tracer teardown), causing a ServiceStatusError; the new try/except ignores that expected case.
DSM context stored on a thread-local/ContextVar was leaking between tests, and many tests had ad-hoc deletions. Centralizing the cleanup at the fixture start makes the intent explicit and reduces duplicated code and the chance of missed cleanup.
Description
This is just a smaller version of #15758, so that we can merge the root fix now without interfering with any changes that would come from this PR: #15715.
Tests were flaky due to:
Race during teardown where processor.shutdown() could be called twice (once by the test fixture and once by global tracer teardown), causing a ServiceStatusError; the new try/except ignores that expected case.
Testing
Risks
Additional Notes