-
Notifications
You must be signed in to change notification settings - Fork 475
test(profiling): unflake a flaky test #15746
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
|
|
e16501f to
44daa97
Compare
0cb1222 to
b5cbfb3
Compare
Performance SLOsComparing candidate kowalski/test-profiling-unflake-a-flaky-test (ddb1f79) with baseline main (6ea41ee) 📈 Performance Regressions (3 suites)📈 iastaspects - 118/118✅ add_aspectTime: ✅ 17.960µs (SLO: <20.000µs 📉 -10.2%) vs baseline: 📈 +20.4% Memory: ✅ 42.546MB (SLO: <43.250MB 🟡 -1.6%) vs baseline: +4.6% ✅ add_inplace_aspectTime: ✅ 15.018µs (SLO: <20.000µs 📉 -24.9%) vs baseline: +0.9% Memory: ✅ 42.605MB (SLO: <43.250MB 🟡 -1.5%) vs baseline: +4.7% ✅ add_inplace_noaspectTime: ✅ 0.337µs (SLO: <10.000µs 📉 -96.6%) vs baseline: -0.7% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +5.0% ✅ add_noaspectTime: ✅ 0.550µs (SLO: <10.000µs 📉 -94.5%) vs baseline: ~same Memory: ✅ 42.703MB (SLO: <43.500MB 🟡 -1.8%) vs baseline: +5.2% ✅ bytearray_aspectTime: ✅ 18.061µs (SLO: <30.000µs 📉 -39.8%) vs baseline: +0.2% Memory: ✅ 42.664MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +5.0% ✅ bytearray_extend_aspectTime: ✅ 23.888µs (SLO: <30.000µs 📉 -20.4%) vs baseline: +0.7% Memory: ✅ 42.507MB (SLO: <43.500MB -2.3%) vs baseline: +4.9% ✅ bytearray_extend_noaspectTime: ✅ 2.730µs (SLO: <10.000µs 📉 -72.7%) vs baseline: -0.8% Memory: ✅ 42.644MB (SLO: <43.500MB 🟡 -2.0%) vs baseline: +4.7% ✅ bytearray_noaspectTime: ✅ 1.470µs (SLO: <10.000µs 📉 -85.3%) vs baseline: -0.8% Memory: ✅ 42.566MB (SLO: <43.500MB -2.1%) vs baseline: +4.6% ✅ bytes_aspectTime: ✅ 16.658µs (SLO: <20.000µs 📉 -16.7%) vs baseline: +0.9% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +5.0% ✅ bytes_noaspectTime: ✅ 1.420µs (SLO: <10.000µs 📉 -85.8%) vs baseline: +0.9% Memory: ✅ 42.526MB (SLO: <43.500MB -2.2%) vs baseline: +4.8% ✅ bytesio_aspectTime: ✅ 55.582µs (SLO: <70.000µs 📉 -20.6%) vs baseline: ~same Memory: ✅ 42.605MB (SLO: <43.500MB -2.1%) vs baseline: +4.8% ✅ bytesio_noaspectTime: ✅ 3.311µs (SLO: <10.000µs 📉 -66.9%) vs baseline: +0.8% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +4.9% ✅ capitalize_aspectTime: ✅ 14.715µs (SLO: <20.000µs 📉 -26.4%) vs baseline: +0.6% Memory: ✅ 42.507MB (SLO: <43.500MB -2.3%) vs baseline: +4.7% ✅ capitalize_noaspectTime: ✅ 2.590µs (SLO: <10.000µs 📉 -74.1%) vs baseline: +0.2% Memory: ✅ 42.566MB (SLO: <43.500MB -2.1%) vs baseline: +4.8% ✅ casefold_aspectTime: ✅ 14.591µs (SLO: <20.000µs 📉 -27.0%) vs baseline: -0.3% Memory: ✅ 42.566MB (SLO: <43.500MB -2.1%) vs baseline: +5.0% ✅ casefold_noaspectTime: ✅ 3.182µs (SLO: <10.000µs 📉 -68.2%) vs baseline: +0.6% Memory: ✅ 42.487MB (SLO: <43.500MB -2.3%) vs baseline: +4.7% ✅ decode_aspectTime: ✅ 15.716µs (SLO: <30.000µs 📉 -47.6%) vs baseline: +0.8% Memory: ✅ 42.605MB (SLO: <43.500MB -2.1%) vs baseline: +4.9% ✅ decode_noaspectTime: ✅ 1.602µs (SLO: <10.000µs 📉 -84.0%) vs baseline: -0.8% Memory: ✅ 42.507MB (SLO: <43.500MB -2.3%) vs baseline: +4.6% ✅ encode_aspectTime: ✅ 18.235µs (SLO: <30.000µs 📉 -39.2%) vs baseline: 📈 +23.3% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +5.0% ✅ encode_noaspectTime: ✅ 1.490µs (SLO: <10.000µs 📉 -85.1%) vs baseline: -0.3% Memory: ✅ 42.566MB (SLO: <43.500MB -2.1%) vs baseline: +4.9% ✅ format_aspectTime: ✅ 171.017µs (SLO: <200.000µs 📉 -14.5%) vs baseline: -0.1% Memory: ✅ 42.605MB (SLO: <43.250MB 🟡 -1.5%) vs baseline: +4.8% ✅ format_map_aspectTime: ✅ 191.463µs (SLO: <200.000µs -4.3%) vs baseline: +0.1% Memory: ✅ 42.644MB (SLO: <43.500MB 🟡 -2.0%) vs baseline: +5.0% ✅ format_map_noaspectTime: ✅ 3.803µs (SLO: <10.000µs 📉 -62.0%) vs baseline: +1.2% Memory: ✅ 42.585MB (SLO: <43.250MB 🟡 -1.5%) vs baseline: +4.8% ✅ format_noaspectTime: ✅ 3.171µs (SLO: <10.000µs 📉 -68.3%) vs baseline: -0.1% Memory: ✅ 42.625MB (SLO: <43.250MB 🟡 -1.4%) vs baseline: +5.0% ✅ index_aspectTime: ✅ 15.416µs (SLO: <20.000µs 📉 -22.9%) vs baseline: +1.3% Memory: ✅ 42.605MB (SLO: <43.250MB 🟡 -1.5%) vs baseline: +4.8% ✅ index_noaspectTime: ✅ 0.462µs (SLO: <10.000µs 📉 -95.4%) vs baseline: -0.3% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +4.9% ✅ join_aspectTime: ✅ 17.124µs (SLO: <20.000µs 📉 -14.4%) vs baseline: +0.4% Memory: ✅ 42.566MB (SLO: <43.500MB -2.1%) vs baseline: +4.8% ✅ join_noaspectTime: ✅ 1.547µs (SLO: <10.000µs 📉 -84.5%) vs baseline: -0.6% Memory: ✅ 42.566MB (SLO: <43.250MB 🟡 -1.6%) vs baseline: +4.8% ✅ ljust_aspectTime: ✅ 20.729µs (SLO: <30.000µs 📉 -30.9%) vs baseline: -0.8% Memory: ✅ 42.625MB (SLO: <43.250MB 🟡 -1.4%) vs baseline: +5.0% ✅ ljust_noaspectTime: ✅ 2.707µs (SLO: <10.000µs 📉 -72.9%) vs baseline: ~same Memory: ✅ 42.605MB (SLO: <43.250MB 🟡 -1.5%) vs baseline: +5.0% ✅ lower_aspectTime: ✅ 17.989µs (SLO: <30.000µs 📉 -40.0%) vs baseline: +0.2% Memory: ✅ 42.644MB (SLO: <43.500MB 🟡 -2.0%) vs baseline: +4.7% ✅ lower_noaspectTime: ✅ 2.430µs (SLO: <10.000µs 📉 -75.7%) vs baseline: +0.2% Memory: ✅ 42.664MB (SLO: <43.250MB 🟡 -1.4%) vs baseline: +5.1% ✅ lstrip_aspectTime: ✅ 17.753µs (SLO: <30.000µs 📉 -40.8%) vs baseline: +0.7% Memory: ✅ 42.684MB (SLO: <43.250MB 🟡 -1.3%) vs baseline: +5.2% ✅ lstrip_noaspectTime: ✅ 1.869µs (SLO: <10.000µs 📉 -81.3%) vs baseline: -0.2% Memory: ✅ 42.605MB (SLO: <43.500MB -2.1%) vs baseline: +5.0% ✅ modulo_aspectTime: ✅ 166.834µs (SLO: <200.000µs 📉 -16.6%) vs baseline: +0.5% Memory: ✅ 42.684MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +5.1% ✅ modulo_aspect_for_bytearray_bytearrayTime: ✅ 174.638µs (SLO: <200.000µs 📉 -12.7%) vs baseline: ~same Memory: ✅ 42.585MB (SLO: <43.500MB -2.1%) vs baseline: +4.8% ✅ modulo_aspect_for_bytesTime: ✅ 168.770µs (SLO: <200.000µs 📉 -15.6%) vs baseline: -0.1% Memory: ✅ 42.684MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +4.7% ✅ modulo_aspect_for_bytes_bytearrayTime: ✅ 172.459µs (SLO: <200.000µs 📉 -13.8%) vs baseline: +0.5% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +4.6% ✅ modulo_noaspectTime: ✅ 3.817µs (SLO: <10.000µs 📉 -61.8%) vs baseline: +4.0% Memory: ✅ 42.526MB (SLO: <43.500MB -2.2%) vs baseline: +4.7% ✅ replace_aspectTime: ✅ 212.288µs (SLO: <300.000µs 📉 -29.2%) vs baseline: +0.4% Memory: ✅ 42.664MB (SLO: <44.000MB -3.0%) vs baseline: +4.9% ✅ replace_noaspectTime: ✅ 2.918µs (SLO: <10.000µs 📉 -70.8%) vs baseline: +0.3% Memory: ✅ 42.605MB (SLO: <43.500MB -2.1%) vs baseline: +4.8% ✅ repr_aspectTime: ✅ 1.418µs (SLO: <10.000µs 📉 -85.8%) vs baseline: -0.5% Memory: ✅ 42.605MB (SLO: <43.500MB -2.1%) vs baseline: +5.0% ✅ repr_noaspectTime: ✅ 0.527µs (SLO: <10.000µs 📉 -94.7%) vs baseline: ~same Memory: ✅ 42.585MB (SLO: <43.500MB -2.1%) vs baseline: +4.9% ✅ rstrip_aspectTime: ✅ 19.049µs (SLO: <30.000µs 📉 -36.5%) vs baseline: ~same Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +4.9% ✅ rstrip_noaspectTime: ✅ 2.058µs (SLO: <10.000µs 📉 -79.4%) vs baseline: +6.2% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +4.8% ✅ slice_aspectTime: ✅ 15.953µs (SLO: <20.000µs 📉 -20.2%) vs baseline: +0.8% Memory: ✅ 42.664MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +5.1% ✅ slice_noaspectTime: ✅ 0.594µs (SLO: <10.000µs 📉 -94.1%) vs baseline: -1.2% Memory: ✅ 42.566MB (SLO: <43.500MB -2.1%) vs baseline: +4.8% ✅ stringio_aspectTime: ✅ 54.244µs (SLO: <80.000µs 📉 -32.2%) vs baseline: +0.5% Memory: ✅ 42.664MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +5.0% ✅ stringio_noaspectTime: ✅ 3.653µs (SLO: <10.000µs 📉 -63.5%) vs baseline: +0.9% Memory: ✅ 42.566MB (SLO: <43.500MB -2.1%) vs baseline: +4.7% ✅ strip_aspectTime: ✅ 17.704µs (SLO: <20.000µs 📉 -11.5%) vs baseline: +0.3% Memory: ✅ 42.605MB (SLO: <43.500MB -2.1%) vs baseline: +4.9% ✅ strip_noaspectTime: ✅ 1.880µs (SLO: <10.000µs 📉 -81.2%) vs baseline: -0.2% Memory: ✅ 42.664MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +5.0% ✅ swapcase_aspectTime: ✅ 18.450µs (SLO: <30.000µs 📉 -38.5%) vs baseline: -0.1% Memory: ✅ 42.644MB (SLO: <43.500MB 🟡 -2.0%) vs baseline: +5.1% ✅ swapcase_noaspectTime: ✅ 2.822µs (SLO: <10.000µs 📉 -71.8%) vs baseline: +1.2% Memory: ✅ 42.605MB (SLO: <43.500MB -2.1%) vs baseline: +4.9% ✅ title_aspectTime: ✅ 21.968µs (SLO: <30.000µs 📉 -26.8%) vs baseline: 📈 +21.1% Memory: ✅ 42.684MB (SLO: <43.000MB 🟡 -0.7%) vs baseline: +4.9% ✅ title_noaspectTime: ✅ 2.691µs (SLO: <10.000µs 📉 -73.1%) vs baseline: +0.9% Memory: ✅ 42.585MB (SLO: <43.500MB -2.1%) vs baseline: +4.8% ✅ translate_aspectTime: ✅ 20.632µs (SLO: <30.000µs 📉 -31.2%) vs baseline: +0.3% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +4.7% ✅ translate_noaspectTime: ✅ 4.330µs (SLO: <10.000µs 📉 -56.7%) vs baseline: ~same Memory: ✅ 42.585MB (SLO: <43.500MB -2.1%) vs baseline: +4.9% ✅ upper_aspectTime: ✅ 17.942µs (SLO: <30.000µs 📉 -40.2%) vs baseline: ~same Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +4.9% ✅ upper_noaspectTime: ✅ 2.443µs (SLO: <10.000µs 📉 -75.6%) vs baseline: ~same Memory: ✅ 42.585MB (SLO: <43.500MB -2.1%) vs baseline: +5.0% 📈 iastaspectsospath - 24/24✅ ospathbasename_aspectTime: ✅ 5.127µs (SLO: <10.000µs 📉 -48.7%) vs baseline: 📈 +20.6% Memory: ✅ 41.563MB (SLO: <43.500MB -4.5%) vs baseline: +5.2% ✅ ospathbasename_noaspectTime: ✅ 4.292µs (SLO: <10.000µs 📉 -57.1%) vs baseline: -1.1% Memory: ✅ 41.563MB (SLO: <43.500MB -4.5%) vs baseline: +4.9% ✅ ospathjoin_aspectTime: ✅ 6.284µs (SLO: <10.000µs 📉 -37.2%) vs baseline: +1.2% Memory: ✅ 41.622MB (SLO: <43.500MB -4.3%) vs baseline: +5.3% ✅ ospathjoin_noaspectTime: ✅ 6.302µs (SLO: <10.000µs 📉 -37.0%) vs baseline: -0.2% Memory: ✅ 41.504MB (SLO: <43.500MB -4.6%) vs baseline: +5.0% ✅ ospathnormcase_aspectTime: ✅ 3.573µs (SLO: <10.000µs 📉 -64.3%) vs baseline: -0.6% Memory: ✅ 41.583MB (SLO: <43.500MB -4.4%) vs baseline: +5.2% ✅ ospathnormcase_noaspectTime: ✅ 3.646µs (SLO: <10.000µs 📉 -63.5%) vs baseline: +1.6% Memory: ✅ 41.484MB (SLO: <43.500MB -4.6%) vs baseline: +4.9% ✅ ospathsplit_aspectTime: ✅ 4.900µs (SLO: <10.000µs 📉 -51.0%) vs baseline: -0.5% Memory: ✅ 41.366MB (SLO: <43.500MB -4.9%) vs baseline: +4.5% ✅ ospathsplit_noaspectTime: ✅ 4.998µs (SLO: <10.000µs 📉 -50.0%) vs baseline: ~same Memory: ✅ 41.524MB (SLO: <43.500MB -4.5%) vs baseline: +4.9% ✅ ospathsplitdrive_aspectTime: ✅ 3.761µs (SLO: <10.000µs 📉 -62.4%) vs baseline: -0.7% Memory: ✅ 41.583MB (SLO: <43.500MB -4.4%) vs baseline: +5.3% ✅ ospathsplitdrive_noaspectTime: ✅ 0.752µs (SLO: <10.000µs 📉 -92.5%) vs baseline: -0.1% Memory: ✅ 41.504MB (SLO: <43.500MB -4.6%) vs baseline: +4.6% ✅ ospathsplitext_aspectTime: ✅ 4.615µs (SLO: <10.000µs 📉 -53.8%) vs baseline: +0.4% Memory: ✅ 41.602MB (SLO: <43.500MB -4.4%) vs baseline: +5.0% ✅ ospathsplitext_noaspectTime: ✅ 4.649µs (SLO: <10.000µs 📉 -53.5%) vs baseline: -0.3% Memory: ✅ 41.543MB (SLO: <43.500MB -4.5%) vs baseline: +5.1% 📈 telemetryaddmetric - 30/30✅ 1-count-metric-1-timesTime: ✅ 3.349µs (SLO: <20.000µs 📉 -83.3%) vs baseline: 📈 +13.5% Memory: ✅ 34.977MB (SLO: <35.500MB 🟡 -1.5%) vs baseline: +5.2% ✅ 1-count-metrics-100-timesTime: ✅ 199.176µs (SLO: <220.000µs -9.5%) vs baseline: +0.3% Memory: ✅ 34.937MB (SLO: <35.500MB 🟡 -1.6%) vs baseline: +5.3% ✅ 1-distribution-metric-1-timesTime: ✅ 3.295µs (SLO: <20.000µs 📉 -83.5%) vs baseline: -0.3% Memory: ✅ 34.878MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +5.0% ✅ 1-distribution-metrics-100-timesTime: ✅ 212.220µs (SLO: <230.000µs -7.7%) vs baseline: -0.4% Memory: ✅ 34.957MB (SLO: <35.500MB 🟡 -1.5%) vs baseline: +4.8% ✅ 1-gauge-metric-1-timesTime: ✅ 2.167µs (SLO: <20.000µs 📉 -89.2%) vs baseline: -0.6% Memory: ✅ 34.898MB (SLO: <35.500MB 🟡 -1.7%) vs baseline: +4.8% ✅ 1-gauge-metrics-100-timesTime: ✅ 136.205µs (SLO: <150.000µs -9.2%) vs baseline: +0.4% Memory: ✅ 34.878MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +5.0% ✅ 1-rate-metric-1-timesTime: ✅ 3.102µs (SLO: <20.000µs 📉 -84.5%) vs baseline: -0.4% Memory: ✅ 34.898MB (SLO: <35.500MB 🟡 -1.7%) vs baseline: +5.0% ✅ 1-rate-metrics-100-timesTime: ✅ 213.621µs (SLO: <250.000µs 📉 -14.6%) vs baseline: +1.1% Memory: ✅ 34.918MB (SLO: <35.500MB 🟡 -1.6%) vs baseline: +5.2% ✅ 100-count-metrics-100-timesTime: ✅ 20.199ms (SLO: <22.000ms -8.2%) vs baseline: -0.3% Memory: ✅ 34.957MB (SLO: <35.500MB 🟡 -1.5%) vs baseline: +5.2% ✅ 100-distribution-metrics-100-timesTime: ✅ 2.249ms (SLO: <2.550ms 📉 -11.8%) vs baseline: +1.2% Memory: ✅ 35.212MB (SLO: <35.500MB 🟡 -0.8%) vs baseline: +4.9% ✅ 100-gauge-metrics-100-timesTime: ✅ 1.412ms (SLO: <1.550ms -8.9%) vs baseline: +1.2% Memory: ✅ 34.839MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +4.7% ✅ 100-rate-metrics-100-timesTime: ✅ 2.187ms (SLO: <2.550ms 📉 -14.2%) vs baseline: -0.2% Memory: ✅ 34.878MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +4.9% ✅ flush-1-metricTime: ✅ 4.592µs (SLO: <20.000µs 📉 -77.0%) vs baseline: +0.5% Memory: ✅ 35.232MB (SLO: <35.500MB 🟡 -0.8%) vs baseline: +4.8% ✅ flush-100-metricsTime: ✅ 172.788µs (SLO: <250.000µs 📉 -30.9%) vs baseline: +0.2% Memory: ✅ 35.291MB (SLO: <35.500MB 🟡 -0.6%) vs baseline: +4.9% ✅ flush-1000-metricsTime: ✅ 2.165ms (SLO: <2.500ms 📉 -13.4%) vs baseline: -0.2% Memory: ✅ 36.019MB (SLO: <36.500MB 🟡 -1.3%) vs baseline: +5.0% 🟡 Near SLO Breach (16 suites)🟡 coreapiscenario - 10/10 (1 unstable)
|
8423e92 to
ac34907
Compare
d1883c9 to
da38dd1
Compare
da38dd1 to
ddb1f79
Compare
Description
This is an attempt at making some of our tests non-flaky by flushing any pre-existing state from
ddup.We are currently seeing some test failures on
mainwith messages likeMy theory is that this
12345is the mock Span ID used in this test that is leaking:dd-trace-py/tests/profiling/exporter/test_ddup.py
Lines 103 to 106 in 3841a70
I fixed this by:
test_stack.py(since we know for sure we aren't interested in them).These three fixes make the test now non-flaky.
The PR also makes the necessary changes in
pprof_utilsto make it possible to implement the fixes.See test runs in Datadog