-
Notifications
You must be signed in to change notification settings - Fork 475
ci(iast): fix flaky benchmark test [backport 4.1] #15765
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: 4.1
Are you sure you want to change the base?
Conversation
Fix flakyness in `lstrip_aspect` https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-py/-/jobs/1310950874 (cherry picked from commit 2484164) Signed-off-by: Alberto Vara <[email protected]>
|
|
Bootstrap import analysisComparison of import times between this PR and base. SummaryThe average import time from this PR is: 256 ± 6 ms. The average import time from base is: 256 ± 5 ms. The import time difference between this PR and base is: -0.1 ± 0.2 ms. The difference is not statistically significant (z = -0.54). Import time breakdownThe following import paths have shrunk:
|
Performance SLOsComparing candidate backport-15756-to-4.1 (42929da) with baseline 4.1 (15f91ba) 📈 Performance Regressions (4 suites)📈 iastaspects - 118/118✅ add_aspectTime: ✅ 18.194µs (SLO: <20.000µs -9.0%) vs baseline: 📈 +21.0% Memory: ✅ 42.979MB (SLO: <43.250MB 🟡 -0.6%) vs baseline: +4.8% ✅ add_inplace_aspectTime: ✅ 14.960µs (SLO: <20.000µs 📉 -25.2%) vs baseline: ~same Memory: ✅ 43.037MB (SLO: <43.250MB 🟡 -0.5%) vs baseline: +4.7% ✅ add_inplace_noaspectTime: ✅ 0.337µs (SLO: <10.000µs 📉 -96.6%) vs baseline: ~same Memory: ✅ 42.979MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.9% ✅ add_noaspectTime: ✅ 0.552µs (SLO: <10.000µs 📉 -94.5%) vs baseline: +0.6% Memory: ✅ 43.057MB (SLO: <43.500MB 🟡 -1.0%) vs baseline: +5.1% ✅ bytearray_aspectTime: ✅ 17.891µs (SLO: <30.000µs 📉 -40.4%) vs baseline: -0.5% Memory: ✅ 43.037MB (SLO: <43.500MB 🟡 -1.1%) vs baseline: +5.0% ✅ bytearray_extend_aspectTime: ✅ 23.618µs (SLO: <30.000µs 📉 -21.3%) vs baseline: -0.6% Memory: ✅ 42.979MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.7% ✅ bytearray_extend_noaspectTime: ✅ 2.675µs (SLO: <10.000µs 📉 -73.3%) vs baseline: +2.1% Memory: ✅ 42.979MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.9% ✅ bytearray_noaspectTime: ✅ 1.448µs (SLO: <10.000µs 📉 -85.5%) vs baseline: -1.1% Memory: ✅ 42.998MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +5.0% ✅ bytes_aspectTime: ✅ 16.736µs (SLO: <20.000µs 📉 -16.3%) vs baseline: +0.6% Memory: ✅ 42.998MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.8% ✅ bytes_noaspectTime: ✅ 1.391µs (SLO: <10.000µs 📉 -86.1%) vs baseline: +0.2% Memory: ✅ 42.998MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.9% ✅ bytesio_aspectTime: ✅ 54.978µs (SLO: <70.000µs 📉 -21.5%) vs baseline: -0.4% Memory: ✅ 42.959MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.8% ✅ bytesio_noaspectTime: ✅ 3.210µs (SLO: <10.000µs 📉 -67.9%) vs baseline: +0.4% Memory: ✅ 42.998MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.8% ✅ capitalize_aspectTime: ✅ 14.597µs (SLO: <20.000µs 📉 -27.0%) vs baseline: -0.3% Memory: ✅ 43.116MB (SLO: <43.500MB 🟡 -0.9%) vs baseline: +5.3% ✅ capitalize_noaspectTime: ✅ 2.574µs (SLO: <10.000µs 📉 -74.3%) vs baseline: +0.3% Memory: ✅ 42.939MB (SLO: <43.500MB 🟡 -1.3%) vs baseline: +4.8% ✅ casefold_aspectTime: ✅ 14.629µs (SLO: <20.000µs 📉 -26.9%) vs baseline: ~same Memory: ✅ 43.018MB (SLO: <43.500MB 🟡 -1.1%) vs baseline: +4.8% ✅ casefold_noaspectTime: ✅ 3.090µs (SLO: <10.000µs 📉 -69.1%) vs baseline: +1.1% Memory: ✅ 43.018MB (SLO: <43.500MB 🟡 -1.1%) vs baseline: +4.7% ✅ decode_aspectTime: ✅ 15.530µs (SLO: <30.000µs 📉 -48.2%) vs baseline: -0.3% Memory: ✅ 43.077MB (SLO: <43.500MB 🟡 -1.0%) vs baseline: +5.3% ✅ decode_noaspectTime: ✅ 1.593µs (SLO: <10.000µs 📉 -84.1%) vs baseline: ~same Memory: ✅ 42.979MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.7% ✅ encode_aspectTime: ✅ 18.178µs (SLO: <30.000µs 📉 -39.4%) vs baseline: 📈 +22.1% Memory: ✅ 42.861MB (SLO: <43.500MB 🟡 -1.5%) vs baseline: +4.2% ✅ encode_noaspectTime: ✅ 1.472µs (SLO: <10.000µs 📉 -85.3%) vs baseline: +0.3% Memory: ✅ 43.018MB (SLO: <43.500MB 🟡 -1.1%) vs baseline: +4.9% ✅ format_aspectTime: ✅ 170.925µs (SLO: <200.000µs 📉 -14.5%) vs baseline: ~same Memory: ✅ 43.057MB (SLO: <43.250MB 🟡 -0.4%) vs baseline: +4.8% ✅ format_map_aspectTime: ✅ 191.541µs (SLO: <200.000µs -4.2%) vs baseline: -0.1% Memory: ✅ 42.998MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.8% ✅ format_map_noaspectTime: ✅ 3.734µs (SLO: <10.000µs 📉 -62.7%) vs baseline: -0.1% Memory: ✅ 43.018MB (SLO: <43.250MB 🟡 -0.5%) vs baseline: +4.7% ✅ format_noaspectTime: ✅ 3.144µs (SLO: <10.000µs 📉 -68.6%) vs baseline: -0.3% Memory: ✅ 42.939MB (SLO: <43.250MB 🟡 -0.7%) vs baseline: +4.5% ✅ index_aspectTime: ✅ 15.275µs (SLO: <20.000µs 📉 -23.6%) vs baseline: -0.2% Memory: ✅ 42.979MB (SLO: <43.250MB 🟡 -0.6%) vs baseline: +4.9% ✅ index_noaspectTime: ✅ 0.462µs (SLO: <10.000µs 📉 -95.4%) vs baseline: -0.3% Memory: ✅ 43.037MB (SLO: <43.500MB 🟡 -1.1%) vs baseline: +4.8% ✅ join_aspectTime: ✅ 16.976µs (SLO: <20.000µs 📉 -15.1%) vs baseline: ~same Memory: ✅ 42.841MB (SLO: <43.500MB 🟡 -1.5%) vs baseline: +4.6% ✅ join_noaspectTime: ✅ 1.491µs (SLO: <10.000µs 📉 -85.1%) vs baseline: ~same Memory: ✅ 42.998MB (SLO: <43.250MB 🟡 -0.6%) vs baseline: +4.9% ✅ ljust_aspectTime: ✅ 20.602µs (SLO: <30.000µs 📉 -31.3%) vs baseline: -1.0% Memory: ✅ 43.018MB (SLO: <43.250MB 🟡 -0.5%) vs baseline: +4.9% ✅ ljust_noaspectTime: ✅ 2.659µs (SLO: <10.000µs 📉 -73.4%) vs baseline: +0.2% Memory: ✅ 43.096MB (SLO: <43.250MB 🟡 -0.4%) vs baseline: +5.0% ✅ lower_aspectTime: ✅ 17.897µs (SLO: <30.000µs 📉 -40.3%) vs baseline: +1.0% Memory: ✅ 43.077MB (SLO: <43.500MB 🟡 -1.0%) vs baseline: +5.1% ✅ lower_noaspectTime: ✅ 2.389µs (SLO: <10.000µs 📉 -76.1%) vs baseline: +0.7% Memory: ✅ 43.018MB (SLO: <43.250MB 🟡 -0.5%) vs baseline: +5.1% ✅ lstrip_aspectTime: ✅ 17.641µs (SLO: <30.000µs 📉 -41.2%) vs baseline: +0.2% Memory: ✅ 43.116MB (SLO: <43.250MB 🟡 -0.3%) vs baseline: +5.3% ✅ lstrip_noaspectTime: ✅ 1.816µs (SLO: <10.000µs 📉 -81.8%) vs baseline: -1.5% Memory: ✅ 43.018MB (SLO: <43.500MB 🟡 -1.1%) vs baseline: +4.9% ✅ modulo_aspectTime: ✅ 166.886µs (SLO: <200.000µs 📉 -16.6%) vs baseline: +0.3% Memory: ✅ 43.057MB (SLO: <43.500MB 🟡 -1.0%) vs baseline: +4.9% ✅ modulo_aspect_for_bytearray_bytearrayTime: ✅ 180.130µs (SLO: <200.000µs -9.9%) vs baseline: +3.5% Memory: ✅ 43.096MB (SLO: <43.500MB 🟡 -0.9%) vs baseline: +4.9% ✅ modulo_aspect_for_bytesTime: ✅ 168.439µs (SLO: <200.000µs 📉 -15.8%) vs baseline: +0.5% Memory: ✅ 43.136MB (SLO: <43.500MB 🟡 -0.8%) vs baseline: +5.0% ✅ modulo_aspect_for_bytes_bytearrayTime: ✅ 171.664µs (SLO: <200.000µs 📉 -14.2%) vs baseline: -0.3% Memory: ✅ 43.037MB (SLO: <43.500MB 🟡 -1.1%) vs baseline: +4.6% ✅ modulo_noaspectTime: ✅ 3.696µs (SLO: <10.000µs 📉 -63.0%) vs baseline: +0.9% Memory: ✅ 42.959MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.8% ✅ replace_aspectTime: ✅ 214.658µs (SLO: <300.000µs 📉 -28.4%) vs baseline: ~same Memory: ✅ 43.116MB (SLO: <44.000MB -2.0%) vs baseline: +5.1% ✅ replace_noaspectTime: ✅ 5.180µs (SLO: <10.000µs 📉 -48.2%) vs baseline: +0.5% Memory: ✅ 42.959MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.6% ✅ repr_aspectTime: ✅ 1.394µs (SLO: <10.000µs 📉 -86.1%) vs baseline: +1.2% Memory: ✅ 42.959MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.7% ✅ repr_noaspectTime: ✅ 0.527µs (SLO: <10.000µs 📉 -94.7%) vs baseline: ~same Memory: ✅ 42.979MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.9% ✅ rstrip_aspectTime: ✅ 19.089µs (SLO: <30.000µs 📉 -36.4%) vs baseline: +0.9% Memory: ✅ 43.018MB (SLO: <43.500MB 🟡 -1.1%) vs baseline: +5.0% ✅ rstrip_noaspectTime: ✅ 1.919µs (SLO: <10.000µs 📉 -80.8%) vs baseline: +3.2% Memory: ✅ 42.998MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +5.0% ✅ slice_aspectTime: ✅ 15.933µs (SLO: <20.000µs 📉 -20.3%) vs baseline: -0.5% Memory: ✅ 43.037MB (SLO: <43.500MB 🟡 -1.1%) vs baseline: +5.0% ✅ slice_noaspectTime: ✅ 0.597µs (SLO: <10.000µs 📉 -94.0%) vs baseline: -0.7% Memory: ✅ 42.939MB (SLO: <43.500MB 🟡 -1.3%) vs baseline: +4.8% ✅ stringio_aspectTime: ✅ 53.610µs (SLO: <80.000µs 📉 -33.0%) vs baseline: ~same Memory: ✅ 42.959MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.7% ✅ stringio_noaspectTime: ✅ 3.522µs (SLO: <10.000µs 📉 -64.8%) vs baseline: +0.4% Memory: ✅ 42.998MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +5.0% ✅ strip_aspectTime: ✅ 17.616µs (SLO: <20.000µs 📉 -11.9%) vs baseline: -0.3% Memory: ✅ 43.057MB (SLO: <43.500MB 🟡 -1.0%) vs baseline: +5.0% ✅ strip_noaspectTime: ✅ 1.856µs (SLO: <10.000µs 📉 -81.4%) vs baseline: +1.2% Memory: ✅ 42.979MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.8% ✅ swapcase_aspectTime: ✅ 18.356µs (SLO: <30.000µs 📉 -38.8%) vs baseline: ~same Memory: ✅ 42.998MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.8% ✅ swapcase_noaspectTime: ✅ 2.745µs (SLO: <10.000µs 📉 -72.5%) vs baseline: +0.3% Memory: ✅ 42.979MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.7% ✅ title_aspectTime: ✅ 18.207µs (SLO: <30.000µs 📉 -39.3%) vs baseline: +0.4% Memory: ✅ 42.998MB (SLO: <43.000MB 🟡 ~same) vs baseline: +4.9% ✅ title_noaspectTime: ✅ 2.637µs (SLO: <10.000µs 📉 -73.6%) vs baseline: -0.3% Memory: ✅ 42.959MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.9% ✅ translate_aspectTime: ✅ 23.954µs (SLO: <30.000µs 📉 -20.2%) vs baseline: 📈 +17.2% Memory: ✅ 42.979MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +5.0% ✅ translate_noaspectTime: ✅ 4.248µs (SLO: <10.000µs 📉 -57.5%) vs baseline: -0.6% Memory: ✅ 43.018MB (SLO: <43.500MB 🟡 -1.1%) vs baseline: +4.9% ✅ upper_aspectTime: ✅ 17.905µs (SLO: <30.000µs 📉 -40.3%) vs baseline: +0.7% Memory: ✅ 42.979MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.9% ✅ upper_noaspectTime: ✅ 2.421µs (SLO: <10.000µs 📉 -75.8%) vs baseline: +2.3% Memory: ✅ 42.979MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.7% 📈 iastaspectsospath - 24/24✅ ospathbasename_aspectTime: ✅ 5.249µs (SLO: <10.000µs 📉 -47.5%) vs baseline: 📈 +22.8% Memory: ✅ 41.465MB (SLO: <43.500MB -4.7%) vs baseline: +4.9% ✅ ospathbasename_noaspectTime: ✅ 4.343µs (SLO: <10.000µs 📉 -56.6%) vs baseline: ~same Memory: ✅ 41.386MB (SLO: <43.500MB -4.9%) vs baseline: +4.6% ✅ ospathjoin_aspectTime: ✅ 6.204µs (SLO: <10.000µs 📉 -38.0%) vs baseline: +1.0% Memory: ✅ 41.406MB (SLO: <43.500MB -4.8%) vs baseline: +4.8% ✅ ospathjoin_noaspectTime: ✅ 6.197µs (SLO: <10.000µs 📉 -38.0%) vs baseline: -0.8% Memory: ✅ 41.406MB (SLO: <43.500MB -4.8%) vs baseline: +4.6% ✅ ospathnormcase_aspectTime: ✅ 3.525µs (SLO: <10.000µs 📉 -64.7%) vs baseline: -0.5% Memory: ✅ 41.484MB (SLO: <43.500MB -4.6%) vs baseline: +4.9% ✅ ospathnormcase_noaspectTime: ✅ 3.594µs (SLO: <10.000µs 📉 -64.1%) vs baseline: ~same Memory: ✅ 41.465MB (SLO: <43.500MB -4.7%) vs baseline: +5.1% ✅ ospathsplit_aspectTime: ✅ 4.883µs (SLO: <10.000µs 📉 -51.2%) vs baseline: +0.5% Memory: ✅ 41.386MB (SLO: <43.500MB -4.9%) vs baseline: +4.9% ✅ ospathsplit_noaspectTime: ✅ 4.977µs (SLO: <10.000µs 📉 -50.2%) vs baseline: ~same Memory: ✅ 41.425MB (SLO: <43.500MB -4.8%) vs baseline: +5.0% ✅ ospathsplitdrive_aspectTime: ✅ 3.798µs (SLO: <10.000µs 📉 -62.0%) vs baseline: +0.7% Memory: ✅ 41.386MB (SLO: <43.500MB -4.9%) vs baseline: +4.9% ✅ ospathsplitdrive_noaspectTime: ✅ 0.757µs (SLO: <10.000µs 📉 -92.4%) vs baseline: +1.0% Memory: ✅ 41.406MB (SLO: <43.500MB -4.8%) vs baseline: +4.8% ✅ ospathsplitext_aspectTime: ✅ 4.653µs (SLO: <10.000µs 📉 -53.5%) vs baseline: -0.1% Memory: ✅ 41.386MB (SLO: <43.500MB -4.9%) vs baseline: +4.7% ✅ ospathsplitext_noaspectTime: ✅ 4.704µs (SLO: <10.000µs 📉 -53.0%) vs baseline: ~same Memory: ✅ 41.524MB (SLO: <43.500MB -4.5%) vs baseline: +5.2% 📈 iastaspectssplit - 12/12✅ rsplit_aspectTime: ✅ 1.690µs (SLO: <10.000µs 📉 -83.1%) vs baseline: 📈 +10.2% Memory: ✅ 41.425MB (SLO: <43.500MB -4.8%) vs baseline: +4.8% ✅ rsplit_noaspectTime: ✅ 1.612µs (SLO: <10.000µs 📉 -83.9%) vs baseline: ~same Memory: ✅ 41.425MB (SLO: <43.500MB -4.8%) vs baseline: +4.8% ✅ split_aspectTime: ✅ 1.565µs (SLO: <10.000µs 📉 -84.4%) vs baseline: +0.3% Memory: ✅ 41.465MB (SLO: <43.500MB -4.7%) vs baseline: +5.0% ✅ split_noaspectTime: ✅ 1.612µs (SLO: <10.000µs 📉 -83.9%) vs baseline: -0.6% Memory: ✅ 41.366MB (SLO: <43.500MB -4.9%) vs baseline: +4.7% ✅ splitlines_aspectTime: ✅ 1.509µs (SLO: <10.000µs 📉 -84.9%) vs baseline: -0.8% Memory: ✅ 41.465MB (SLO: <43.500MB -4.7%) vs baseline: +5.0% ✅ splitlines_noaspectTime: ✅ 1.563µs (SLO: <10.000µs 📉 -84.4%) vs baseline: -0.4% Memory: ✅ 41.445MB (SLO: <43.500MB -4.7%) vs baseline: +4.8% 📈 telemetryaddmetric - 30/30✅ 1-count-metric-1-timesTime: ✅ 3.380µs (SLO: <20.000µs 📉 -83.1%) vs baseline: 📈 +12.6% Memory: ✅ 34.918MB (SLO: <35.500MB 🟡 -1.6%) vs baseline: +5.1% ✅ 1-count-metrics-100-timesTime: ✅ 202.251µs (SLO: <220.000µs -8.1%) vs baseline: +0.4% Memory: ✅ 34.918MB (SLO: <35.500MB 🟡 -1.6%) vs baseline: +5.2% ✅ 1-distribution-metric-1-timesTime: ✅ 3.370µs (SLO: <20.000µs 📉 -83.2%) vs baseline: +0.6% Memory: ✅ 34.800MB (SLO: <35.500MB 🟡 -2.0%) vs baseline: +4.9% ✅ 1-distribution-metrics-100-timesTime: ✅ 220.806µs (SLO: <230.000µs -4.0%) vs baseline: +1.4% Memory: ✅ 34.878MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +4.9% ✅ 1-gauge-metric-1-timesTime: ✅ 2.184µs (SLO: <20.000µs 📉 -89.1%) vs baseline: -0.3% Memory: ✅ 34.800MB (SLO: <35.500MB 🟡 -2.0%) vs baseline: +4.8% ✅ 1-gauge-metrics-100-timesTime: ✅ 136.795µs (SLO: <150.000µs -8.8%) vs baseline: +0.2% Memory: ✅ 34.721MB (SLO: <35.500MB -2.2%) vs baseline: +4.4% ✅ 1-rate-metric-1-timesTime: ✅ 3.159µs (SLO: <20.000µs 📉 -84.2%) vs baseline: -0.3% Memory: ✅ 34.780MB (SLO: <35.500MB -2.0%) vs baseline: +4.3% ✅ 1-rate-metrics-100-timesTime: ✅ 216.206µs (SLO: <250.000µs 📉 -13.5%) vs baseline: ~same Memory: ✅ 34.839MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +4.9% ✅ 100-count-metrics-100-timesTime: ✅ 20.110ms (SLO: <22.000ms -8.6%) vs baseline: -0.5% Memory: ✅ 34.878MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +5.1% ✅ 100-distribution-metrics-100-timesTime: ✅ 2.251ms (SLO: <2.550ms 📉 -11.7%) vs baseline: -2.1% Memory: ✅ 34.839MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +5.2% ✅ 100-gauge-metrics-100-timesTime: ✅ 1.402ms (SLO: <1.550ms -9.5%) vs baseline: +0.2% Memory: ✅ 34.780MB (SLO: <35.500MB -2.0%) vs baseline: +4.6% ✅ 100-rate-metrics-100-timesTime: ✅ 2.221ms (SLO: <2.550ms 📉 -12.9%) vs baseline: +1.1% Memory: ✅ 34.800MB (SLO: <35.500MB 🟡 -2.0%) vs baseline: +5.0% ✅ flush-1-metricTime: ✅ 4.665µs (SLO: <20.000µs 📉 -76.7%) vs baseline: ~same Memory: ✅ 34.780MB (SLO: <35.500MB -2.0%) vs baseline: +4.9% ✅ flush-100-metricsTime: ✅ 175.052µs (SLO: <250.000µs 📉 -30.0%) vs baseline: ~same Memory: ✅ 35.154MB (SLO: <35.500MB 🟡 -1.0%) vs baseline: +5.0% ✅ flush-1000-metricsTime: ✅ 2.186ms (SLO: <2.500ms 📉 -12.6%) vs baseline: +0.6% Memory: ✅ 36.097MB (SLO: <36.500MB 🟡 -1.1%) vs baseline: +5.1% 🟡 Near SLO Breach (15 suites)🟡 coreapiscenario - 10/10 (1 unstable)
|
589d541 to
ad4db3c
Compare
Backport 2484164 from #15756 to 4.1.
Fix flakyness in
lstrip_aspecthttps://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-py/-/jobs/1310950874