-
Notifications
You must be signed in to change notification settings - Fork 475
fix(ffe): register FFE product before fork to receive remote config [FFL-1623] #15761
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
… config FFE was not receiving remote configuration in forking environments (gunicorn, uWSGI) because it was being registered after the fork when user code instantiated DataDogProvider(). The parent process runs the remote config poller and distributes configurations to child workers via shared memory, but it didn't know about FFE_FLAGS product since registration happened in the child. This fix integrates FFE into the product lifecycle pattern used by other products (AppSec, IAST, etc.): - Created ddtrace/internal/openfeature/product.py with lifecycle hooks - start() registers FFE_FLAGS with remote config before fork - restart() restarts subscribers after fork - Registered as entry point in pyproject.toml - Removed on-demand registration from DataDogProvider.initialize() Now FFE is registered during tracer initialization before any fork (when DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED=true), allowing the parent process to recognize and dispatch FFE_FLAGS configurations to child workers.
|
|
Performance SLOsComparing candidate FFL-1623-register-ffe-product (beec3e4) with baseline main (6ea41ee) 📈 Performance Regressions (3 suites)📈 iastaspects - 118/118✅ add_aspectTime: ✅ 18.048µs (SLO: <20.000µs -9.8%) vs baseline: 📈 +21.4% Memory: ✅ 42.605MB (SLO: <43.250MB 🟡 -1.5%) vs baseline: +5.1% ✅ add_inplace_aspectTime: ✅ 14.939µs (SLO: <20.000µs 📉 -25.3%) vs baseline: +0.3% Memory: ✅ 42.723MB (SLO: <43.250MB 🟡 -1.2%) vs baseline: +5.3% ✅ add_inplace_noaspectTime: ✅ 0.337µs (SLO: <10.000µs 📉 -96.6%) vs baseline: -0.5% Memory: ✅ 42.507MB (SLO: <43.500MB -2.3%) vs baseline: +4.7% ✅ add_noaspectTime: ✅ 0.545µs (SLO: <10.000µs 📉 -94.6%) vs baseline: -0.5% Memory: ✅ 42.644MB (SLO: <43.500MB 🟡 -2.0%) vs baseline: +5.0% ✅ bytearray_aspectTime: ✅ 17.939µs (SLO: <30.000µs 📉 -40.2%) vs baseline: ~same Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +5.1% ✅ bytearray_extend_aspectTime: ✅ 23.899µs (SLO: <30.000µs 📉 -20.3%) vs baseline: -0.1% Memory: ✅ 42.526MB (SLO: <43.500MB -2.2%) vs baseline: +4.9% ✅ bytearray_extend_noaspectTime: ✅ 2.756µs (SLO: <10.000µs 📉 -72.4%) vs baseline: +0.5% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +5.0% ✅ bytearray_noaspectTime: ✅ 1.455µs (SLO: <10.000µs 📉 -85.5%) vs baseline: ~same Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +4.8% ✅ bytes_aspectTime: ✅ 16.805µs (SLO: <20.000µs 📉 -16.0%) vs baseline: +1.3% Memory: ✅ 42.507MB (SLO: <43.500MB -2.3%) vs baseline: +4.7% ✅ bytes_noaspectTime: ✅ 1.422µs (SLO: <10.000µs 📉 -85.8%) vs baseline: -0.6% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +5.0% ✅ bytesio_aspectTime: ✅ 55.652µs (SLO: <70.000µs 📉 -20.5%) vs baseline: ~same Memory: ✅ 42.585MB (SLO: <43.500MB -2.1%) vs baseline: +4.9% ✅ bytesio_noaspectTime: ✅ 3.275µs (SLO: <10.000µs 📉 -67.2%) vs baseline: +0.2% Memory: ✅ 42.585MB (SLO: <43.500MB -2.1%) vs baseline: +4.8% ✅ capitalize_aspectTime: ✅ 14.748µs (SLO: <20.000µs 📉 -26.3%) vs baseline: +0.9% Memory: ✅ 42.605MB (SLO: <43.500MB -2.1%) vs baseline: +5.0% ✅ capitalize_noaspectTime: ✅ 2.611µs (SLO: <10.000µs 📉 -73.9%) vs baseline: +1.0% Memory: ✅ 42.566MB (SLO: <43.500MB -2.1%) vs baseline: +4.7% ✅ casefold_aspectTime: ✅ 14.733µs (SLO: <20.000µs 📉 -26.3%) vs baseline: +0.9% Memory: ✅ 42.566MB (SLO: <43.500MB -2.1%) vs baseline: +4.7% ✅ casefold_noaspectTime: ✅ 3.149µs (SLO: <10.000µs 📉 -68.5%) vs baseline: -0.2% Memory: ✅ 42.664MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +4.9% ✅ decode_aspectTime: ✅ 15.643µs (SLO: <30.000µs 📉 -47.9%) vs baseline: -0.3% Memory: ✅ 42.526MB (SLO: <43.500MB -2.2%) vs baseline: +4.6% ✅ decode_noaspectTime: ✅ 1.622µs (SLO: <10.000µs 📉 -83.8%) vs baseline: +1.9% Memory: ✅ 42.585MB (SLO: <43.500MB -2.1%) vs baseline: +4.9% ✅ encode_aspectTime: ✅ 18.306µs (SLO: <30.000µs 📉 -39.0%) vs baseline: 📈 +23.5% Memory: ✅ 42.546MB (SLO: <43.500MB -2.2%) vs baseline: +4.5% ✅ encode_noaspectTime: ✅ 1.500µs (SLO: <10.000µs 📉 -85.0%) vs baseline: -0.8% Memory: ✅ 42.566MB (SLO: <43.500MB -2.1%) vs baseline: +4.5% ✅ format_aspectTime: ✅ 171.503µs (SLO: <200.000µs 📉 -14.2%) vs baseline: +0.3% Memory: ✅ 42.526MB (SLO: <43.250MB 🟡 -1.7%) vs baseline: +4.4% ✅ format_map_aspectTime: ✅ 190.753µs (SLO: <200.000µs -4.6%) vs baseline: ~same Memory: ✅ 42.723MB (SLO: <43.500MB 🟡 -1.8%) vs baseline: +4.9% ✅ format_map_noaspectTime: ✅ 3.774µs (SLO: <10.000µs 📉 -62.3%) vs baseline: -1.0% Memory: ✅ 42.566MB (SLO: <43.250MB 🟡 -1.6%) vs baseline: +4.8% ✅ format_noaspectTime: ✅ 3.119µs (SLO: <10.000µs 📉 -68.8%) vs baseline: -0.3% Memory: ✅ 42.546MB (SLO: <43.250MB 🟡 -1.6%) vs baseline: +4.6% ✅ index_aspectTime: ✅ 15.365µs (SLO: <20.000µs 📉 -23.2%) vs baseline: ~same Memory: ✅ 42.585MB (SLO: <43.250MB 🟡 -1.5%) vs baseline: +5.0% ✅ index_noaspectTime: ✅ 0.470µs (SLO: <10.000µs 📉 -95.3%) vs baseline: +1.6% Memory: ✅ 42.526MB (SLO: <43.500MB -2.2%) vs baseline: +4.9% ✅ join_aspectTime: ✅ 17.076µs (SLO: <20.000µs 📉 -14.6%) vs baseline: ~same Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +5.0% ✅ join_noaspectTime: ✅ 1.552µs (SLO: <10.000µs 📉 -84.5%) vs baseline: -0.4% Memory: ✅ 42.625MB (SLO: <43.250MB 🟡 -1.4%) vs baseline: +4.9% ✅ ljust_aspectTime: ✅ 20.883µs (SLO: <30.000µs 📉 -30.4%) vs baseline: +0.4% Memory: ✅ 42.585MB (SLO: <43.250MB 🟡 -1.5%) vs baseline: +4.8% ✅ ljust_noaspectTime: ✅ 2.723µs (SLO: <10.000µs 📉 -72.8%) vs baseline: +1.6% Memory: ✅ 42.585MB (SLO: <43.250MB 🟡 -1.5%) vs baseline: +4.9% ✅ lower_aspectTime: ✅ 17.980µs (SLO: <30.000µs 📉 -40.1%) vs baseline: ~same Memory: ✅ 42.605MB (SLO: <43.500MB -2.1%) vs baseline: +4.9% ✅ lower_noaspectTime: ✅ 2.406µs (SLO: <10.000µs 📉 -75.9%) vs baseline: +0.1% Memory: ✅ 42.605MB (SLO: <43.250MB 🟡 -1.5%) vs baseline: +4.8% ✅ lstrip_aspectTime: ✅ 17.681µs (SLO: <30.000µs 📉 -41.1%) vs baseline: ~same Memory: ✅ 42.625MB (SLO: <43.250MB 🟡 -1.4%) vs baseline: +4.8% ✅ lstrip_noaspectTime: ✅ 1.870µs (SLO: <10.000µs 📉 -81.3%) vs baseline: -0.2% Memory: ✅ 42.566MB (SLO: <43.500MB -2.1%) vs baseline: +4.9% ✅ modulo_aspectTime: ✅ 166.634µs (SLO: <200.000µs 📉 -16.7%) vs baseline: +0.2% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +4.5% ✅ modulo_aspect_for_bytearray_bytearrayTime: ✅ 180.010µs (SLO: <200.000µs -10.0%) vs baseline: +3.0% Memory: ✅ 42.585MB (SLO: <43.500MB -2.1%) vs baseline: +4.5% ✅ modulo_aspect_for_bytesTime: ✅ 168.635µs (SLO: <200.000µs 📉 -15.7%) vs baseline: -0.1% Memory: ✅ 42.546MB (SLO: <43.500MB -2.2%) vs baseline: +4.3% ✅ modulo_aspect_for_bytes_bytearrayTime: ✅ 171.969µs (SLO: <200.000µs 📉 -14.0%) vs baseline: -0.2% Memory: ✅ 42.802MB (SLO: <43.500MB 🟡 -1.6%) vs baseline: +5.1% ✅ modulo_noaspectTime: ✅ 3.664µs (SLO: <10.000µs 📉 -63.4%) vs baseline: +0.4% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +5.0% ✅ replace_aspectTime: ✅ 212.691µs (SLO: <300.000µs 📉 -29.1%) vs baseline: +0.4% Memory: ✅ 42.703MB (SLO: <44.000MB -2.9%) vs baseline: +4.8% ✅ replace_noaspectTime: ✅ 2.897µs (SLO: <10.000µs 📉 -71.0%) vs baseline: -0.2% Memory: ✅ 42.566MB (SLO: <43.500MB -2.1%) vs baseline: +4.8% ✅ repr_aspectTime: ✅ 1.401µs (SLO: <10.000µs 📉 -86.0%) vs baseline: -1.8% Memory: ✅ 42.566MB (SLO: <43.500MB -2.1%) vs baseline: +4.8% ✅ repr_noaspectTime: ✅ 0.525µs (SLO: <10.000µs 📉 -94.8%) vs baseline: -0.5% Memory: ✅ 42.566MB (SLO: <43.500MB -2.1%) vs baseline: +4.9% ✅ rstrip_aspectTime: ✅ 18.991µs (SLO: <30.000µs 📉 -36.7%) vs baseline: ~same Memory: ✅ 42.566MB (SLO: <43.500MB -2.1%) vs baseline: +4.7% ✅ rstrip_noaspectTime: ✅ 2.035µs (SLO: <10.000µs 📉 -79.6%) vs baseline: +5.1% Memory: ✅ 42.644MB (SLO: <43.500MB 🟡 -2.0%) vs baseline: +4.9% ✅ slice_aspectTime: ✅ 15.927µs (SLO: <20.000µs 📉 -20.4%) vs baseline: +0.6% Memory: ✅ 42.585MB (SLO: <43.500MB -2.1%) vs baseline: +4.8% ✅ slice_noaspectTime: ✅ 0.595µs (SLO: <10.000µs 📉 -94.1%) vs baseline: -0.4% Memory: ✅ 42.566MB (SLO: <43.500MB -2.1%) vs baseline: +4.5% ✅ stringio_aspectTime: ✅ 54.018µs (SLO: <80.000µs 📉 -32.5%) vs baseline: -0.9% Memory: ✅ 42.566MB (SLO: <43.500MB -2.1%) vs baseline: +4.8% ✅ stringio_noaspectTime: ✅ 3.656µs (SLO: <10.000µs 📉 -63.4%) vs baseline: ~same Memory: ✅ 42.526MB (SLO: <43.500MB -2.2%) vs baseline: +4.9% ✅ strip_aspectTime: ✅ 17.640µs (SLO: <20.000µs 📉 -11.8%) vs baseline: ~same Memory: ✅ 42.526MB (SLO: <43.500MB -2.2%) vs baseline: +4.7% ✅ strip_noaspectTime: ✅ 1.856µs (SLO: <10.000µs 📉 -81.4%) vs baseline: -0.7% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +5.0% ✅ swapcase_aspectTime: ✅ 18.427µs (SLO: <30.000µs 📉 -38.6%) vs baseline: -0.2% Memory: ✅ 42.585MB (SLO: <43.500MB -2.1%) vs baseline: +5.0% ✅ swapcase_noaspectTime: ✅ 2.797µs (SLO: <10.000µs 📉 -72.0%) vs baseline: +0.5% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +5.0% ✅ title_aspectTime: ✅ 18.190µs (SLO: <30.000µs 📉 -39.4%) vs baseline: -0.2% Memory: ✅ 42.625MB (SLO: <43.000MB 🟡 -0.9%) vs baseline: +5.0% ✅ title_noaspectTime: ✅ 2.688µs (SLO: <10.000µs 📉 -73.1%) vs baseline: +0.6% Memory: ✅ 42.605MB (SLO: <43.500MB -2.1%) vs baseline: +4.8% ✅ translate_aspectTime: ✅ 24.366µs (SLO: <30.000µs 📉 -18.8%) vs baseline: 📈 +19.4% Memory: ✅ 42.526MB (SLO: <43.500MB -2.2%) vs baseline: +4.7% ✅ translate_noaspectTime: ✅ 4.343µs (SLO: <10.000µs 📉 -56.6%) vs baseline: -0.4% Memory: ✅ 42.605MB (SLO: <43.500MB -2.1%) vs baseline: +5.1% ✅ upper_aspectTime: ✅ 18.030µs (SLO: <30.000µs 📉 -39.9%) vs baseline: +0.3% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +5.1% ✅ upper_noaspectTime: ✅ 2.419µs (SLO: <10.000µs 📉 -75.8%) vs baseline: -1.4% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +4.9% 📈 iastaspectsospath - 24/24✅ ospathbasename_aspectTime: ✅ 5.236µs (SLO: <10.000µs 📉 -47.6%) vs baseline: 📈 +23.8% Memory: ✅ 41.622MB (SLO: <43.500MB -4.3%) vs baseline: +5.1% ✅ ospathbasename_noaspectTime: ✅ 4.345µs (SLO: <10.000µs 📉 -56.5%) vs baseline: +0.8% Memory: ✅ 41.602MB (SLO: <43.500MB -4.4%) vs baseline: +5.4% ✅ ospathjoin_aspectTime: ✅ 6.225µs (SLO: <10.000µs 📉 -37.8%) vs baseline: -0.2% Memory: ✅ 41.465MB (SLO: <43.500MB -4.7%) vs baseline: +5.1% ✅ ospathjoin_noaspectTime: ✅ 6.325µs (SLO: <10.000µs 📉 -36.8%) vs baseline: ~same Memory: ✅ 41.583MB (SLO: <43.500MB -4.4%) vs baseline: +4.9% ✅ ospathnormcase_aspectTime: ✅ 3.601µs (SLO: <10.000µs 📉 -64.0%) vs baseline: +0.2% Memory: ✅ 41.524MB (SLO: <43.500MB -4.5%) vs baseline: +5.1% ✅ ospathnormcase_noaspectTime: ✅ 3.608µs (SLO: <10.000µs 📉 -63.9%) vs baseline: -1.2% Memory: ✅ 41.406MB (SLO: <43.500MB -4.8%) vs baseline: +4.7% ✅ ospathsplit_aspectTime: ✅ 4.926µs (SLO: <10.000µs 📉 -50.7%) vs baseline: +0.3% Memory: ✅ 41.425MB (SLO: <43.500MB -4.8%) vs baseline: +4.8% ✅ ospathsplit_noaspectTime: ✅ 5.039µs (SLO: <10.000µs 📉 -49.6%) vs baseline: +0.7% Memory: ✅ 41.543MB (SLO: <43.500MB -4.5%) vs baseline: +4.8% ✅ ospathsplitdrive_aspectTime: ✅ 3.744µs (SLO: <10.000µs 📉 -62.6%) vs baseline: -0.4% Memory: ✅ 41.484MB (SLO: <43.500MB -4.6%) vs baseline: +4.8% ✅ ospathsplitdrive_noaspectTime: ✅ 0.758µs (SLO: <10.000µs 📉 -92.4%) vs baseline: +1.5% Memory: ✅ 41.563MB (SLO: <43.500MB -4.5%) vs baseline: +4.8% ✅ ospathsplitext_aspectTime: ✅ 4.648µs (SLO: <10.000µs 📉 -53.5%) vs baseline: +0.6% Memory: ✅ 41.642MB (SLO: <43.500MB -4.3%) vs baseline: +5.2% ✅ ospathsplitext_noaspectTime: ✅ 4.655µs (SLO: <10.000µs 📉 -53.5%) vs baseline: +0.5% Memory: ✅ 41.465MB (SLO: <43.500MB -4.7%) vs baseline: +4.9% 📈 telemetryaddmetric - 30/30✅ 1-count-metric-1-timesTime: ✅ 3.364µs (SLO: <20.000µs 📉 -83.2%) vs baseline: 📈 +12.4% Memory: ✅ 34.878MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +4.9% ✅ 1-count-metrics-100-timesTime: ✅ 199.037µs (SLO: <220.000µs -9.5%) vs baseline: -1.3% Memory: ✅ 34.859MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +4.7% ✅ 1-distribution-metric-1-timesTime: ✅ 3.288µs (SLO: <20.000µs 📉 -83.6%) vs baseline: -1.6% Memory: ✅ 34.878MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +4.9% ✅ 1-distribution-metrics-100-timesTime: ✅ 215.470µs (SLO: <230.000µs -6.3%) vs baseline: -0.7% Memory: ✅ 34.898MB (SLO: <35.500MB 🟡 -1.7%) vs baseline: +4.7% ✅ 1-gauge-metric-1-timesTime: ✅ 2.191µs (SLO: <20.000µs 📉 -89.0%) vs baseline: +0.1% Memory: ✅ 34.918MB (SLO: <35.500MB 🟡 -1.6%) vs baseline: +5.4% ✅ 1-gauge-metrics-100-timesTime: ✅ 137.551µs (SLO: <150.000µs -8.3%) vs baseline: +0.6% Memory: ✅ 34.859MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +4.9% ✅ 1-rate-metric-1-timesTime: ✅ 3.121µs (SLO: <20.000µs 📉 -84.4%) vs baseline: -0.6% Memory: ✅ 34.819MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +4.7% ✅ 1-rate-metrics-100-timesTime: ✅ 215.451µs (SLO: <250.000µs 📉 -13.8%) vs baseline: ~same Memory: ✅ 34.937MB (SLO: <35.500MB 🟡 -1.6%) vs baseline: +5.3% ✅ 100-count-metrics-100-timesTime: ✅ 20.100ms (SLO: <22.000ms -8.6%) vs baseline: +1.0% Memory: ✅ 34.918MB (SLO: <35.500MB 🟡 -1.6%) vs baseline: +4.8% ✅ 100-distribution-metrics-100-timesTime: ✅ 2.215ms (SLO: <2.550ms 📉 -13.1%) vs baseline: -0.5% Memory: ✅ 34.878MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +4.9% ✅ 100-gauge-metrics-100-timesTime: ✅ 1.413ms (SLO: <1.550ms -8.9%) vs baseline: +1.4% Memory: ✅ 34.918MB (SLO: <35.500MB 🟡 -1.6%) vs baseline: +4.9% ✅ 100-rate-metrics-100-timesTime: ✅ 2.183ms (SLO: <2.550ms 📉 -14.4%) vs baseline: +0.4% Memory: ✅ 34.859MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +4.9% ✅ flush-1-metricTime: ✅ 4.591µs (SLO: <20.000µs 📉 -77.0%) vs baseline: +0.1% Memory: ✅ 34.839MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +3.9% ✅ flush-100-metricsTime: ✅ 173.377µs (SLO: <250.000µs 📉 -30.6%) vs baseline: -0.3% Memory: ✅ 35.232MB (SLO: <35.500MB 🟡 -0.8%) vs baseline: +5.0% ✅ flush-1000-metricsTime: ✅ 2.187ms (SLO: <2.500ms 📉 -12.5%) vs baseline: ~same Memory: ✅ 35.940MB (SLO: <36.500MB 🟡 -1.5%) vs baseline: +4.7% 🟡 Near SLO Breach (16 suites)🟡 coreapiscenario - 10/10 (1 unstable)
|
|
In our system tests (https://github.com/DataDog/system-tests/blob/main/manifests/python.yml#L978-L980) will be able to replicate this fix with pre-working servers; is that what test type |
|
@leoromanovsky yes! This bug is reproducible/verifiable with at least flask-poc type (and probably other as well) |
|
ugh, why pr name lint is so stupid |
Description
FFE was not receiving remote configuration in forking environments (gunicorn, uWSGI) because it was being registered after the fork when user code instantiated DataDogProvider(). The parent process runs the remote config poller and distributes configurations to child workers via shared memory, but it didn't know about FFE_FLAGS product since registration happened in the child.
This fix integrates FFE into the product lifecycle pattern used by other products (AppSec, IAST, etc.):
Now FFE is registered during tracer initialization before any fork (when DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED=true), allowing the parent process to recognize and dispatch FFE_FLAGS configurations to child workers.
Testing
Tested with local repo tests, as well as with system tests.
Risks
Additional Notes