v4.0: Admin can deadlock with plugin init() when a queued command runs after PROXYSQL START (--no-start) #6500

Summary

With --no-start, an Admin statement that was already queued on Admin's global SQL mutex when PROXYSQL START ran can execute while the plugin lifecycle (phase 3) is in progress, even though the startup gate is closed at that point. If that statement needs the plugin manager (for example LOAD MYSQL SERVERS TO RUNTIME) while a plugin init() needs the global SQL mutex (for example mysql_router reading its metadata secret), the two threads deadlock. ProxySQL never starts its listeners and Admin stops answering.

The window is narrow: it needs --no-start, a plugin whose init() takes the global SQL mutex, and an Admin command that is waiting on the mutex at the exact moment PROXYSQL START runs.

Affected versions / tiers

• v4.0 (PROXYSQL40=1) only. On Stable and v3.1 the gate is never reset, so it stays open and nothing waits.
• Not present in v3.0.11. Introduced by d59f106 ("fix(startup): run the plugin lifecycle before Admin commands and Cluster sync"), which added ProxySQL_StartupGate.

Root cause

References are to d7ce280.

admin_session_handler() checks the gate before taking sql_query_global_mutex and never checks it again (lib/Admin_Handler.cpp:3204 and :3211):

proxysql_startup_gate_wait_for_admin([] { return glovars.shutdown != 0; });
...
proxysql_admin_global_sql_lock(pa->sql_query_global_mutex);

PROXYSQL START closes the gate from inside a statement that already holds that mutex (lib/Admin_Handler.cpp:786), and then releases the main thread into phase 3:

if (admin_nostart_) {
    proxysql_startup_gate_close_for_start();
    rc = __sync_bool_compare_and_swap(&GloVars.global.nostart, 1, 0);
}

wait_until() (lib/ProxySQL_StartupGate.cpp:15-23) only blocks callers that arrive after the state changed. A session that passed the gate while it was prestart_open and is queued on the mutex therefore runs its statement as soon as PROXYSQL START releases the mutex.

The two sides of the deadlock:

• Main thread: proxysql_init_configured_plugins() holds g_active_plugin_manager_mutex exclusively while plugin init() runs (lib/ProxySQL_PluginManager.cpp:2387). mysql_router init() builds its reconcile backend (plugins/mysql_router/src/plugin.cpp:85), which calls services_.get_secret(...) (plugins/mysql_router/src/reconciler_runtime.cpp:261). get_secret_service() takes ConfigurationLock (lib/ProxySQL_PluginManager.cpp:560), which is sql_query_global_mutex (lib/ProxySQL_Admin.cpp:425-427).
• Admin session: LOAD MYSQL SERVERS TO RUNTIME holds sql_query_global_mutex and reaches ScopedActiveManagerPin (a shared lock on the same plugin-manager mutex, lib/ProxySQL_PluginManager.cpp:118) through proxysql_install_active_server_runtime() or proxysql_active_server_hostgroup_claims(). The exclusive-owner bypass in the pin is thread-local to the lifecycle thread, so it does not help the Admin thread.

How to trigger / reproduce

1. Build v4.0 and configure a bootstrapped mysql_router plugin (so its init() reads the metadata secret).
2. Start ProxySQL with --no-start. The gate is prestart_open.
3. Admin session C runs any slow statement, so it holds the global SQL mutex.
4. While C is running, session B sends LOAD MYSQL SERVERS TO RUNTIME and session A sends PROXYSQL START. Both pass the gate and queue on the mutex.
5. A gets the mutex first, closes the gate and releases the main thread, which enters InitConfiguredPlugins and takes the plugin manager exclusively.
6. B gets the mutex, blocks on the plugin-manager pin; mysql_router init() blocks on the global SQL mutex held by B.

The ordering in steps 5/6 is a race and has not been reproduced on a running instance; it was traced through the code. The interleaving is reproduced deterministically with stand-in locks by the new unit test startup_gate_admin_lock_unit-t.

Impact

Phase 3 never completes, so the gate never opens and no listener starts. Every later Admin command blocks in the gate, so ProxySQL cannot be stopped through Admin; only an external kill recovers. It requires --no-start and is a startup-time race, so it is unlikely in practice, but tooling that configures a --no-start instance and then sends PROXYSQL START from a separate connection while other configuration statements are still in flight can hit it.

Proposed fix

Re-check the gate after the global SQL mutex is acquired. If it was closed while the session was queued, release the mutex, wait for the gate to reopen and try again. The check under the mutex is final: once Admin is up, the only transition to closed is PROXYSQL START, and it runs under the same mutex. Wrap this in a small helper in ProxySQL_StartupGate (proxysql_startup_gate_lock_for_admin(lock, unlock, stop)) so it can be unit tested, and use it in admin_session_handler(). Behaviour with the gate open is unchanged.

Relation to #6489

#6489 proposes replacing the startup gate with deferred accept(), and for --no-start proposes that from PROXYSQL START until the ready flag is set, Admin closes any session that issues a command. That design would also cover this case, provided the "issues a command" check is made after the global SQL mutex is held (or the command is otherwise refused before it can take it): a session already queued on the mutex when PROXYSQL START runs has, by definition, already issued its command. This fix is the small interim correction for the current gate and is removed together with the gate if #6489 lands; the same check-under-the-lock requirement should carry over to the replacement.

Is this really a bug?

Yes. The gate's documented contract (include/ProxySQL_StartupGate.h) is that while the plugin lifecycle runs, "Admin commands and Cluster threads wait", and the sequence above violates it and ends in a deadlock. The only argument against is how narrow the trigger is (--no-start, a plugin init() that takes the global mutex, and a command queued at the exact moment of PROXYSQL START), which affects the severity, not whether it is a defect.