feat: add db migrations and live-reload for per-request content storage and raw override flags (#3117)

## Summary

Persisting raw provider bytes in logs is gated on content logging being active. This PR makes that dependency explicit in the database schema, config store, HTTP handler, UI, and documentation — and removes a spurious restart requirement when toggling `disable_content_logging`.

## Changes

- Added `allow_per_request_content_storage_override` and `allow_per_request_raw_override` columns to `TableClientConfig` with corresponding database migrations, and wired both fields through `UpdateClientConfig` / `GetClientConfig` in the RDB config store.
- Propagated both new fields through the HTTP config update handler so they are persisted when the settings are changed via the API or UI.
- Removed the restart trigger for `disable_content_logging` changes in both the Go handler and the React UI, since the logging plugin holds a live pointer to `ClientConfig` and picks up the new value on the next request without a restart.
- Updated the UI description for **Disable Content Logging** to make clear that raw provider bytes (`store_raw_request_response`) are also dropped when content logging is off, while send-back behavior (`send_back_raw_*`) is unaffected.
- Updated the UI description for **Allow Per-Request Content Storage Override** to clarify that raw-byte storage requires content logging to be on — either globally or via `x-bf-disable-content-logging: false` on the same request.
- Added a documentation warning in `request-options.mdx` stating that raw bytes are only persisted when content logging is active, and updated the `x-bf-disable-content-logging` header description to reflect that it also gates raw-byte storage and can be used to opt a single request into full content+raw capture while content logging is globally disabled.

## Type of change

- [ ] Bug fix
- [x] Feature
- [ ] Refactor
- [x] Documentation
- [ ] Chore/CI

## Affected areas

- [x] Core (Go)
- [x] Transports (HTTP)
- [ ] Providers/Integrations
- [ ] Plugins
- [x] UI (React)
- [x] Docs

## How to test

```sh
# Core/Transports
go test ./framework/configstore/... ./transports/bifrost-http/...

# Verify migrations run cleanly against a fresh or existing DB
# Start the gateway and confirm allow_per_request_content_storage_override
# and allow_per_request_raw_override columns exist in config_client.

# UI
cd ui
pnpm i
pnpm build
```

1. Enable logging, then toggle **Disable Content Logging** — confirm no restart banner appears.
2. Enable **Allow Per-Request Content Storage Override** and send a request with `x-bf-store-raw-request-response: true` while `x-bf-disable-content-logging: true` — confirm raw bytes are absent from the log record.
3. Repeat with `x-bf-disable-content-logging: false` — confirm raw bytes are present.

## Breaking changes

- [ ] Yes
- [x] No

## Security considerations

`allow_per_request_content_storage_override` and `allow_per_request_raw_override` are opt-in and default to `false`, so existing deployments retain their current behavior. Operators must explicitly enable these flags before per-request headers can influence content or raw-byte storage, preserving the existing access control boundary around sensitive log data.

## Checklist

- [ ] I read `docs/contributing/README.md` and followed the guidelines
- [ ] I added/updated tests where appropriate
- [x] I updated documentation where needed
- [ ] I verified builds succeed (Go and UI)
- [ ] I verified the CI pipeline passes locally if applicable

Author: Pratham-Mishra04

docs/providers/request-options.mdx

@@ -418,6 +418,10 @@ This is orthogonal to the send-back flags: enabling this does not affect whether
 Per-request overrides are **disabled by default**. You must first enable `allow_per_request_content_storage_override` in your logging configuration (or in the UI under **Logs Settings**) before this header or context key has any effect. Note that this is gated by the **content storage** override, not the raw override — `allow_per_request_raw_override` only gates `x-bf-send-back-raw-request` and `x-bf-send-back-raw-response` (sending raw bytes back to the caller).
 </Warning>
 
+<Warning>
+**Content logging must also be enabled for raw bytes to be persisted.** The logging plugin only writes raw bytes when content logging is on — i.e. either global `disable_content_logging` is `false`, or the request sets `x-bf-disable-content-logging: false` (with `allow_per_request_content_storage_override` enabled). If content logging is off, raw bytes are dropped from the log row even when `x-bf-store-raw-request-response: true`.
+</Warning>
+
 <Tabs>
 <Tab title="Gateway (cURL)">
 ```bash
@@ -460,9 +464,9 @@ Input: messages,
 **Type:** `bool` (header values: `"true"` or `"false"`)
 **Required:** No
 
-Override the logging plugin's global `disable_content_logging` config for a single request. When set to `true`, messages, parameters, tool arguments, and tool results are omitted from the log record for that request. When set to `false`, content is recorded even if the global toggle is off.
+Override the logging plugin's global `disable_content_logging` config for a single request. When set to `true`, messages, parameters, tool arguments, tool results, **and raw provider bytes** are omitted from the log record for that request. When set to `false`, content (and raw bytes, if `x-bf-store-raw-request-response` is also enabled) is recorded even if the global toggle is off.
 
-This is useful when you need to suppress sensitive data (e.g. PII, credentials) for specific requests while keeping content logging enabled globally.
+This is useful when you need to suppress sensitive data (e.g. PII, credentials) for specific requests while keeping content logging enabled globally — or, conversely, to opt a single request into full content+raw capture while content logging is globally disabled.
 
 <Warning>
 Per-request overrides are **disabled by default**. You must first enable `allow_per_request_content_storage_override` in your logging configuration (or in the UI under **Logs Settings**) before this header or context key has any effect. When the toggle is off, the global `disable_content_logging` setting is authoritative and this value is ignored.

framework/configstore/migrations.go

@@ -623,6 +623,12 @@ func triggerMigrations(ctx context.Context, db *gorm.DB) error {
 	if err := migrationMakeOAuthTokenExpiryNullable(ctx, db); err != nil {
 		return err
 	}
+	if err := migrationAddAllowPerRequestContentStorageOverrideColumn(ctx, db); err != nil {
+		return err
+	}
+	if err := migrationAddAllowPerRequestRawOverrideColumn(ctx, db); err != nil {
+		return err
+	}
 	return nil
 }
 
@@ -6564,6 +6570,76 @@ func migrationMakeOAuthTokenExpiryNullable(ctx context.Context, db *gorm.DB) err
 	return nil
 }
 
+// migrationAddAllowPerRequestContentStorageOverrideColumn adds the allow_per_request_content_storage_override column to config_client.
+func migrationAddAllowPerRequestContentStorageOverrideColumn(ctx context.Context, db *gorm.DB) error {
+	m := migrator.New(db, migrator.DefaultOptions, []*migrator.Migration{{
+		ID: "add_allow_per_request_content_storage_override_column",
+		Migrate: func(tx *gorm.DB) error {
+			tx = tx.WithContext(ctx)
+			migrator := tx.Migrator()
+
+			if !migrator.HasColumn(&tables.TableClientConfig{}, "allow_per_request_content_storage_override") {
+				if err := migrator.AddColumn(&tables.TableClientConfig{}, "AllowPerRequestContentStorageOverride"); err != nil {
+					return fmt.Errorf("failed to add allow_per_request_content_storage_override column: %w", err)
+				}
+			}
+
+			return nil
+		},
+		Rollback: func(tx *gorm.DB) error {
+			tx = tx.WithContext(ctx)
+			migrator := tx.Migrator()
+
+			if migrator.HasColumn(&tables.TableClientConfig{}, "allow_per_request_content_storage_override") {
+				if err := migrator.DropColumn(&tables.TableClientConfig{}, "allow_per_request_content_storage_override"); err != nil {
+					return fmt.Errorf("failed to drop allow_per_request_content_storage_override column: %w", err)
+				}
+			}
+
+			return nil
+		},
+	}})
+	if err := m.Migrate(); err != nil {
+		return fmt.Errorf("error running allow_per_request_content_storage_override migration: %s", err.Error())
+	}
+	return nil
+}
+
+// migrationAddAllowPerRequestRawOverrideColumn adds the allow_per_request_raw_override column to config_client.
+func migrationAddAllowPerRequestRawOverrideColumn(ctx context.Context, db *gorm.DB) error {
+	m := migrator.New(db, migrator.DefaultOptions, []*migrator.Migration{{
+		ID: "add_allow_per_request_raw_override_column",
+		Migrate: func(tx *gorm.DB) error {
+			tx = tx.WithContext(ctx)
+			migrator := tx.Migrator()
+
+			if !migrator.HasColumn(&tables.TableClientConfig{}, "allow_per_request_raw_override") {
+				if err := migrator.AddColumn(&tables.TableClientConfig{}, "AllowPerRequestRawOverride"); err != nil {
+					return fmt.Errorf("failed to add allow_per_request_raw_override column: %w", err)
+				}
+			}
+
+			return nil
+		},
+		Rollback: func(tx *gorm.DB) error {
+			tx = tx.WithContext(ctx)
+			migrator := tx.Migrator()
+
+			if migrator.HasColumn(&tables.TableClientConfig{}, "allow_per_request_raw_override") {
+				if err := migrator.DropColumn(&tables.TableClientConfig{}, "allow_per_request_raw_override"); err != nil {
+					return fmt.Errorf("failed to drop allow_per_request_raw_override column: %w", err)
+				}
+			}
+
+			return nil
+		},
+	}})
+	if err := m.Migrate(); err != nil {
+		return fmt.Errorf("error running allow_per_request_raw_override migration: %s", err.Error())
+	}
+	return nil
+}
+
 // migrationAddMCPClientDiscoveredToolsColumns adds discovered_tools_json and tool_name_mapping_json columns to the mcp_client table
 func migrationAddMCPClientDiscoveredToolsColumns(ctx context.Context, db *gorm.DB) error {
 	m := migrator.New(db, migrator.DefaultOptions, []*migrator.Migration{{

framework/configstore/rdb.go

@@ -153,38 +153,40 @@ func mcpExternalBaseURLToString(e *schemas.EnvVar) string {
 // UpdateClientConfig updates the client configuration in the database.
 func (s *RDBConfigStore) UpdateClientConfig(ctx context.Context, config *ClientConfig) error {
 	dbConfig := tables.TableClientConfig{
-		DropExcessRequests:              config.DropExcessRequests,
-		InitialPoolSize:                 config.InitialPoolSize,
-		EnableLogging:                   config.EnableLogging,
-		DisableContentLogging:           config.DisableContentLogging,
-		DisableDBPingsInHealth:          config.DisableDBPingsInHealth,
-		LogRetentionDays:                config.LogRetentionDays,
-		EnforceAuthOnInference:          config.EnforceAuthOnInference,
-		EnforceGovernanceHeader:         config.EnforceGovernanceHeader,
-		EnforceSCIMAuth:                 config.EnforceSCIMAuth,
-		AllowDirectKeys:                 config.AllowDirectKeys,
-		PrometheusLabels:                config.PrometheusLabels,
-		AllowedOrigins:                  config.AllowedOrigins,
-		AllowedHeaders:                  config.AllowedHeaders,
-		MaxRequestBodySizeMB:            config.MaxRequestBodySizeMB,
-		CompatConvertTextToChat:         config.Compat.ConvertTextToChat,
-		CompatConvertChatToResponses:    config.Compat.ConvertChatToResponses,
-		CompatShouldDropParams:          config.Compat.ShouldDropParams,
-		CompatShouldConvertParams:       config.Compat.ShouldConvertParams,
-		MCPAgentDepth:                   config.MCPAgentDepth,
-		MCPToolExecutionTimeout:         config.MCPToolExecutionTimeout,
-		MCPCodeModeBindingLevel:         config.MCPCodeModeBindingLevel,
-		MCPToolSyncInterval:             config.MCPToolSyncInterval,
-		MCPDisableAutoToolInject:        config.MCPDisableAutoToolInject,
-		AsyncJobResultTTL:               config.AsyncJobResultTTL,
-		RequiredHeaders:                 config.RequiredHeaders,
-		LoggingHeaders:                  config.LoggingHeaders,
-		WhitelistedRoutes:               config.WhitelistedRoutes,
-		HideDeletedVirtualKeysInFilters: config.HideDeletedVirtualKeysInFilters,
-		RoutingChainMaxDepth:            config.RoutingChainMaxDepth,
-		MCPExternalBaseURL:                 mcpExternalBaseURLToString(config.MCPExternalBaseURL),
-		HeaderFilterConfig:              config.HeaderFilterConfig,
-		ConfigHash:                      config.ConfigHash,
+		DropExcessRequests:                    config.DropExcessRequests,
+		InitialPoolSize:                       config.InitialPoolSize,
+		EnableLogging:                         config.EnableLogging,
+		DisableContentLogging:                 config.DisableContentLogging,
+		DisableDBPingsInHealth:                config.DisableDBPingsInHealth,
+		LogRetentionDays:                      config.LogRetentionDays,
+		EnforceAuthOnInference:                config.EnforceAuthOnInference,
+		EnforceGovernanceHeader:               config.EnforceGovernanceHeader,
+		EnforceSCIMAuth:                       config.EnforceSCIMAuth,
+		AllowDirectKeys:                       config.AllowDirectKeys,
+		PrometheusLabels:                      config.PrometheusLabels,
+		AllowedOrigins:                        config.AllowedOrigins,
+		AllowedHeaders:                        config.AllowedHeaders,
+		MaxRequestBodySizeMB:                  config.MaxRequestBodySizeMB,
+		CompatConvertTextToChat:               config.Compat.ConvertTextToChat,
+		CompatConvertChatToResponses:          config.Compat.ConvertChatToResponses,
+		CompatShouldDropParams:                config.Compat.ShouldDropParams,
+		CompatShouldConvertParams:             config.Compat.ShouldConvertParams,
+		MCPAgentDepth:                         config.MCPAgentDepth,
+		MCPToolExecutionTimeout:               config.MCPToolExecutionTimeout,
+		MCPCodeModeBindingLevel:               config.MCPCodeModeBindingLevel,
+		MCPToolSyncInterval:                   config.MCPToolSyncInterval,
+		MCPDisableAutoToolInject:              config.MCPDisableAutoToolInject,
+		AsyncJobResultTTL:                     config.AsyncJobResultTTL,
+		RequiredHeaders:                       config.RequiredHeaders,
+		LoggingHeaders:                        config.LoggingHeaders,
+		WhitelistedRoutes:                     config.WhitelistedRoutes,
+		HideDeletedVirtualKeysInFilters:       config.HideDeletedVirtualKeysInFilters,
+		RoutingChainMaxDepth:                  config.RoutingChainMaxDepth,
+		MCPExternalBaseURL:                    mcpExternalBaseURLToString(config.MCPExternalBaseURL),
+		HeaderFilterConfig:                    config.HeaderFilterConfig,
+		AllowPerRequestContentStorageOverride: config.AllowPerRequestContentStorageOverride,
+		AllowPerRequestRawOverride:            config.AllowPerRequestRawOverride,
+		ConfigHash:                            config.ConfigHash,
 	}
 	// Delete existing client config and create new one in a transaction
 	return s.DB().WithContext(ctx).Transaction(func(tx *gorm.DB) error {
@@ -382,20 +384,22 @@ func (s *RDBConfigStore) GetClientConfig(ctx context.Context) (*ClientConfig, er
 			ShouldDropParams:       dbConfig.CompatShouldDropParams,
 			ShouldConvertParams:    dbConfig.CompatShouldConvertParams,
 		},
-		MCPAgentDepth:                   dbConfig.MCPAgentDepth,
-		MCPToolExecutionTimeout:         dbConfig.MCPToolExecutionTimeout,
-		MCPCodeModeBindingLevel:         dbConfig.MCPCodeModeBindingLevel,
-		MCPToolSyncInterval:             dbConfig.MCPToolSyncInterval,
-		MCPDisableAutoToolInject:        dbConfig.MCPDisableAutoToolInject,
-		AsyncJobResultTTL:               dbConfig.AsyncJobResultTTL,
-		RequiredHeaders:                 dbConfig.RequiredHeaders,
-		LoggingHeaders:                  dbConfig.LoggingHeaders,
-		WhitelistedRoutes:               dbConfig.WhitelistedRoutes,
-		HideDeletedVirtualKeysInFilters: dbConfig.HideDeletedVirtualKeysInFilters,
-		RoutingChainMaxDepth:            dbConfig.RoutingChainMaxDepth,
-		MCPExternalBaseURL:                 schemas.NewEnvVar(dbConfig.MCPExternalBaseURL),
-		HeaderFilterConfig:              dbConfig.HeaderFilterConfig,
-		ConfigHash:                      dbConfig.ConfigHash,
+		MCPAgentDepth:                         dbConfig.MCPAgentDepth,
+		MCPToolExecutionTimeout:               dbConfig.MCPToolExecutionTimeout,
+		MCPCodeModeBindingLevel:               dbConfig.MCPCodeModeBindingLevel,
+		MCPToolSyncInterval:                   dbConfig.MCPToolSyncInterval,
+		MCPDisableAutoToolInject:              dbConfig.MCPDisableAutoToolInject,
+		AsyncJobResultTTL:                     dbConfig.AsyncJobResultTTL,
+		RequiredHeaders:                       dbConfig.RequiredHeaders,
+		LoggingHeaders:                        dbConfig.LoggingHeaders,
+		WhitelistedRoutes:                     dbConfig.WhitelistedRoutes,
+		HideDeletedVirtualKeysInFilters:       dbConfig.HideDeletedVirtualKeysInFilters,
+		RoutingChainMaxDepth:                  dbConfig.RoutingChainMaxDepth,
+		MCPExternalBaseURL:                    schemas.NewEnvVar(dbConfig.MCPExternalBaseURL),
+		HeaderFilterConfig:                    dbConfig.HeaderFilterConfig,
+		AllowPerRequestContentStorageOverride: dbConfig.AllowPerRequestContentStorageOverride,
+		AllowPerRequestRawOverride:            dbConfig.AllowPerRequestRawOverride,
+		ConfigHash:                            dbConfig.ConfigHash,
 	}, nil
 }
 
@@ -1249,7 +1253,7 @@ func (s *RDBConfigStore) GetMCPConfig(ctx context.Context) (*schemas.MCPConfig,
 				ClientConfigs: clientConfigs,
 				ToolManagerConfig: &schemas.MCPToolManagerConfig{
 					ToolExecutionTimeout: schemas.Duration(30 * time.Second), // default from TableClientConfig
-					MaxAgentDepth:        10,               // default from TableClientConfig
+					MaxAgentDepth:        10,                                 // default from TableClientConfig
 				},
 			}, nil
 		}