Fix automation actions failing silently in read-only containers

[zzstoatzz]

Closes #19317

Summary

Fixes automation actions that fail silently when background services run in read-only containers (common in rootless/secure deployments).

Root Cause

Background services (actions service) use OrchestrationClient for in-process API calls. OrchestrationClient was creating a full FastAPI app with UI support, which attempts to create the UI static directory. In read-only containers, this raises PermissionError, causing actions to fail and messages to go to DLQ with no error logs.

Changes

1. Skip UI creation for OrchestrationClient (Primary Fix)

File: src/prefect/server/api/clients.py

# Before
api_app = create_app()

# After  
api_app = create_app(ephemeral=True)

• Background services use in-memory ASGI transport and don't need the UI
• ephemeral=True skips UI directory creation
• Prevents PermissionError in read-only containers

2. Add error logging for action failures

File: src/prefect/server/events/actions.py

• Catch and log unexpected exceptions in action consumer
• Provides visibility when actions fail and messages go to DLQ
• Exceptions are still propagated to messaging system for proper retry handling

Testing

Unit Test Added

test_orchestration_client_works_with_readonly_ui_directory in tests/server/api/test_clients.py

• Simulates read-only filesystem scenario
• Fails before fix with PermissionError
• Passes after fix - client created successfully

Manual Testing

• Created HA Docker Compose setup with read-only background services
• Reproduced original issue (actions in DLQ, no logs)
• Verified fix resolves the issue
• All 19 client tests pass

Impact

✅ Fixes:

• Automations now work in secure/rootless container deployments
• Action failures are no longer silent - full error logging with context

✅ No breaking changes:

• Existing deployments unaffected
• ephemeral=True is already used for ephemeral servers
• Only changes behavior for OrchestrationClient (internal use only)

Related

• User reported workaround: PREFECT_UI_STATIC_DIRECTORY=/writable/dir
• This fix makes the workaround unnecessary

🤖 Generated with Claude Code

[codspeed-hq]

CodSpeed Performance Report

Merging #19319 will not alter performance

_{Comparing fix-19317-automation-readonly-failure (181ab54) with main (7d86c93)}

Summary

✅ 2 untouched