feat: Support orchestration prompt module fallback (with stream)

[davidkna-sap]

Context

Closes https://github.com/SAP/ai-sdk-js-backlog/issues/461

What this PR does and why it is needed

Streaming-related part of #1454

[hyperspace-insights]

Summary

The pull request adds streaming support for module fallback configurations in the orchestration service. The implementation is generally well-structured with comprehensive test coverage. I identified one typo that has been addressed. The code properly handles streaming with fallback configs, validates stream options arrays, and merges intermediate failures from multiple config attempts.

PR Bot Information

Version: 1.17.50 | 📖 Documentation | 🚨 Create Incident | 💬 Feedback

• Event Trigger: issue_comment.created
• Correlation ID: 92e3e530-01c5-11f1-8168-561f032297a5
• LLM: anthropic--claude-4.5-sonnet

[hyperspace-insights]

Summary

The following content is AI-generated and provides a summary of the pull request:

⚠️ Warnings:

• Could not get access to repository SAP/ai-sdk-js-backlog to retrieve Issue 461. Status: 404 - UnknownObjectException

---

feat: Support orchestration prompt module fallback with streaming

New Features

✨ Streaming Support for Module Fallback: Extended orchestration module fallback to support streaming requests, allowing automatic failover between model configurations during stream operations.

Changes

This PR completes the streaming implementation for orchestration prompt module fallback, building on the non-streaming support added in #1454.

Core Implementation:

• orchestration-client.ts: Removed the restriction that prevented streaming with multiple module configurations
• util/module-config.ts:
  • Added support for per-config stream option overrides via StreamOptions.overrides
  • Implemented buildModules() to apply stream options to module configuration arrays
  • Added validation and warning system for invalid override indices
  • Enhanced addStreamOptions() to handle both single configs and config arrays
• orchestration-stream.ts: Added logging for fallback usage during streaming
• orchestration-stream-response.ts: Added getIntermediateFailures() method to access errors from failed module attempts

Type Definitions:

• orchestration-types.ts: Added ModuleStreamOptions type and overrides property to StreamOptions for per-config stream customization

Testing:

• orchestration-client.test.ts: Added comprehensive tests for streaming with fallback configs and per-config stream options
• util/module-config.test.ts: Added extensive test coverage for stream option handling, override validation, and warning scenarios
• util/stream.test.ts: Added tests for merging intermediate failures from fallback attempts
• tests/e2e-tests/: Added end-to-end test for streaming with module fallback
• tests/type-tests/: Added type tests for StreamOptions, ModuleStreamOptions, and override configurations

Sample Code:

• sample-code/src/orchestration.ts: Added example demonstrating streaming with module fallback configurations

GitHub Issues

• Closes #461

Related Pull Requests

• Builds on #1454

---

• 🔄 Regenerate and Update Summary
• ✏️ Insert as PR Description (deletes this comment)
• 🗑️ Delete comment

PR Bot Information

Version: 1.17.61 | 📖 Documentation | 🚨 Create Incident | 💬 Feedback

• Correlation ID: 0f9f7660-0c03-11f1-96d7-4e512e15ce13
• GithubContextProvider: feat: Support orchestration prompt module fallback #1454
• Output Template: Default Template
• LLM: anthropic--claude-4.5-sonnet
• Summary Prompt: Default Prompt
• Event Trigger: issue_comment.edited

[KavithaSiva]

Initial review done, with few refactoring requests.

[KavithaSiva]

[req] This function is currently difficult to read with multiple cases tackled together for single/multiple module config and stream options.

Could you please refactor the function to have cleaner separations for each of these different cases?

[hyperspace-insights]

The pull request adds streaming support for orchestration module fallback configurations. I found a typo and a logic issue where output filtering warnings are not issued when using array stream options. The refactoring suggestion from earlier comments has been partially addressed, though the function remains complex.

PR Bot Information

Version: 1.17.61 | 📖 Documentation | 🚨 Create Incident | 💬 Feedback

• LLM: anthropic--claude-4.5-sonnet
• Event Trigger: issue_comment.created
• Correlation ID: 1b2b8f70-0b43-11f1-8660-bf77760f4759

[KavithaSiva]

[suggestion] I was brainstorming. Instead of providing an array of stream options what about having a type like this:

  interface StreamOptionsWithOverrides {                                                                                                                     
    global?: GlobalStreamOptions;                         // Top level only                                                                                 
    default?: Omit<StreamOptions, 'global'>;             // No global here                                                                                   
    overrides?: Record<number, Omit<StreamOptions, 'global'>>;  // No global here                                                                            
  }                                                                                                                                                          

Which makes it simpler to provide global config and default config that has to fit all orchestration configs?

So, the usage looks cleaner:

// Array config - structured                                                                                                                               
  const arrayClient = new OrchestrationClient([config1, config2, config3]);                                                                                  
  arrayClient.stream(req, sig, {                                                                                                                             
    global: { chunk_size: 100 },                                                                                                                             
    default: { promptTemplating: { include_usage: false } },                                                                                                 
    overrides: {                                                                                                                                             
      0: { promptTemplating: { include_usage: true } }                                                                                                       
    }                                                                                                                                                        
  });                                                                                                                                                        
                                                                                                                                                             
  // Array config - minimal (just global)                                                                                                                    
  arrayClient.stream(req, sig, {                                                                                                                             
    global: { chunk_size: 100 }                                                                                                                              
  });                                                                                                                                                        
                                                                                                                                                             
  // Array config - just overrides                                                                                                                           
  arrayClient.stream(req, sig, {                                                                                                                             
    overrides: {                                                                                                                                             
      1: { outputFiltering: { overlap: 50 } }                                                                                                                
    }                                                                                                                                                        
  });           

Honestly, it would have been much easier if the stream options in module configuration had also been directly initialised in the constructor instead, a consideration for v3 maybe.

Another option would be to only allow global and default options for now and only allow overrides if there is enough demand.

What do you think of these suggestions?

[davidkna-sap]

I do think this is cleaner, though as far as ease-of-use goes, simple cases can still use a single non-list StreamOptions as usual anyway. I mostly only implemented this more complicated way of doing this because the documentation/demos stressed the importance of each module being independent and very customised for each the module configuration.

That said, I will try implementing your suggestion to see how it looks.

[marikaner]

@davidkna-sap I would like to throw in a slight change to @KavithaSiva's idea: You could also change the API to be something like:

type StreamOptionsWithOverrides  = StreamOptions & {                                                                                                                                                                                                      
  overrides?: Record<number, Omit<StreamOptions, 'global'>>;  // No global here                                                                            
}

That would keep the stream options as is for the simple case and allow overrides for fallbacks.

[KavithaSiva]

Wait a bit before implementing my suggestion. I discussed the API with @marikaner today and she also has some suggestions for the API. Let's discuss before we implement more.

[davidkna-sap]

@KavithaSiva Another thing I was considering: How you feel about removing the merging with the shared stream configuration? Currently it's not possible for e.g. a {} override to remove settings that are included in the main/shared stream configuration, so it would have to be left empty in some cases.

[KavithaSiva]

@KavithaSiva Another thing I was considering: How you feel about removing the merging with the shared stream configuration? Currently it's not possible for e.g. a {} override to remove settings that are included in the main/shared stream configuration, so it would have to be left empty in some cases.

You mean overrides completely overrides all shared values in promptTemplating and outputFiltering?
I think that's a good idea.