Orchestration client API

[KavithaSiva]

Current proposed state:

client.chatCompletion({
    deploymentConfiguration: { deploymentId: 'id' },
      llmConfig: {
        model_name: 'gpt-35-turbo-16k',
        model_params: { max_tokens: 50, temperature: 0.1 }
      },
      prompt: {
        template: [{ role: 'user', content: 'Am I {{?usr}}?' }],
        template_params: { usr: 'Alice' },
        messages_history: [
          {
            role: 'system',
            content:
              'You are a helpful assistant who remembers all details the user shares with you.'
          },
          {
            role: 'user',
            content: 'Hi! Im Bob'
          },
          {
            role: 'assistant',
            content:
              "Hi Bob, nice to meet you! I'm an AI assistant. I'll remember that your name is Bob as we continue our conversation."
          }
        ]
      },
      filterConfig: { 
        input: { azureContentSafety: { Hate: 0, SelfHarm: 2 }},
        output: { 
          azureContentSafety: { Sexual: 4, Violence: 6 },
          someOtherFilterServiceProvider: { Hate: 0, SelfHarm: 2 }
        },
      }
  })
);

(Vanilla) Orchestration client :

client.chatCompletion({
    deploymentConfiguration: { deploymentId: 'id' },
    orchestration_config: {
      module_configurations: {
        templating_module_config: {
          template: [{ role: 'user', content: 'Am I {{?usr}}?' }]
        },
        llm_module_config: {
        model_name: 'gpt-35-turbo-16k',
        model_params: { max_tokens: 50, temperature: 0.1 }
      },
       filtering_module_config: {
        "input": {
          "filters": [
            {
              "type": "azure_content_safety",
              "config": {
                "Hate": 4,
                "SelfHarm": 4,
                "Sexual": 4,
                "Violence": 4
              }
            }
          ]
        },
        "output": {
          "filters": [
            {
              "type": "azure_content_safety",
              "config": {
                "Hate": 0,
                "SelfHarm": 2,
                "Sexual": 2,
                "Violence": 2
              }
            }
          ]
        }
      }
      },
      input_params: { usr: 'Alice' },
      messages_history: [
          {
            role: 'system',
            content:
              'You are a helpful assistant who remembers all details the user shares with you.'
          },
          {
            role: 'user',
            content: 'Hi! Im Bob'
          },
          {
            role: 'assistant',
            content:
              "Hi Bob, nice to meet you! I'm an AI assistant. I'll remember that your name is Bob as we continue our conversation."
          }
        ]
    })
);

[KavithaSiva]

Proposal 1:

Use functions along with generated orchestration client

client.chatCompletion({
    deploymentConfiguration: { deploymentId: 'id' },
    orchestration_config: {
      module_configurations: {
        templating_module_config: {
          template: [{ role: 'user', content: 'Am I {{?usr}}?' }]
        },
        llm_module_config: {
        model_name: 'gpt-35-turbo-16k',
        model_params: { max_tokens: 50, temperature: 0.1 }
      },
       createFilterConfig(
        createInputFilter({ type: 'azureContentSafety', config: { Hate: 0, SelfHarm: 2}}),
        createOutputFilter({ type: 'azureContentSafety', config: { Sexual: 4, Violence: 6 }})
        )
      },
      input_params: { usr: 'Alice' },
      messages_history: [
          {
            role: 'system',
            content:
              'You are a helpful assistant who remembers all details the user shares with you.'
          },
          {
            role: 'user',
            content: 'Hi! Im Bob'
          },
          {
            role: 'assistant',
            content:
              "Hi Bob, nice to meet you! I'm an AI assistant. I'll remember that your name is Bob as we continue our conversation."
          }
        ]
    })
);

Pros:

• Generated client can mostly be directly used, without many additional wrappers.

Cons:

• Verbose
• Discoverability of convenience functions?

Doubts:

• Should we consider a convenience function for every module?
• What about params that don't fit into any modules like for e.g input_params and messages_history?

[KavithaSiva]

Follow-up ideas for the API:

1. Builder API - Marika
2. Shorten the input JSON, and use convenience functions + varargs
3. Input much closer to the Orchestration API, with camel casing

[MatKuhr]

I'd argue (3) is not worth the effort, we barely get any API quality improvement (camel case) but have to invest a lot of work (many properties are snake case currently, so we have to change/wrap many files). If we invest such effort, it would only be worthwhile if we also significantly change/simplify the usage. Or I am misunderstanding the idea somehow 😄

[KavithaSiva]

No, you are right, I agree with you too.

[KavithaSiva]

The following structure was finalised for the Orchestration client:

client.chatCompletion({
  deploymentConfiguration: { deploymentId: 'id' },
  llmConfig: {
    model_name: 'gpt-35-turbo-16k',
    model_params: { max_tokens: 50, temperature: 0.1 }
  },
  prompt: {
    template: [{ role: 'user', content: 'What is the capital of Germany?' }]
  },
  filterConfig: {
    input: azureContentFilter({ Hate: 4, SelfHarm: 2 }),
    output: azureContentFilter({ Sexual: 0, Violence: 6 })
  }
});