Conditional Routing

This feature is available on all Portkey plans.

Using Portkey Gateway, you can route your requests to different provider targets based on custom conditions you define. These can be conditions like:

• If this user is on the paid plan, route their request to a custom fine-tuned model
• If the model parameter is set to fastest, route to gpt-4o-mini, if smartest, route to openai o1
• If this user is an EU resident, call an EU hosted model
• If the temperature parameter is above 0.7, route to a more creative model
• If the request is coming from testing environment with a llm-pass-through flag, route it to the cheapest model
• ..and more!

Using this strategy, you can set up sophisticated routing logic based on:

1. Metadata - Custom key-value pairs you pass with your requests
2. Request Parameters - Any parameter that you send in your original LLM request (like model, temperature, max_tokens)

All routing happens very fast on the gateway, on edge.

Enabling Conditional Routing

Conditional routing is one of the strategies in Portkey’s Gateway Configs. (others being fallback and loadbalance). To use it in your app,

1. You need to create a conditional config in Portkey UI.
2. Save the Config and get an associated Config ID.
3. And just pass the Config ID along with your requests using the config param.

1. Creating the conditional Config

Here’s how a sample conditional config looks (along with its simpler, tree view).

Sample Config

{
  "strategy": {
    "mode": "conditional",
    "conditions": [
      ...conditions
    ],
    "default": "target_1"
  },
  "targets": [
    {
      "name": "target_1",
      "virtual_key":"xx"
    },
    {
      "name": "target_2",
      "virtual_key":"yy"
    }
  ]
}

• strategy.mode: Set to conditional
• strategy.conditions: Query conditions with rules applied on metadata values or request parameters along with which target to call when the condition passes
• strategy.default: The default target name to call when none of the conditions pass
• targets: Array of target objects with unique names and provider details. These target names are referenced in the conditions objects above.

conditions and default are required params for the conditional strategy.

Structure of conditions Object

conditions are where you will actually write the routing rules. Here’s a sample condition object:

Metadata Based

  {
    "query": { "metadata.user_plan": { "$eq": "paid" } },
    "then": "finetuned-gpt4"
  }

query: Write the exact rule for checking metadata values or request parameters

then: Define which target to call if the query PASSES

List of Condition Query Operators

Operator	Description
$eq	Equals
$ne	Not equals
$in	In array
$nin	Not in array
$regex	Match the regex
$gt	Greater than
$gte	Greater than or equal to
$lt	Less than
$lte	Less than or equal to

Logical Query Operators

• $and: All conditions must be true
• $or: At least one condition must be true

** Example Condition objects with Logical Operators**

AND

{
  "$and": [
    { "metadata.user_type": { "$eq": "pro" } },
    { "metadata.model": { "$eq": "gpt-4" } }
  ]
}

1. You can write nested queries (with $and, $or operators)
2. When a condition is incorrect or it fails, Portkey moves on to the next condition until it finds a successful condition.
3. If no conditions pass, then the default target name is called
4. Since Portkey iterates through the queries sequentially, the order of your conditions is important
5. When using parameter-based routing, the user must send the exact parameters expected in the conditions, or the request will give an error

2. Getting Config ID

Based on the conditions and the Config structure described above, you can create your Config in Portkey UI, and save it to get Config ID. The UI also helps you autofill and autoformat your Config.

3. Implementing Conditional Routing

Conditional routing is enabled through Portkey’s Gateway Configs by following these steps:

1. Create a conditional config in Portkey UI (app.portkey.ai/configs)
2. Save the config to get a Config ID
3. Use this Config ID in your requests

Parameter-Based Routing

Route requests based on the values of parameters like model, temperature, or max_tokens

Metadata-Based Routing

Route requests based on custom metadata fields you include with your request

You can also combine both approaches for more sophisticated routing logic

Routing Based on Request Parameters

With conditional routing, you can route based on any parameter in your LLM request (model, temperature, max_tokens, etc.).

Portkey only supports params routing for primitive types (string, number, boolean). Complex types like arrays and objects are not supported.

Example: Model Parameter Based Routing

This example routes based on the model parameter, allowing you to use aliases instead of specific model names:

{
  "strategy": {
    "mode": "conditional",
    "conditions": [
      {
        "query": {
          "params.model": {
            "$eq": "fastest"
          }
        },
        "then": "fastest-model-target"
      },
      {
        "query": {
          "params.model": {
            "$eq": "smartest"
          }
        },
        "then": "smartest-model-target"
      }
    ],
    "default": "fastest-model-target"
  },
  "targets": [
    {
      "name": "smartest-model-target",
      "virtual_key": "anthropic-vk",
      "override_params": {
        "model": "claude-3.5-sonnet"
      }
    },
    {
      "name": "fastest-model-target",
      "virtual_key": "oai-vk",
      "override_params": {
        "model": "gpt-4o-mini"
      }
    }
  ]
}

Using the config:

from portkey_ai import Portkey
client = Portkey(
    api_key="PORTKEY_API_KEY",
    config="model-routing-config-id"
)

# This will use the "smartest" model (claude-3.5-sonnet)
completion = client.chat.completions.create(
  model="smartest",  # This value matches our conditional routing condition
  messages=[
    {"role": "developer", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Explain quantum computing to a 5-year-old"}
  ]
)

# This will use the "fastest" model (gpt-4o-mini)
completion = client.chat.completions.create(
  model="fastest",  # This value matches our conditional routing condition
  messages=[
    {"role": "developer", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What's 2+2?"}
  ]
)

When using parameter-based routing, make sure to include the parameter specified in your routing condition in your request. In the example above, you need to include model in your request for routing to work properly.

Routing Based on Metadata

Portkey support metadata-based routing, which requires sending custom metadata with your request.

Applying Conditional Routing Based on Metadata

Simple Conditional Config

In this example we are routing our request based on user_plan metadata sent along request. If the user is on a paid plan, we route to a finetuned-gpt4 model, otherwise we route to a base-gpt4 model.

{
  "strategy": {
    "mode": "conditional",
    "conditions": [
      {
        "query": { "metadata.user_plan": { "$eq": "paid" } },
        "then": "finetuned-gpt4"
      },
      {
        "query": { "metadata.user_plan": { "$eq": "free" } },
        "then": "base-gpt4"
      }
    ],
    "default": "base-gpt4"
  },
  "targets": [
    {
      "name": "finetuned-gpt4",
      "virtual_key":"xx",
      "override_params": {
          "model": "ft://gpt4-xxxxx"
      }
    },
    {
      "name": "base-gpt4",
      "virtual_key":"yy",
      "override_params": {
          "model": "gpt-4"
      }
    }
}

Using the config:

from portkey_ai import Portkey

portkey = Portkey(
    api_key="PORTKEY_API_KEY",
    config="my-conditional-router-config"
)

response = portkey.with_options(
    metadata = {
        "user_plan": "free",
        "environment": "production",
        "session_id": "1729"
}).chat.completions.create(
    messages = [{ "role": 'user', "content": 'What is 1729' }]
)

You can combine both metadata and request parameters in your conditions. For example, you can route based on a combination of metadata.user_plan and params.model.

Combined Routing with Multiple Conditions

You can combine metadata and parameter conditions for more sophisticated routing:

from portkey_ai import Portkey

client = Portkey(
    api_key="PORTKEY_API_KEY",
    config="combined-routing-config-id"
)

# Route based on both metadata and temperature parameter
completion = client.with_options(
    metadata={"user_tier": "enterprise"}
).chat.completions.create(
    temperature=0.9,  # High temperature will route to creative model
    messages=[{"role": "user", "content": "Write a story about dragons"}]
)

Config example for combined conditions:

{
  "strategy": {
    "mode": "conditional",
    "conditions": [
      {
        "query": {
          "$and": [
            { "metadata.user_tier": { "$eq": "enterprise" } },
            { "params.temperature": { "$gte": 0.7 } }
          ]
        },
        "then": "creative-premium-target"
      }
    ],
    "default": "standard-target"
  },
  "targets": [
    {
      "name": "creative-premium-target",
      "virtual_key": "anthropic-vk",
      "override_params": { "model": "claude-3-opus" }
    },
    {
      "name": "standard-target",
      "virtual_key": "openai-vk",
      "override_params": { "model": "gpt-3.5-turbo" }
    }
  ]
}

More Examples Using Conditional Routing

Here are practical examples of how you can leverage conditional routing to handle real-world challenges in your LLM applications. These examples showcase common patterns that help optimize cost, performance, compliance, and feature deployment without changing your application code.

User-Based Routing

User Subscription Tier

Route premium users to advanced models and free users to cost-effective ones.

{
  "strategy": {
    "mode": "conditional",
    "conditions": [
      {
        "query": { "metadata.user_tier": { "$eq": "premium" } },
        "then": "premium-model"
      },
      {
        "query": { "metadata.user_tier": { "$eq": "free" } },
        "then": "basic-model"
      }
    ],
    "default": "basic-model"
  },
  "targets": [
    {
      "name": "premium-model", "virtual_key": "openai-vk",
      "override_params": { "model": "gpt-4o" }
    },
    {
      "name": "basic-model", "virtual_key": "openai-vk",
      "override_params": { "model": "gpt-3.5-turbo" }
    }
  ]
}

Model Selection & Parameter Routing

Routing Based on Model Name Aliases

Use intuitive aliases instead of remembering specific model names.

{
  "strategy": {
    "mode": "conditional",
    "conditions": [
      {
        "query": { "params.model": { "$eq": "fastest" } },
        "then": "fast-model"
      },
      {
        "query": { "params.model": { "$eq": "smartest" } },
        "then": "smart-model"
      }
    ],
    "default": "balanced-model"
  },
  "targets": [
    {
      "name": "smart-model", "virtual_key": "anthropic-vk",
      "override_params": { "model": "claude-3.5-sonnet" }
    },
    {
      "name": "fast-model", "virtual_key": "openai-vk",
      "override_params": { "model": "gpt-4o-mini" }
    },
    {
      "name": "balanced-model", "virtual_key": "openai-vk",
      "override_params": { "model": "gpt-4o" }
    }
  ]
}

Application Features & Testing

Task-Specific Routing

Route different tasks to specialized models for optimal results.

{
  "strategy": {
    "mode": "conditional",
    "conditions": [
      {
        "query": { "metadata.task": { "$eq": "coding" } },
        "then": "coding-model"
      },
      {
        "query": { "metadata.task": { "$eq": "writing" } },
        "then": "writing-model"
      }
    ],
    "default": "general-model"
  },
  "targets": [
    {
      "name": "coding-model", "virtual_key": "openai-vk",
      "override_params": { "model": "gpt-4o" }
    },
    {
      "name": "writing-model", "virtual_key": "anthropic-vk",
      "override_params": { "model": "claude-3-opus" }
    },
    {
      "name": "general-model", "virtual_key": "openai-vk",
      "override_params": { "model": "gpt-3.5-turbo" }
    }
  ]
}

Using Conditional Router with Guardrails

Input Guardrails

You can apply input guardrails to specific targets in your conditional routing configuration. This allows you to validate or transform the input before it’s sent to the LLM:

{
  "strategy": {
    "mode": "conditional",
    "conditions": [
      {
        "query": {
          "metadata.user_plan": { "$eq": "paid" }
        },
        "then": "premium-model"
      },
      {
        "query": {
          "metadata.user_plan": { "$eq": "free" }
        },
        "then": "standard-model"
      }
    ],
    "default": "standard-model"
  },
  "targets": [
    {
      "name": "premium-model",
      "virtual_key": "openai-vk-xxx",
      "input_guardrails": ["pg-pii-detector-xxx", "pg-toxic-content-xxx"]
    },
    {
      "name": "standard-model",
      "virtual_key": "openai-vk-yyy"
    }
  ]
}

In this example, when a user with the “paid” plan sends a request, it’s routed to the premium model but first runs through two input guardrails: one for PII detection and another for toxic content filtering.

Guardrails are referenced by their IDs in the configuration. You can create guardrails in the Portkey UI and then reference them in your conditional routing config.

Learn more about Portkey Guardrails to understand how to create and manage different types of guardrails for your LLM applications.

Important Note

1. Parameter Presence: When routing based on request parameters, make sure the parameters specified in your conditions are included in your requests.

2. Primitive Types Only: Parameter-based routing works with primitive data types (strings, numbers, booleans) but not with arrays or nested objects.

3. Order Matters: Conditions are evaluated in the order they’re defined, so place more specific conditions before more general ones.

4. Error Handling: If your condition references a parameter that doesn’t exist in the request, the request will fail with an error.

5. Multiple Parameter Types: Portkey supports routing based on any parameter that can be sent in LLM requests including model, temperature, top_p, frequency_penalty, presence_penalty, max_tokens, and many others.

Join us on Discord to share your thoughts on this feature or to get help with setting up advanced routing scenarios.