OpenAI Swarm

The Portkey x Swarm integration brings advanced AI gateway capabilities, full-stack observability, and reliability features to build production-ready AI agents.

Swarm is an experimental framework by OpenAI for building multi-agent systems. It showcases the handoff & routines pattern, making agent coordination and execution lightweight, highly controllable, and easily testable. Portkey integration extends Swarm’s capabilities with production-ready features like observability, reliability, and more.

Getting Started

1. 1. Install the Portkey SDK

   pip install -U portkey-ai
   
2. 2. Configure the LLM Client used in OpenAI Swarm

   To build Swarm Agents with Portkey, you’ll need two keys:

   • Portkey API Key: Sign up on the Portkey app and copy your API key.

   • Provider: Add your LLM provider in Model Catalog to get a provider slug (e.g., @openai-prod)

   from swarm import Swarm, Agent
   from portkey_ai import Portkey
   
   portkey = Portkey(
       api_key="YOUR_PORTKEY_API_KEY", # defaults to os.environ.get("PORTKEY_API_KEY")
       provider="@YOUR_PROVIDER" 
       )
   
   client = Swarm(client=portkey)
   
3. 3. Create and Run an Agent

   In this example we are building a simple Weather Agent using OpenAI Swarm with Portkey.

   def get_weather(location) -> str:
       return "{'temp':67, 'unit':'F'}"
   
   
   agent = Agent(
       name="Agent",
       instructions="You are a helpful agent.",
       functions=[get_weather],
   )
   
   messages = [{"role": "user", "content": "What's the weather in NYC?"}]
   
   response = client.run(agent=agent, messages=messages)
   print(response.messages[-1]["content"])
   

E2E example with Function Calling in OpenAI Swarm

Here’s a complete example showing function calling and agent interaction:

from swarm import Swarm, Agent
from portkey_ai import Portkey

portkey = Portkey(
    api_key="YOUR_PORTKEY_API_KEY", # defaults to os.environ.get("PORTKEY_API_KEY")
    provider="@YOUR_PROVIDER" 
    )

client = Swarm(client=portkey)


def get_weather(location) -> str:
    return "{'temp':67, 'unit':'F'}"


agent = Agent(
    name="Agent",
    instructions="You are a helpful agent.",
    functions=[get_weather],
)

messages = [{"role": "user", "content": "What's the weather in NYC?"}]

response = client.run(agent=agent, messages=messages)
print(response.messages[-1]["content"])

The current temperature in New York City is 67°F.

Enabling Portkey Features

By routing your OpenAI Swarm requests through Portkey, you get access to the following production-grade features:

Interoperability

Call various LLMs like Anthropic, Gemini, Mistral, Azure OpenAI, Google Vertex AI, and AWS Bedrock with minimal code changes.

Caching

Speed up agent responses and save costs by storing past responses in the Portkey cache. Choose between Simple and Semantic cache modes.

Reliability

Set up fallbacks between different LLMs, load balance requests across multiple instances, set automatic retries, and request timeouts.

Metrics

Get comprehensive logs of agent interactions, including cost, tokens used, response time, and function calls. Send custom metadata for better analytics.

Logs

Access detailed logs of agent executions, function calls, and interactions. Debug and optimize your agents effectively.

Security & Compliance

Implement budget limits, role-based access control, and audit trails for your agent operations.

Continuous Improvement

Capture and analyze user feedback to improve agent performance over time.

1. Interoperability - Calling Different LLMs

When building with Swarm, you might want to experiment with different LLMs or use specific providers for different agent tasks. Portkey makes this seamless - you can switch between OpenAI, Anthropic, Gemini, Mistral, or cloud providers without changing your agent code.

Instead of managing multiple API keys and provider-specific configurations, Portkey’s Model Catalog gives you a single point of control. Here’s how you can use different LLMs with your Swarm agents:

Anthropic

portkey = Portkey(
    api_key="YOUR_PORTKEY_API_KEY",
    provider="@ANTHROPIC_PROVIDER" #Just change the provider slug to your preferred LLM provider
)

client = Swarm(client=portkey)

Azure OpenAI

portkey = Portkey(
    api_key="YOUR_PORTKEY_API_KEY",
    provider="@AZURE_OPENAI_PROVIDER" #Just change the provider slug to your preferred LLM provider
)

client = Swarm(client=portkey)

2. Caching - Speed Up Agent Responses

Agent operations often involve repetitive queries or similar tasks. Every time your agent makes an LLM call, you’re paying for tokens and waiting for responses. Portkey’s caching system can significantly reduce both costs and latency.

Portkey offers two powerful caching modes:

Simple Cache: Perfect for exact matches - when your agents make identical requests. Ideal for deterministic operations like function calling or FAQ-type queries.

Semantic Cache: Uses embedding-based matching to identify similar queries. Great for natural language interactions where users might ask the same thing in different ways.

config = {
    "cache": {
        "mode": "semantic",  # or "simple" for exact matching
        "max_age": 3600000  # cache duration in milliseconds
    }
}

portkey = Portkey(
    api_key="YOUR_PORTKEY_API_KEY",
    provider="@YOUR_PROVIDER",
    config=config
)

3. Reliability - Keep Your Agents Running Smoothly

When running agents in production, things can go wrong - API rate limits, network issues, or provider outages. Portkey’s reliability features ensure your agents keep running smoothly even when problems occur.

Automatic Retries

Handles temporary failures automatically. If an LLM call fails, Portkey will retry the same request for the specified number of times - perfect for rate limits or network blips.

Request Timeouts

Prevent your agents from hanging. Set timeouts to ensure you get responses (or can fail gracefully) within your required timeframes.

Conditional Routing

Send different requests to different providers. Route complex reasoning to GPT-4, creative tasks to Claude, and quick responses to Gemini based on your needs.

Fallbacks

Keep running even if your primary provider fails. Automatically switch to backup providers to maintain availability.

Load Balancing

Spread requests across multiple API keys or providers. Great for high-volume agent operations and staying within rate limits.

4. Observability - Understand Your Agents

Building agents is the first step - but how do you know they’re working effectively? Portkey provides comprehensive visibility into your agent operations through multiple lenses:

Metrics Dashboard: Track 40+ key performance indicators like:

• Cost per agent interaction
• Response times and latency
• Token usage and efficiency
• Success/failure rates
• Cache hit rates

Send Custom Metadata with your requests

Add trace IDs to track specific workflows:

portkey = Portkey(
    api_key="YOUR_PORTKEY_API_KEY",
    provider="@YOUR_PROVIDER",
    trace_id="weather_workflow_123",
    metadata={
        "agent": "weather_agent",
        "environment": "production"
    }
)

5. Logs and Traces

Logs are essential for understanding agent behavior, diagnosing issues, and improving performance. They provide a detailed record of agent activities and tool use, which is crucial for debugging and optimizing processes.

Access a dedicated section to view records of agent executions, including parameters, outcomes, function calls, and errors. Filter logs based on multiple parameters such as trace ID, model, tokens used, and metadata.

6. Security & Compliance - Enterprise-Ready Controls

When deploying agents in production, security is crucial. Portkey provides enterprise-grade security features:

Budget Controls

Set and monitor spending limits per provider. Get alerts before costs exceed thresholds.

Access Management

Control who can access what. Assign roles and permissions for your team members.

Audit Logging

Track all changes and access. Know who modified agent settings and when.

Data Privacy

Configure data retention and processing policies to meet your compliance needs.

Configure these settings in the Portkey Dashboard or programmatically through the API.

7. Continuous Improvement

Now that you know how to trace & log your Llamaindex requests to Portkey, you can also start capturing user feedback to improve your app!

You can append qualitative as well as quantitative feedback to any trace ID with the portkey.feedback.create method:

Adding Feedback

from portkey_ai import Portkey

portkey = Portkey(
    api_key="PORTKEY_API_KEY",
    provider="@YOUR_OPENAI_PROVIDER"
)

feedback = portkey.feedback.create(
    trace_id="YOUR_LLAMAINDEX_TRACE_ID",
    value=5,  # Integer between -10 and 10
    weight=1,  # Optional
    metadata={
        # Pass any additional context here like comments, _user and more
    }
)

print(feedback)

Portkey Config

Many of these features are driven by Portkey’s Config architecture. The Portkey app simplifies creating, managing, and versioning your Configs.

For more information on using these features and setting up your Config, please refer to the Portkey documentation.