> ## Documentation Index
> Fetch the complete documentation index at: https://docs.portkey.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Promptfoo

> Run Promptfoo evals on 3,000+ LLMs with observability and cost tracking via Prisma AIRS AI Gateway.

[Promptfoo](https://promptfoo.dev/docs/intro) is an open source library for evaluating LLM output quality.

With the AI Gateway, you can:

* **Manage prompts** with versioning and call them in Promptfoo
* **Run evals on 3,000+ LLMs** including private/local models
* **Track costs and metrics** for all eval runs
* **Avoid rate limits** with load balancing and caching

## 1. Reference AI Gateway Prompts

Use prompts stored in the AI Gateway directly in Promptfoo:

1. Set `PORTKEY_API_KEY` environment variable
2. Use `portkey://` prefix with your prompt ID:

```yaml theme={"system"}
prompts:
  - portkey://pp-test-prompt-669f48
providers:
  - openai:gpt-3.5-turbo-0613
tests:
  - vars:
      topic: ...
```

Variables from test cases are automatically plugged into the AI Gateway prompt.

> [!NOTE]
>
> Promptfoo doesn't follow temperature, model, and other parameters set in the AI Gateway. Set them in the providers configuration.

## 2. Route to Any Provider

Set `PORTKEY_API_KEY` and configure providers with the `portkey:` prefix:

```yaml OpenAI theme={"system"}
providers:
  id: portkey:gpt-4o
  config:
    portkeyProvider: openai
```

```yaml Anthropic theme={"system"}
providers:
  id: portkey:claude-3-opus-20240229
  config:
    portkeyProvider: anthropic
    portkeyVirtualKey: "@anthropic-prod"
```

```yaml Google Gemini theme={"system"}
providers:
  id: portkey:gemini-1.5-flash-latest
  config:
    portkeyProvider: google
    portkeyVirtualKey: "@google-prod"
```

```yaml Groq theme={"system"}
providers:
  id: portkey:llama3-8b-8192
  config:
    portkeyProvider: groq
    portkeyVirtualKey: "@groq-prod"
```

```yaml Ollama (Local) theme={"system"}
providers:
  id: portkey:llama3
  config:
    portkeyProvider: ollama
    portkeyCustomHost: YOUR_OLLAMA_NGROK_URL
```

### Cloud Providers (Azure, Bedrock, Vertex)

```yaml Azure OpenAI theme={"system"}
providers:
  id: portkey:xxx
  config:
    portkeyVirtualKey: "@azure-openai-prod"
```

```yaml AWS Bedrock theme={"system"}
providers:
  id: portkey:anthropic.claude-3-sonnet-20240229-v1:0
  config:
    portkeyVirtualKey: "@bedrock-prod"
```

```yaml Google Vertex AI theme={"system"}
providers:
  id: portkey:gemini-1.5-flash-latest
  config:
    portkeyVirtualKey: "@vertex-prod"
```

## 3. Track Costs & Metrics

Add metadata to segment requests and track costs per team/project:

```yaml theme={"system"}
providers:
  id: portkey:claude-3-opus-20240229
  config:
    portkeyVirtualKey: "@anthropic-prod"
    portkeyMetadata:
      team: alpha9
      prompt: classification
    portkeyTraceId: run_1
```

Filter and group by these keys in Strata Cloud Manager.

## 4. Avoid Rate Limits with Caching

Create a config with load balancing and caching for high-volume evals:

```json theme={"system"}
{
  "cache": { "mode": "simple" },
  "strategy": { "mode": "loadbalance" },
  "targets": [
    { "provider": "@openai-key-1" },
    { "provider": "@openai-key-2" },
    { "provider": "@openai-key-3" }
  ]
}
```

Reference the config in your YAML:

```yaml theme={"system"}
providers:
  id: portkey:claude-3-opus-20240229
  config:
    portkeyConfig: pc-your-config-id
```

***

## Next Steps

> **[Model Catalog](https://docs.portkey.ai/docs/aigw/product/model-catalog.md)**
>
> Set up providers


## Related topics

- [Overview](https://docs.portkey.ai/docs/aigw/integrations/llms.md)
- [PII Redaction](https://docs.portkey.ai/docs/aigw/product/guardrails/pii-redaction.md)
- [Enterprise Gateway](https://docs.portkey.ai/docs/changelog/enterprise.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.