promptfoo is an open source library (and CLI) for evaluating LLM output quality.

​
By using Portkey with Promptfoo you can:

  1. Manage, version, and collaborate on various prompts with Portkey and easily call them in Promptfoo
  2. Run Promptfoo on 1600+ LLMs, including locally or privately hosted LLMs
  3. Log all requests, segment them as needed with custom metadata, and get granular cost, performance metrics for all Promptfoo runs
  4. Avoid Promptfoo rate limits & leverage cache

Let’s see how these work!

​
1. Reference Prompts from Portkey in Promptfoo

  1. Set the PORTKEY_API_KEY environment variable in your Promptfoo project
  2. In your configuration YAML, use the portkey:// prefix for your prompts, followed by your Portkey prompt ID.

For example:

prompts:
  - portkey://pp-test-promp-669f48
providers:
  - openai:gpt-3.5-turbo-0613
tests:
  - vars:
      topic: ...

Variables from your Promptfoo test cases will be automatically plugged into the Portkey prompt as variables. The resulting prompt will be rendered and returned to promptfoo, and used as the prompt for the test case.

Note that promptfoo does not follow the temperature, model, and other parameters set in Portkey. You must set them in the providers configuration yourself.

​
2. Route to Anthropic, Google, Groq, and More

  1. Set the PORTKEY_API_KEY environment variable
  2. While adding the provider in your config YAML, set the model name with portkey prefix (like portkey:gpt-4o)
  3. And in the configparam, set the relevant provider for the above chosen model with portkeyProvider (like portkeyProvider:openai)

​
For Example, to Call OpenAI

providers:
  id: portkey:gpt-4o
  config:
    portkeyProvider: openai

That’s it! With this, all your Promptfoo calls will now start showing up on your Portkey dashboard.

​
Let’s now call Anthropic, Google, Groq, Ollama

providers:
  id: portkey:claude-3-opus20240229
  config:
    portkeyProvider: anthropic
    # You can also add your Anthropic API key to Portkey and pass the virtual key here
    portkeyVirtualKey: ANTHROPIC_PROVIDER

​
Examples for Azure OpenAI, AWS Bedrock, Google Vertex AI

​
Using Virtual Keys

providers:
  id: portkey:xxx
  config:
    portkeyVirtualKey: YOUR_PORTKEY_AZURE_OPENAI_PROVIDER

​
Without Using Virtual Keys

First, set the AZURE_OPENAI_API_KEY environment variable.

providers:
  id: portkey:xxx
  config:
    portkeyProvider: azure-openai
    portkeyAzureResourceName: AZURE_RESOURCE_NAME
    portkeyAzureDeploymentId: AZURE_DEPLOYMENT_NAME
    portkeyAzureApiVersion: AZURE_API_VERSION

​
Using Client Credentials (JSON Web Token)

You can generate a JSON web token for your client creds, and add it to the AZURE_OPENAI_API_KEY environment variable.

providers:
  id: portkey:xxx
  config:
    portkeyProvider: azure-openai
    portkeyAzureResourceName: AZURE_RESOURCE_NAME
    portkeyAzureDeploymentId: AZURE_DEPLOYMENT_NAME
    portkeyAzureApiVersion: AZURE_API_VERSION
    # Pass your JSON Web Token with AZURE_OPENAI_API_KEY env var
    portkeyForwardHeaders: [ "Authorization" ]

​
3. Segment Requests, View Cost & Performance Metrics

Portkey automatically logs all the key details about your requests, including cost, tokens used, response time, request and response bodies, and more.

Using Portkey, you can also send custom metadata with each of your requests to further segment your logs for better analytics. Similarly, you can also trace multiple requests to a single trace ID and filter or view them separately in Portkey logs.

providers:
  id: portkey:claude-3-opus20240229
  config:
    portkeyVirtualKey: ANTHROPIC_PROVIDER
    portkeyMetadata:
      team: alpha9
      prompt: classification
    portkeyTraceId: run_1

You can filter or group data by these metadata keys on Portkey dashboards.

​
4. Avoid Promptfoo Rate Limits & Leverage Cache

Since promptfoo can make a lot of calls very quickly, you can use a loadbalanced config in Portkey with cache enabled. You can pass the config header in the same YAML.

Here’s a sample Config that you can save in the Portkey UI and get a respective config slug:

{
	"cache": { "mode": "simple" },
	"strategy": { "mode": "loadbalance" },
	"targets": [
		{ "provider":"@ACCOUNT_ONE" },
		{ "provider":"@ACCOUNT_TWO" },
		{ "provider":"@ACCOUNT_THREE" }
	]
}

And then we can just add the saved Config’s slug in the YAML:

providers:
  id: portkey:claude-3-opus20240229
  config:
    portkeyConfig: PORTKEY_CONFIG_SLUG

​
[Roadmap] View the Results of Promptfoo Evals in Portkey

We’re building support to view the eval results of promptfoo in Portkey that will let you view the results of promptfoo evals within the feedback section of Portkey.