Predibase

Portkey provides a robust and secure gateway to seamlessly integrate open-source and fine-tuned LLMs from Predibase into your applications. With Portkey, you can leverage powerful features like fast AI gateway, caching, observability, prompt management, and more, while securely managing your LLM API keys through a virtual key system.

Provider Slug. predibase

Portkey SDK Integration with Predibase

Using Portkey, you can call your Predibase models in the familar OpenAI-spec and try out your existing pipelines on Predibase fine-tuned models with 2 LOC change.

1. Install the Portkey SDK

Install the Portkey SDK in your project using npm or pip:

NodeJS

npm install --save portkey-ai

npm install --save portkey-ai

pip install portkey-ai

2. Initialize Portkey with the Virtual Key

To use Predibase with Portkey, get your API key from here, then add it to Portkey to create the virtual key.

NodeJS SDK

import Portkey from 'portkey-ai'

const portkey = new Portkey({
    apiKey: "PORTKEY_API_KEY", // defaults to process.env["PORTKEY_API_KEY"]
    virtualKey: "VIRTUAL_KEY" // Your Predibase Virtual Key
})

import Portkey from 'portkey-ai'

const portkey = new Portkey({
    apiKey: "PORTKEY_API_KEY", // defaults to process.env["PORTKEY_API_KEY"]
    virtualKey: "VIRTUAL_KEY" // Your Predibase Virtual Key
})

from portkey_ai import Portkey

portkey = Portkey(
    api_key="PORTKEY_API_KEY",  # Replace with your Portkey API key
    virtual_key="VIRTUAL_KEY"   # Replace with your virtual key for Predibase
)

import OpenAI from "openai";
import { PORTKEY_GATEWAY_URL, createHeaders } from "portkey-ai";

const portkey = new OpenAI({
  baseURL: PORTKEY_GATEWAY_URL,
  defaultHeaders: createHeaders({
    apiKey: "PORTKEY_API_KEY",
    virtualKey: "PREDIBASE_VIRTUAL_KEY",
  }),
});

from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders

portkey = OpenAI(
    base_url=PORTKEY_GATEWAY_URL,
    default_headers=createHeaders(
        api_key="PORTKEY_API_KEY",
        virtual_key="PREDIBASE_VIRTUAL_KEY"
    )
)

3. Invoke Chat Completions on Predibase Serverless Endpoints

Predibase offers LLMs like Llama 3, Mistral, Gemma, etc. on its serverless infra that you can query instantly.

Sending Predibase Tenand ID

Predibase expects your account tenant ID along with the API key in each request. With Portkey, you can send your Tenand ID with the user param while making your request.

Portkey OR OpenAI NodeJS SDK

const chatCompletion = await portkey.chat.completions.create({
    messages: [{ role: 'user', content: 'Say this is a test' }],
    model: 'llama-3-8b	',
    user: 'PREDIBASE_TENANT_ID'
});

console.log(chatCompletion.choices);

const chatCompletion = await portkey.chat.completions.create({
    messages: [{ role: 'user', content: 'Say this is a test' }],
    model: 'llama-3-8b	',
    user: 'PREDIBASE_TENANT_ID'
});

console.log(chatCompletion.choices);

completion = portkey.chat.completions.create(
    messages= [{ "role": 'user', "content": 'Say this is a test' }],
    model= 'llama-3-8b',
    user= "PREDIBASE_TENANT_ID"
)

print(completion)

curl https://api.portkey.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "x-portkey-api-key: $PORTKEY_API_KEY" \
  -H "x-portkey-virtual-key: $PREDIBASE_VIRTUAL_KEY" \
  -d '{
    "messages": [{"role": "user","content": "Hello!"}],
    "model": "llama-3-8b",
    "user": "PREDIBASE_TENANT_ID"
  }'

4. Invoke Predibase Fine-Tuned Models

With Portkey, you can send your fine-tune model & adapter details directly with the model param while making a request.

The format is:

model = <base_model>:<adapter-repo-name/adapter-version-number>

For example, if your base model is llama-3-8b and the adapter repo name is sentiment-analysis, you can make a request like this:

Portkey OR OpenAI NodeJS SDK

const chatCompletion = await portkey.chat.completions.create({
    messages: [{ role: 'user', content: 'Say this is a test' }],
    model: 'llama-3-8b:sentiment-analysis/1',
    user: 'PREDIBASE_TENANT_ID'
});

console.log(chatCompletion.choices);

const chatCompletion = await portkey.chat.completions.create({
    messages: [{ role: 'user', content: 'Say this is a test' }],
    model: 'llama-3-8b:sentiment-analysis/1',
    user: 'PREDIBASE_TENANT_ID'
});

console.log(chatCompletion.choices);

completion = portkey.chat.completions.create(
    messages= [{ "role": 'user', "content": 'Say this is a test' }],
    model= 'llama-3-8b:sentiment-analysis/1',
    user= "PREDIBASE_TENANT_ID"
)

print(completion)

curl https://api.portkey.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "x-portkey-api-key: $PORTKEY_API_KEY" \
  -H "x-portkey-virtual-key: $PREDIBASE_VIRTUAL_KEY" \
  -d '{
    "messages": [{"role": "user","content": "Hello!"}],
    "model": "llama-3-8b:sentiment-analysis/1",
    "user": "PREDIBASE_TENANT_ID"
  }'

---

Routing to Dedicated Deployments

Using Portkey, you can easily route to your dedicatedly deployed models as well. Just pass the dedicated deployment name in the model param:

model = "my-dedicated-mistral-deployment-name"

JSON Schema Mode

You can enforce JSON schema for all Predibase models - just set the response_format to json_object and pass the relevant schema while making your request. Portkey logs will show your JSON output separately

Portkey OR OpenAI NodeJS SDK

const chatCompletion = await portkey.chat.completions.create({
    messages: [{ role: 'user', content: 'Say this is a test' }],
    model: 'llama-3-8b	',
    user: 'PREDIBASE_TENANT_ID',
    response_format: {
      "type": "json_object",
      "schema": {"properties": {
        "name": {"maxLength": 10, "title": "Name", "type": "string"},
        "age": {"title": "Age", "type": "integer"},
        "required": ["name", "age", "strength"],
        "title": "Character",
        "type": "object"
      }
    }
});

console.log(chatCompletion.choices);

const chatCompletion = await portkey.chat.completions.create({
    messages: [{ role: 'user', content: 'Say this is a test' }],
    model: 'llama-3-8b	',
    user: 'PREDIBASE_TENANT_ID',
    response_format: {
      "type": "json_object",
      "schema": {"properties": {
        "name": {"maxLength": 10, "title": "Name", "type": "string"},
        "age": {"title": "Age", "type": "integer"},
        "required": ["name", "age", "strength"],
        "title": "Character",
        "type": "object"
      }
    }
});

console.log(chatCompletion.choices);

# Using Pydantic to define the schema
from pydantic import BaseModel, constr

# Define JSON Schema
class Character(BaseModel):
    name: constr(max_length=10)
    age: int
    strength: int

completion = portkey.chat.completions.create(
    messages= [{ "role": 'user', "content": 'Say this is a test' }],
    model= 'llama-3-8b',
    user= "PREDIBASE_TENANT_ID",
    response_format={
        "type": "json_object",
        "schema": Character.schema(),
    },
)

print(completion)

curl https://api.portkey.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "x-portkey-api-key: $PORTKEY_API_KEY" \
  -H "x-portkey-virtual-key: $PREDIBASE_VIRTUAL_KEY" \
  -d '{
    "messages": [{"role": "user","content": "Hello!"}],
    "model": "llama-3-8b",
    "user": "PREDIBASE_TENANT_ID",
    "response_format": {
      "type": "json_object",
      "schema": {"properties": {
        "name": {"maxLength": 10, "title": "Name", "type": "string"},
        "age": {"title": "Age", "type": "integer"},
        "required": ["name", "age", "strength"],
        "title": "Character",
        "type": "object"
      }
    }
  }'

---

Next Steps

The complete list of features supported in the SDK are available on the link below.

SDK

You’ll find more information in the relevant sections:

1. Add metadata to your requests
2. Add gateway configs to your Predibase requests
3. Tracing Predibase requests
4. Setup a fallback from OpenAI to Predibase