Hugging Face

Portkey provides a robust and secure gateway to facilitate the integration of various Large Language Models (LLMs) into your applications, including all the text generation models supported by Huggingface’s Inference endpoints.

With Portkey, you can take advantage of features like fast AI gateway access, observability, prompt management, and more, all while ensuring the secure management of your LLM API keys through a virtual key system.

Provider Slug. huggingface

Portkey SDK Integration with Huggingface

Portkey provides a consistent API to interact with models from various providers. To integrate Huggingface with Portkey:

1. Install the Portkey SDK

Add the Portkey SDK to your application to interact with Huggingface’s API through Portkey’s gateway.

NodeJS

npm install --save portkey-ai

npm install --save portkey-ai

pip install portkey-ai

2. Initialize Portkey with the Virtual Key

To use Huggingface with Portkey, get your Huggingface Access token from here, then add it to Portkey to create the virtual key.

NodeJS SDK

import Portkey from 'portkey-ai'

const portkey = new Portkey({
    apiKey: "PORTKEY_API_KEY", // defaults to process.env["PORTKEY_API_KEY"]
    virtualKey: "VIRTUAL_KEY", // Your Huggingface Virtual Key
    huggingfaceBaseUrl: "HUGGINGFACE_DEDICATED_URL" // Optional: Use this if you have a dedicated server hosted on Huggingface
})

import Portkey from 'portkey-ai'

const portkey = new Portkey({
    apiKey: "PORTKEY_API_KEY", // defaults to process.env["PORTKEY_API_KEY"]
    virtualKey: "VIRTUAL_KEY", // Your Huggingface Virtual Key
    huggingfaceBaseUrl: "HUGGINGFACE_DEDICATED_URL" // Optional: Use this if you have a dedicated server hosted on Huggingface
})

from portkey_ai import Portkey

portkey = Portkey(
    api_key="PORTKEY_API_KEY",  # Replace with your Portkey API key
    virtual_key="VIRTUAL_KEY",   # Replace with your virtual key for Huggingface
    huggingface_base_url="HUGGINGFACE_DEDICATED_URL" # Optional: Use this if you have a dedicated server hosted on Huggingface
)

from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders

client = OpenAI(
    api_key="HUGGINGFACE_ACCESS_TOKEN",
    base_url=PORTKEY_GATEWAY_URL,
    default_headers=createHeaders(
        api_key="PORTKEY_API_KEY",
        provider="huggingface",
        huggingface_base_url="HUGGINGFACE_DEDICATED_URL"
    )
)

import OpenAI from "openai";
import { PORTKEY_GATEWAY_URL, createHeaders } from "portkey-ai";

const client = new OpenAI({
  apiKey: "HUGGINGFACE_ACCESS_TOKEN",
  baseURL: PORTKEY_GATEWAY_URL,
  defaultHeaders: createHeaders({
    provider: "huggingface",
    apiKey: "PORTKEY_API_KEY",
    huggingfaceBaseUrl: "HUGGINGFACE_DEDICATED_URL"
  }),
});

3. Invoke Chat Completions with Huggingface

Use the Portkey instance to send requests to Huggingface. You can also override the virtual key directly in the API call if needed.

NodeJS SDK

const chatCompletion = await portkey.chat.completions.create({
    messages: [{ role: 'user', content: 'Say this is a test' }],
    model: 'meta-llama/Meta-Llama-3.1-8B-Instruct', // make sure your model is hot
});

console.log(chatCompletion.choices[0].message.content);

const chatCompletion = await portkey.chat.completions.create({
    messages: [{ role: 'user', content: 'Say this is a test' }],
    model: 'meta-llama/Meta-Llama-3.1-8B-Instruct', // make sure your model is hot
});

console.log(chatCompletion.choices[0].message.content);

chat_completion = portkey.chat.completions.create(
    messages= [{ "role": 'user', "content": 'Say this is a test' }],
    model= 'meta-llama/meta-llama-3.1-8b-instruct', # make sure your model is hot
)

print(chat_completion.choices[0].message.content)

chat_completion = client.chat.completions.create(
    messages = [{ "role": 'user', "content": 'Say this is a test' }],
    model = 'meta-llama/meta-llama-3.1-8b-instruct', # make sure your model is hot
)

print(chat_completion.choices[0].message.content)

async function main() {
    const chatCompletion = await client.chat.completions.create({
        model: "meta-llama/meta-llama-3.1-8b-instruct", // make sure your model is hot
        messages: [{ role: "user", content: "How many points to Gryffindor?" }],
    });
    console.log(chatCompletion.choices[0].message.content);
}

main();

chat_completion = client.chat.completions.create(
    messages = [{ "role": 'user', "content": 'Say this is a test' }],
    model = 'meta-llama/meta-llama-3.1-8b-instruct', # make sure your model is hot
)

print(chat_completion.choices[0].message.content)

Using Virtual Keys

Virtual Keys serve as Portkey’s unified authentication system for all LLM interactions, simplifying the use of multiple providers and Portkey features within your application. For self-hosted LLMs, you can configure custom authentication requirements including authorization keys, bearer tokens, or any other headers needed to access your model:

1. Navigate to Virtual Keys in your Portkey dashboard
2. Click “Add Key” and enable the “Local/Privately hosted provider” toggle
3. Configure your deployment:
   • Select the matching provider API specification (typically OpenAI)
   • Enter your model’s base URL in the Custom Host field
   • Add required authentication headers and their values
4. Click “Create” to generate your virtual key

You can now use this virtual key in your requests:

NodeJS

const portkey = new Portkey({
    apiKey: "PORTKEY_API_KEY",
    virtualKey: "YOUR_SELF_HOSTED_LLM_VIRTUAL_KEY"

async function main() {
  const response = await client.chat.completions.create({
    messages: [{ role: "user", content: "Bob the builder.." }],
    model: "your-self-hosted-model-name",
  });

console.log(response.choices[0].message.content);
})

const portkey = new Portkey({
    apiKey: "PORTKEY_API_KEY",
    virtualKey: "YOUR_SELF_HOSTED_LLM_VIRTUAL_KEY"

async function main() {
  const response = await client.chat.completions.create({
    messages: [{ role: "user", content: "Bob the builder.." }],
    model: "your-self-hosted-model-name",
  });

console.log(response.choices[0].message.content);
})

portkey = Portkey(
    api_key="PORTKEY_API_KEY",
    virtual_key="YOUR_SELF_HOSTED_LLM_VIRTUAL_KEY"
)

response = portkey.chat.completions.create(
  model="your-self-hosted-model-name",
  messages=[
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Hello!"}
  ]

print(response)
)

For more information about managing self-hosted LLMs with Portkey, see Bring Your Own LLM.

Next Steps

The complete list of features supported in the SDK are available on the link below.

SDK

You’ll find more information in the relevant sections:

1. Add metadata to your requests
2. Add gateway configs to your Huggingface requests requests
3. Tracing Huggingface requests
4. Setup a fallback from OpenAI to Huggingface APIs