Triton Inference Server

Integrate Triton-hosted custom models with Portkey for production observability and reliability.

Portkey provides a robust platform to observe, govern, and manage your locally or privately hosted custom models using Triton Inference Server.

Here’s the official Triton Inference Server documentation for more details.

Integration Steps

1. Expose your Triton Server

   Expose your Triton server using a tunneling service like ngrok or make it publicly accessible. Skip this if you’re self-hosting the Gateway.

   ngrok http 8000 --host-header="localhost:8080"
   
2. Add to Model Catalog

   1. Go to Model Catalog → Add Provider
   2. Enable “Local/Privately hosted provider” toggle
   3. Select Triton as the provider type
   4. Enter your Triton server URL in Custom Host: http://localhost:8000/v2/models/mymodel
   5. Add authentication headers if needed
   6. Name your provider (e.g., my-triton)

   Complete Setup Guide

   See all setup options

3. Use in Your Application

   Python

   from portkey_ai import Portkey
   
   portkey = Portkey(
       api_key="PORTKEY_API_KEY",
       provider="@my-triton"
   )
   
   response = portkey.chat.completions.create(
       model="your-model-name",
       messages=[{"role": "user", "content": "Hello!"}]
   )
   
   print(response.choices[0].message.content)
   

   Or use custom host directly:

   Python

   from portkey_ai import Portkey
   
   portkey = Portkey(
       api_key="PORTKEY_API_KEY",
       provider="triton",
       custom_host="http://localhost:8000/v2/models/mymodel",
       Authorization="AUTH_KEY"  # If needed
   )
   

---

Next Steps

Gateway Configs

Add retries, timeouts, and fallbacks

Observability

Monitor your Triton deployments

Custom Host Guide

Learn more about custom host setup

BYOLLM Guide

Complete guide for private LLMs

For complete SDK documentation:

SDK Reference

Complete Portkey SDK documentation