Batches

Perform batch inference with Azure OpenAI

With Portkey, you can perform Azure OpenAI Batch Inference operations. This is the most efficient way to

• Test your data with different foundation models
• Perform A/B testing with different foundation models
• Perform batch inference with different foundation models

Portkey supports two modes on Azure OpenAI:

• Provider Batch API (cheapest, completion_window: 24h, 48h, etc.)
• Portkey Batch API (fast, provider-agnostic, completion_window: immediate)

Using Azure OpenAI Batch API through Portkey

Create Batch Job

Python

from portkey_ai import Portkey

# Initialize the Portkey client
portkey = Portkey(
    api_key="PORTKEY_API_KEY",  # Replace with your Portkey API key
    provider="@PROVIDER"   
)

start_batch_response = portkey.batches.create(
  input_file_id="file_id", # file id of the input file
  endpoint="endpoint", # ex: /v1/chat/completions
  completion_window="completion_window", # ex: 24h
  metadata={} # metadata for the batch
)

print(start_batch_response)

List Batch Jobs

Python

from portkey_ai import Portkey

# Initialize the Portkey client
portkey = Portkey(
    api_key="PORTKEY_API_KEY",  # Replace with your Portkey API key
    provider="@PROVIDER"   
)

batches = portkey.batches.list()

print(batches)

Get Batch Job Details

Python

from portkey_ai import Portkey

# Initialize the Portkey client
portkey = Portkey(
    api_key="PORTKEY_API_KEY",  # Replace with your Portkey API key
    provider="@PROVIDER"   
)

batch = portkey.batches.retrieve(batch_id="batch_id")

print(batch)

Get Batch Output

curl

curl --location 'https://api.portkey.ai/v1/batches/<batch_id>/output' \
--header 'x-portkey-api-key: <portkey_api_key>' \
--header 'x-portkey-provider: @provider'

List Batch Jobs

Python

from portkey_ai import Portkey

# Initialize the Portkey client
portkey = Portkey(
    api_key="PORTKEY_API_KEY",  # Replace with your Portkey API key
    provider="@PROVIDER"   
)

batches = portkey.batches.list()

print(batches)

Cancel Batch Job

Python

from portkey_ai import Portkey

# Initialize the Portkey client
portkey = Portkey(
    api_key="PORTKEY_API_KEY",  # Replace with your Portkey API key
    provider="@PROVIDER"   
)

cancel_batch_response = portkey.batches.cancel(batch_id="batch_id")

print(cancel_batch_response)