> ## Documentation Index
> Fetch the complete documentation index at: https://docs.portkey.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Batches

> Perform batch inference with Vertex AI

With Prisma AIRS AI Gateway, you can perform batch inference operations with Vertex AI models. This is the most efficient way to:

* Process large volumes of data with Vertex AI models
* Test your data with different foundation models
* Perform A/B testing with different foundation models

## Before You Start

1. **AI Gateway API key** and a **Vertex AI provider** configured in Model Catalog.
2. A **GCS bucket** in the same region as your model + `aiplatform-service-agent` permission on the file.
3. *(Only for gateway-native batching)* A **AI Gateway File** (`input_file_id`).
4. Familiarity with the [Create Batch OpenAPI spec](https://docs.portkey.ai/docs/aigw/api-reference/batch/create-batch.md).

> [!NOTE]
>
> The AI Gateway supports **two modes** on Vertex:
>
> * **[Provider Batch API](#using-vertex-batch-api-through-the-ai-gateway)** (cheapest, completion\_window: `24h`, `48h`, etc.)
> * **[AI Gateway Batch API](https://docs.portkey.ai/docs/aigw/product/ai-gateway/batches.md#portkey-batch-api-mode)** (fast, provider-agnostic, completion\_window: `immediate`)

## Using Vertex Batch API through the AI Gateway

### Upload a file for batch inference

```bash curl theme={"system"}
curl -X POST --header 'Authorization: Bearer <portkey_api_key>' \
 --header 'x-portkey-provider: @your-vertex-provider' \
 --header 'x-portkey-vertex-storage-bucket-name: <bucket_name>' \
 --header 'x-portkey-provider-file-name: <file_name>.jsonl' \
 --header 'x-portkey-provider-model: <model_name>' \
 --form 'purpose="batch"' \
 --form 'file=@dataset.jsonl' \
 'https://aigw.portkey.ai/v1/files'
```

```python OpenAI Python theme={"system"}
from openai import OpenAI

openai = OpenAI(
    api_key="PORTKEY_API_KEY",
    base_url="https://aigw.portkey.ai/v1",
    default_headers={"x-portkey-provider": "@VERTEX_PROVIDER", "x-portkey-vertex-storage-bucket-name": "your_bucket_name", "x-portkey-provider-file-name": "your_file_name.jsonl", "x-portkey-provider-model": "gemini-1.5-flash-001"}
)

# Upload a file for batch inference
file = openai.files.create(
    file=open("dataset.jsonl", "rb"),
    purpose="batch"
)

print(file)
```

```javascript OpenAI NodeJS theme={"system"}
import OpenAI from 'openai';
import * as fs from 'fs';

const openai = new OpenAI({
    apiKey: "PORTKEY_API_KEY",
    baseURL: "https://aigw.portkey.ai/v1",
    defaultHeaders: { "x-portkey-provider": "@VERTEX_PROVIDER", "x-portkey-vertex-storage-bucket-name": "your_bucket_name", "x-portkey-provider-file-name": "your_file_name.jsonl", "x-portkey-provider-model": "gemini-1.5-flash-001" }
});

(async () => {
    // Upload a file for batch inference
    const file = await openai.files.create({
        file: fs.createReadStream("dataset.jsonl"),
        purpose: "batch"
    });

    console.log(file);
})();
```

### Create a batch job

```bash curl theme={"system"}
curl -X POST --header 'Content-Type: application/json' \
 --header 'Authorization: Bearer <portkey_api_key>' \
 --data \
 $'{"input_file_id": "<file_id>", "endpoint": "/v1/chat/completions", "completion_window": "24h", "model":"@your-vertex-provider/gemini-1.5-flash-001"}' \
'https://aigw.portkey.ai/v1/batches'
```

```python OpenAI Python theme={"system"}
from openai import OpenAI

openai = OpenAI(
    api_key="PORTKEY_API_KEY",
    base_url="https://aigw.portkey.ai/v1"
)

# Create a batch inference job
batch_job = openai.batches.create(
    input_file_id="<file_id>", # File ID from the upload step
    endpoint="/v1/chat/completions", # API endpoint to use
    completion_window="24h", # Time window for completion
    model="@VERTEX_PROVIDER/gemini-1.5-flash-001"
)

print(batch_job)
```

```javascript OpenAI NodeJS theme={"system"}
import OpenAI from 'openai';

const openai = new OpenAI({
    apiKey: "PORTKEY_API_KEY",
    baseURL: "https://aigw.portkey.ai/v1"
});

(async () => {
    // Create a batch inference job
    const batchJob = await openai.batches.create({
        input_file_id: "<file_id>", // File ID from the upload step
        endpoint: "/v1/chat/completions", // API endpoint to use
        completion_window: "24h", // Time window for completion
        model: "@VERTEX_PROVIDER/gemini-1.5-flash-001"
    });

    console.log(batchJob);
})();
```

### List batch jobs

```bash curl theme={"system"}
curl -X GET --header 'Authorization: Bearer <portkey_api_key>' \
--header 'x-portkey-provider: @your-vertex-provider' \
'https://aigw.portkey.ai/v1/batches'
```

```python OpenAI Python theme={"system"}
from openai import OpenAI

openai = OpenAI(
    api_key="PORTKEY_API_KEY",
    base_url="https://aigw.portkey.ai/v1",
    default_headers={"x-portkey-provider": "@VERTEX_PROVIDER"}
)

# List all batch jobs
jobs = openai.batches.list(
    limit=10  # Optional: Number of jobs to retrieve (default: 20)
)

print(jobs)
```

```javascript OpenAI NodeJS theme={"system"}
import OpenAI from 'openai';

const openai = new OpenAI({
    apiKey: "PORTKEY_API_KEY",
    baseURL: "https://aigw.portkey.ai/v1",
    defaultHeaders: { "x-portkey-provider": "@VERTEX_PROVIDER" }
});

(async () => {
    // List all batch jobs
    const jobs = await openai.batches.list({
        limit: 10  // Optional:
    });

    console.log(jobs);
})();
```

### Get a batch job

```bash curl theme={"system"}
curl -X GET --header 'Authorization: Bearer <portkey_api_key>' \
--header 'x-portkey-provider: @your-vertex-provider' \
'https://aigw.portkey.ai/v1/batches/<job_id>'
```

```python OpenAI Python theme={"system"}
from openai import OpenAI

openai = OpenAI(
    api_key="PORTKEY_API_KEY",
    base_url="https://aigw.portkey.ai/v1",
    default_headers={"x-portkey-provider": "@VERTEX_PROVIDER"}
)

# Retrieve a specific batch job
job = openai.batches.retrieve(
    "job_id"  # The ID of the batch job to retrieve
)

print(job)
```

```javascript OpenAI NodeJS theme={"system"}
import OpenAI from 'openai';

const openai = new OpenAI({
    apiKey: "PORTKEY_API_KEY",
    baseURL: "https://aigw.portkey.ai/v1",
    defaultHeaders: { "x-portkey-provider": "@VERTEX_PROVIDER" }
});

(async () => {
    // Retrieve a specific batch job
    const job = await openai.batches.retrieve(
        "job_id"  // The ID of the batch job to retrieve
    );

    console.log(job);
})();
```

### Get batch job output

```bash curl theme={"system"}
curl -X GET --header 'Authorization: Bearer <portkey_api_key>' \
--header 'x-portkey-provider: @your-vertex-provider' \
'https://aigw.portkey.ai/v1/batches/<job_id>/output'
```


## Related topics

- [Batches](https://docs.portkey.ai/docs/aigw/integrations/llms/claude-platform-aws/batches.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.