> ## Documentation Index
> Fetch the complete documentation index at: https://docs.portkey.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Batches

> Perform batch inference with Bedrock

The Prisma AIRS AI Gateway lets you run Bedrock batch jobs without any manual S3 wrangling—simply upload an [OpenAI-format .jsonl file](https://platform.openai.com/docs/guides/batch#1-preparing-your-batch-file) and the AI Gateway converts it on-the-fly to the Bedrock format.

Supported batch endpoints:

* **Chat Completions** (`/v1/chat/completions`)
* **Embeddings** (`/v1/embeddings`)

This is the most efficient way to

* Test your data with different foundation models
* Perform A/B testing with different foundation models
* Perform batch inference with different foundation models

> [!NOTE]
>
> The AI Gateway supports **two modes** on Bedrock:
>
> * **[Provider Batch API](#using-bedrock-batch-api-through-the-ai-gateway)** (cheapest, completion\_window: `24h`, `48h`, etc.)
> * **[AI Gateway Batch API  ⭐️](https://docs.portkey.ai/docs/aigw/product/ai-gateway/batches.md#portkey-batch-api-mode)** (fast, provider-agnostic, completion\_window: `immediate`)

## Before You Start

1. **AI Gateway API key** (`$PORTKEY_API_KEY`).
2. **Bedrock credentials** — either an AI Gateway *Provider from Model Catalog* or explicit AWS keys (`aws_access_key_id`, `aws_secret_access_key`, `aws_region`, optional `aws_session_token`).
3. **S3 bucket** with read/write access for inputs and outputs.
4. **IAM roles** (see *Permissions & IAM* below).
5. Optional: a **AI Gateway File** (`input_file_id`) — **required only when you set `completion_window:"immediate"`** (gateway-batch mode).

## Using Bedrock Batch API through the AI Gateway

### Create Batch Job

```bash curl theme={"system"}
curl --location 'https://aigw.portkey.ai/v1/batches' \
--header 'Authorization: Bearer <portkey_api_key>' \
--header 'x-portkey-aws-access-key-id: {YOUR_AWS_ACCESS_KEY_ID}' \
--header 'x-portkey-aws-secret-access-key: {YOUR_AWS_SECRET_ACCESS_KEY}' \
--header 'x-portkey-aws-region: {YOUR_AWS_REGION}' \
--header 'Content-Type: application/json' \
--data '{
    "model": "meta.llama3-1-8b-instruct-v1:0",
    "input_file_id": "s3%3A%2F%2Fgenerations-raw-west-2%2Fbatch_files%2Fllama2%2Fbatch_chat_completions_101_requests.jsonl",
    "role_arn": "arn:aws:iam::12312:role/BedrockBatchRole", // the role to use for creating the batch job
    "output_data_config": {            // output_data_config is optional, if you want to specify a different output location for the batch job, default is the same as the input file
        "s3OutputDataConfig": {
            "s3Uri": "s3://generations-raw/",
            "s3EncryptionKeyId": "arn:aws:kms:us-west-2:517194595696:key/89b483cb-130d-497b-aa37-7db177e7cd32" // this is optional, if you want to use a KMS key to encrypt the output data
        }
    },
    "job_name": "anthropi-requests" // optional
}'
```

```javascript OpenAI NodeJS theme={"system"}
import OpenAI from 'openai'; // We're using the v4 SDK

const openai = new OpenAI({
  apiKey: "PORTKEY_API_KEY", // defaults to process.env["OPENAI_API_KEY"],
  baseURL: "https://aigw.portkey.ai/v1",
  defaultHeaders: {
      "x-portkey-provider": "openai",
      // defaults to process.env["PORTKEY_API_KEY"]
      "x-portkey-aws-access-key-id": "YOUR_AWS_ACCESS_KEY_ID",
      "x-portkey-aws-secret-access-key": "YOUR_AWS_SECRET_ACCESS_KEY",
      "x-portkey-aws-region": "YOUR_AWS_REGION",
      "x-portkey-aws-s3-bucket": "YOUR_AWS_S3_BUCKET",
      "x-portkey-aws-s3-object-key": "YOUR_AWS_S3_OBJECT_KEY",
      "x-portkey-aws-bedrock-model": "YOUR_AWS_BEDROCK_MODEL",
  }
});

const startBatch = async () => {
  const startBatchResponse = await openai.batches.create({
    input_file_id: "file_id", // file id of the input file
    endpoint: "endpoint", // ex: /v1/chat/completions
    completion_window: "completion_window", // ex: 24h
    metadata: {}, // metadata for the batch
    role_arn: "arn:aws:iam::12312:role/BedrockBatchRole", // the role to use for creating the batch job
    model: "anthropic.claude-3-5-sonnet-20240620-v1:0", // the model to use for the batch
    output_data_config: {
      s3OutputDataConfig: {
        s3Uri: "s3://generations-raw/",
        s3EncryptionKeyId: "arn:aws:kms:us-west-2:517194595696:key/89b483cb-130d-497b-aa37-7db177e7cd32" // this is optional, if you want to use a KMS key to encrypt the output data
      }
    }, // output_data_config is optional, if you want to specify a different output location for the batch job, default is the same as the input file
    job_name: "anthropi-requests-test" // optional
  });

  console.log(startBatchResponse);
}

await startBatch();
```

```python OpenAI Python theme={"system"}
from openai import OpenAI

openai = OpenAI(
    api_key="PORTKEY_API_KEY",
    base_url="https://aigw.portkey.ai/v1",
    default_headers={"x-portkey-provider": "openai", "x-portkey-aws-access-key-id": "YOUR_AWS_ACCESS_KEY_ID", "x-portkey-aws-secret-access-key": "YOUR_AWS_SECRET_ACCESS_KEY", "x-portkey-aws-region": "YOUR_AWS_REGION", "x-portkey-aws-s3-bucket": "YOUR_AWS_S3_BUCKET", "x-portkey-aws-s3-object-key": "YOUR_AWS_S3_OBJECT_KEY", "x-portkey-aws-bedrock-model": "YOUR_AWS_BEDROCK_MODEL"}
)

start_batch_response = openai.batches.create(
  input_file_id="file_id", # file id of the input file
  endpoint="endpoint", # ex: /v1/chat/completions
  completion_window="completion_window", # ex: 24h
  metadata={}, # metadata for the batch
  role_arn="arn:aws:iam::12312:role/BedrockBatchRole", # the role to use for creating the batch job
  model="anthropic.claude-3-5-sonnet-20240620-v1:0", # the model to use for the batch
  output_data_config={
    "s3OutputDataConfig": {
      "s3Uri": "s3://generations-raw/",
      "s3EncryptionKeyId": "arn:aws:kms:us-west-2:517194595696:key/89b483cb-130d-497b-aa37-7db177e7cd32" // this is optional, if you want to use a KMS key to encrypt the output data
    }
  }, # output_data_config is optional, if you want to specify a different output location for the batch job, default is the same as the input file
  job_name="anthropi-requests-test" # optional
)
print(start_batch_response)
```

### List Batch Jobs

```bash curl theme={"system"}
curl --location 'https://aigw.portkey.ai/v1/batches' \
--header 'Authorization: Bearer <portkey_api_key>' \
--header 'x-portkey-provider: @provider' \
```

```javascript OpenAI NodeJS theme={"system"}
import OpenAI from 'openai'; // We're using the v4 SDK

const openai = new OpenAI({
  apiKey: "PORTKEY_API_KEY", // defaults to process.env["OPENAI_API_KEY"],
  baseURL: "https://aigw.portkey.ai/v1",
  defaultHeaders: {
      "x-portkey-provider": "openai",
      // defaults to process.env["PORTKEY_API_KEY"]
      "x-portkey-aws-access-key-id": "YOUR_AWS_ACCESS_KEY_ID",
      "x-portkey-aws-secret-access-key": "YOUR_AWS_SECRET_ACCESS_KEY",
      "x-portkey-aws-region": "YOUR_AWS_REGION",
  }
});

const listBatches = async () => {
  const batches = await openai.batches.list();

  console.log(batches);
}

await listBatches();
```

```python OpenAI Python theme={"system"}
from openai import OpenAI

openai = OpenAI(
    api_key="PORTKEY_API_KEY",
    base_url="https://aigw.portkey.ai/v1",
    default_headers={"x-portkey-provider": "openai", "x-portkey-aws-access-key-id": "YOUR_AWS_ACCESS_KEY_ID", "x-portkey-aws-secret-access-key": "YOUR_AWS_SECRET_ACCESS_KEY", "x-portkey-aws-region": "YOUR_AWS_REGION"}
)

batches = openai.batches.list()

print(batches)
```

### Get Batch Job Details

```bash curl theme={"system"}
curl --location 'https://aigw.portkey.ai/v1/batches/<batch_id>' \
--header 'Authorization: Bearer <portkey_api_key>' \
--header 'x-portkey-provider: @provider' \
```

```javascript OpenAI NodeJS theme={"system"}
import OpenAI from 'openai'; // We're using the v4 SDK

const openai = new OpenAI({
  apiKey: "PORTKEY_API_KEY", // defaults to process.env["OPENAI_API_KEY"],
  baseURL: "https://aigw.portkey.ai/v1",
  defaultHeaders: {
      // defaults to process.env["PORTKEY_API_KEY"]
      "x-portkey-provider": "@BEDROCK_PROVIDER",
  }
});

const getBatch = async () => {
  const batch = await openai.batches.retrieve(batch_id="batch_id");

  console.log(batch);
}

await getBatch();
```

```python OpenAI Python theme={"system"}
from openai import OpenAI

openai = OpenAI(
    api_key="PORTKEY_API_KEY",
    base_url="https://aigw.portkey.ai/v1",
    default_headers={"x-portkey-provider": "@BEDROCK_PROVIDER"}
)

batch = openai.batches.retrieve(batch_id="batch_id")

print(batch)
```

### Get Batch Output

```bash curl theme={"system"}
curl --location 'https://aigw.portkey.ai/v1/batches/<batch_id>/output' \
--header 'Authorization: Bearer <portkey_api_key>' \
--header 'x-portkey-provider: @provider' \
```

### List Batch Jobs

```bash curl theme={"system"}
curl --location 'https://aigw.portkey.ai/v1/batches' \
--header 'Authorization: Bearer <portkey_api_key>' \
--header 'x-portkey-provider: @provider' \
```

```javascript OpenAI NodeJS theme={"system"}
import OpenAI from 'openai'; // We're using the v4 SDK

const openai = new OpenAI({
  apiKey: "PORTKEY_API_KEY", // defaults to process.env["OPENAI_API_KEY"],
  baseURL: "https://aigw.portkey.ai/v1",
  defaultHeaders: {
      // defaults to process.env["PORTKEY_API_KEY"]
      "x-portkey-provider": "@BEDROCK_PROVIDER",
  }
});

const listBatchingJobs = async () => {
  const batching_jobs = await openai.batches.list();

  console.log(batching_jobs);
}

await listBatchingJobs();
```

```python OpenAI Python theme={"system"}
from openai import OpenAI

openai = OpenAI(
    api_key="PORTKEY_API_KEY",
    base_url="https://aigw.portkey.ai/v1",
    default_headers={"x-portkey-provider": "@BEDROCK_PROVIDER"}
)

batching_jobs = openai.batches.list()

print(batching_jobs)
```

### Cancel Batch Job

```bash curl theme={"system"}
curl --request POST --location 'https://aigw.portkey.ai/v1/batches/<batch_id>/cancel' \
--header 'Authorization: Bearer <portkey_api_key>' \
--header 'x-portkey-provider: @provider' \
```

```javascript OpenAI NodeJS theme={"system"}
import OpenAI from 'openai'; // We're using the v4 SDK

const openai = new OpenAI({
  apiKey: "PORTKEY_API_KEY", // defaults to process.env["OPENAI_API_KEY"],
  baseURL: "https://aigw.portkey.ai/v1",
  defaultHeaders: {
      // defaults to process.env["PORTKEY_API_KEY"]
      "x-portkey-provider": "@BEDROCK_PROVIDER",
  }
});

const cancelBatch = async () => {
  const cancel_batch_response = await openai.batches.cancel(batch_id="batch_id");

  console.log(cancel_batch_response);
}

await cancelBatch();
```

```python OpenAI Python theme={"system"}
from openai import OpenAI

openai = OpenAI(
    api_key="PORTKEY_API_KEY",
    base_url="https://aigw.portkey.ai/v1",
    default_headers={"x-portkey-provider": "@BEDROCK_PROVIDER"}
)

cancel_batch_response = openai.batches.cancel(batch_id="batch_id")

print(cancel_batch_response)
```

## Information about Permissions and IAM Roles

> **For Principal Role (Used in Provider)**
>
> These are the minimum permissions required to use the Bedrock Batch APIs. (Bedrock Docs]\([https://docs.aws.amazon.com/bedrock/latest/userguide/batch-inference-permissions.html](https://docs.aws.amazon.com/bedrock/latest/userguide/batch-inference-permissions.html)))
>
> ```json theme={"system"}
> {
>   "Version": "2012-10-17",
>   "Statement": [
>       {
>           "Effect": "Allow",
>           "Action": [
>               "bedrock:ListFoundationModels",
>               "bedrock:GetFoundationModel",
>               "bedrock:ListInferenceProfiles",
>               "bedrock:GetInferenceProfile",
>               "bedrock:ListCustomModels",
>               "bedrock:GetCustomModel",
>               "bedrock:TagResource",
>               "bedrock:UntagResource",
>               "bedrock:ListTagsForResource",
>               "bedrock:CreateModelInvocationJob",
>               "bedrock:GetModelInvocationJob",
>               "bedrock:ListModelInvocationJobs",
>               "bedrock:StopModelInvocationJob"
>           ],
>           "Resource": [
>               "arn:aws:bedrock:<region>:<account_id>:model-invocation-job/*",
>               "arn:aws:bedrock:<region>:<account_id>:custom-model/*", // Can be a custom model available for batching (optional)
>               "arn:aws:bedrock:<region>::foundation-model/*"
>           ]
>       },
>       {
>           "Effect": "Allow",
>           "Action": [
>               "s3:ListBucket",
>               "s3:PutObject",
>               "s3:GetObject",
>               "s3:GetObjectAttributes"
>           ],
>           "Resource": [
>               "arn:aws:s3:::<bucket>",
>               "arn:aws:s3:::<bucket>/*"
>           ]
>       },
>       {
>           "Action": [
>               "iam:PassRole"
>           ],
>           "Effect": "Allow",
>           "Resource": "arn:aws:iam::<account_id>:role/<service_role_name>",
>           "Condition": {
>               "StringEquals": {
>                   "iam:PassedToService": [
>                       "bedrock.amazonaws.com"
>                   ]
>               }
>           }
>       }
>   ]
> }
> ```

> **For Service Role (role\_arn) used for creating and executing the batch job**
>
> These are the minimum permissions required to use the Bedrock Batch APIs.
>
> Trust relationship:
>
> ```json theme={"system"}
> {
>   "Version": "2012-10-17",
>   "Statement": [
>       {
>           "Effect": "Allow",
>           "Principal": {
>               "Service": "bedrock.amazonaws.com"
>           },
>           "Action": "sts:AssumeRole",
>           "Condition": {
>               "StringEquals": {
>                   "aws:SourceAccount": "<account_id>"
>               },
>               "ArnEquals": {
>                   "aws:SourceArn": "arn:aws:bedrock:<region>:<account_id>:model-invocation-job/*"
>               }
>           }
>   ]
> }
> ```
>
> Permission Policy:
>
> ```json theme={"system"}
> {
>   "Version": "2012-10-17",
>   "Statement": [
>       {
>           "Effect": "Allow",
>           "Action": [
>               "s3:GetObject",
>               "s3:PutObject",
>               "s3:ListBucket"
>           ],
>           "Resource": [
>               "arn:aws:s3:::<bucket>",
>               "arn:aws:s3:::<bucket>/*"
>           ]
>       }
>   ]
> }
> ```

## Defaults & Limits (Bedrock Native)

| Property | Default | Notes |
| - | - | - |
| `completion_window` | `24h` | Fixed by Bedrock. |
| Job quota | 50k/day | Subject to your AWS account. |
| Max input file size | 10 GB | Across all `.jsonl` objects. |

[Portkey-Batch mode](https://docs.portkey.ai/docs/aigw/product/ai-gateway/batches.md) inherits the Gateway defaults (**25 req / 5 s**) and retries (**3×**) per request.

***

## Glossary

| Term | Meaning |
| - | - |
| **Batch Job** | Collection of asynchronous completions. |
| **AI Gateway File** (`input_file_id`) | File uploaded to the AI Gateway and re-uploaded to the provider for batch processing. |
| **Provider Slug** | Bedrock credential stored in the AI Gateway via Model Catalog; referenced by slug. |
| **Completion Window** | `immediate` → gateway batch; `24h` → Bedrock native. |

***

## See Also

* [Unified Batch Guide](https://docs.portkey.ai/docs/aigw/product/ai-gateway/batches.md)


## Related topics

- [Batches](https://docs.portkey.ai/docs/aigw/integrations/llms/claude-platform-aws/batches.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.