Batches

Perform batch inference with Bedrock

Portkey lets you run Bedrock batch jobs without any manual S3 wrangling—simply upload an OpenAI-format .jsonl file and Portkey converts it on-the-fly to the Bedrock format.

This is the most efficient way to

• Test your data with different foundation models
• Perform A/B testing with different foundation models
• Perform batch inference with different foundation models

Portkey supports two modes on Bedrock:

• Provider Batch API (cheapest, completion_window: 24h, 48h, etc.)
• Portkey Batch API ⭐️ (fast, provider-agnostic, completion_window: immediate)

Before You Start

1. Portkey API key ($PORTKEY_API_KEY).
2. Bedrock credentials — either a Portkey Virtual Key or explicit AWS keys (aws_access_key_id, aws_secret_access_key, aws_region, optional aws_session_token).
3. S3 bucket with read/write access for inputs and outputs.
4. IAM roles (see Permissions & IAM below).
5. Optional: a Portkey File (input_file_id) — required only when you set completion_window:"immediate" (Portkey-Batch mode).

Using Bedrock Batch API through Portkey

Create Batch Job

Python

from portkey_ai import Portkey

# Initialize the Portkey client
portkey = Portkey(
    api_key="PORTKEY_API_KEY",  # Replace with your Portkey API key
    provider="bedrock",
    aws_access_key_id="YOUR_AWS_ACCESS_KEY_ID",
    aws_secret_access_key="YOUR_AWS_SECRET_ACCESS_KEY",
    aws_region="YOUR_AWS_REGION",
    aws_s3_bucket="YOUR_AWS_S3_BUCKET",
    aws_s3_object_key="YOUR_AWS_S3_OBJECT_KEY",
    aws_bedrock_model="YOUR_AWS_BEDROCK_MODEL"
)

start_batch_response = portkey.batches.create(
  input_file_id="file_id", # file id of the input file
  endpoint="endpoint", # ex: /v1/chat/completions
  completion_window="completion_window", # ex: 24h
  metadata={}, # metadata for the batch,
  role_arn="arn:aws:iam::12312:role/BedrockBatchRole", # the role to use for creating the batch job
  model="anthropic.claude-3-5-sonnet-20240620-v1:0", # the model to use for the batch
  output_data_config={
    "s3OutputDataConfig": {
      "s3Uri": "s3://generations-raw/",
      "s3EncryptionKeyId": "arn:aws:kms:us-west-2:517194595696:key/89b483cb-130d-497b-aa37-7db177e7cd32" # this is optional, if you want to use a KMS key to encrypt the output data
    }
  }, # output_data_config is optional, if you want to specify a different output location for the batch job, default is the same as the input file
  job_name="anthropi-requests-test" # optional
)

print(start_batch_response)

List Batch Jobs

Python

from portkey_ai import Portkey

# Initialize the Portkey client
portkey = Portkey(
    api_key="PORTKEY_API_KEY",  # Replace with your Portkey API key
    provider="bedrock",
    aws_access_key_id="YOUR_AWS_ACCESS_KEY_ID",
    aws_secret_access_key="YOUR_AWS_SECRET_ACCESS_KEY",
    aws_region="YOUR_AWS_REGION",
)

batches = portkey.batches.list()

print(batches)

Get Batch Job Details

Python

from portkey_ai import Portkey

# Initialize the Portkey client
portkey = Portkey(
    api_key="PORTKEY_API_KEY",  # Replace with your Portkey API key
    provider="@PROVIDER",   
)

batch = portkey.batches.retrieve(batch_id="batch_id")

print(batch)

Get Batch Output

curl

curl --location 'https://api.portkey.ai/v1/batches/<batch_id>/output' \
--header 'x-portkey-api-key: <portkey_api_key>' \
--header 'x-portkey-provider: @provider' \

List Batch Jobs

Python

from portkey_ai import Portkey

# Initialize the Portkey client
portkey = Portkey(
    api_key="PORTKEY_API_KEY",  # Replace with your Portkey API key
    provider="@PROVIDER",   
)

batches = portkey.batches.list()

print(batches)

Cancel Batch Job

Python

from portkey_ai import Portkey

# Initialize the Portkey client
portkey = Portkey(
    api_key="PORTKEY_API_KEY",  # Replace with your Portkey API key
    provider="@PROVIDER",   
)

cancel_batch_response = portkey.batches.cancel(batch_id="batch_id")

print(cancel_batch_response)

Information about Permissions and IAM Roles

For Principal Role (Used in Virtual Key)

These are the minimum permissions required to use the Bedrock Batch APIs.

{
  "Version": "2012-10-17",
  "Statement": [
      {
          "Effect": "Allow",
          "Action": [
              "bedrock:ListFoundationModels",
              "bedrock:GetFoundationModel",
              "bedrock:ListInferenceProfiles",
              "bedrock:GetInferenceProfile",
              "bedrock:ListCustomModels",
              "bedrock:GetCustomModel",
              "bedrock:TagResource", 
              "bedrock:UntagResource", 
              "bedrock:ListTagsForResource",
              "bedrock:CreateModelInvocationJob",
              "bedrock:GetModelInvocationJob",
              "bedrock:ListModelInvocationJobs",
              "bedrock:StopModelInvocationJob"
          ],
          "Resource": [
              "arn:aws:bedrock:<region>:<account_id>:model-customization-job/*",
              "arn:aws:bedrock:<region>:<account_id>:custom-model/*",
              "arn:aws:bedrock:<region>::foundation-model/*"
          ]
      },
      {
          "Effect": "Allow",
          "Action": [
              "s3:ListBucket",
              "s3:PutObject",
              "s3:GetObject",
              "s3:GetObjectAttributes"
          ],
          "Resource": [
              "arn:aws:s3:::<bucket>",
              "arn:aws:s3:::<bucket>/*"
          ]
      },
      {
          "Action": [
              "iam:PassRole"
          ],
          "Effect": "Allow",
          "Resource": "arn:aws:iam::<account_id>:role/<service_role_name>",
          "Condition": {
              "StringEquals": {
                  "iam:PassedToService": [
                      "bedrock.amazonaws.com"
                  ]
              }
          }
      }
  ]
}

For Service Role (role_arn) used for creating and executing the batch job

These are the minimum permissions required to use the Bedrock Batch APIs.

Trust relationship:

{
  "Version": "2012-10-17",
  "Statement": [
      {
          "Effect": "Allow",
          "Principal": {
              "Service": "bedrock.amazonaws.com"
          },
          "Action": "sts:AssumeRole",
          "Condition": {
              "StringEquals": {
                  "aws:SourceAccount": "<account_id>"
              },
              "ArnEquals": {
                  "aws:SourceArn": "arn:aws:bedrock:<region>:<account_id>:model-invocation-job/*"
              }
          }
  ]
}

Permission Policy:

{
  "Version": "2012-10-17",
  "Statement": [
      {
          "Effect": "Allow",
          "Action": [
              "s3:GetObject",
              "s3:PutObject",
              "s3:ListBucket"
          ],
          "Resource": [
              "arn:aws:s3:::<bucket>",
              "arn:aws:s3:::<bucket>/*"
          ]
      }
  ]
}

Defaults & Limits (Bedrock Native)

Property	Default	Notes
completion_window	24h	Fixed by Bedrock.
Job quota	50k/day	Subject to your AWS account.
Max input file size	10 GB	Across all .jsonl objects.

Portkey-Batch mode inherits the Gateway defaults (25 req / 5 s) and retries (3×) per request.

---

Glossary

Term	Meaning
Batch Job	Collection of asynchronous completions.
Portkey File (input_file_id)	File uploaded to Portkey and re-uploaded to the provider for batch processing.
Virtual Key	Bedrock credential stored in Portkey; referenced by ID, not secret.
Completion Window	immediate → Portkey-Batch; 24h → Bedrock native.

---

See Also

• Unified Batch Guide