> ## Documentation Index > Fetch the complete documentation index at: https://docs.portkey.ai/docs/llms.txt > Use this file to discover all available pages before exploring further. # Thinking Mode Thinking/Reasoning models represent a new generation of LLMs specifically trained to expose their internal reasoning process. Unlike traditional LLMs that only show final outputs, thinking models like Claude 3.7 Sonnet, OpenAI o1/o3, and Deepseek R1 are designed to "think out loud" - producing a detailed chain of thought before delivering their final response. These reasoning-optimized models are built to excel in tasks requiring complex analysis, multi-step problem solving, and structured logical thinking. Portkey makes these advanced models accessible through a unified API specification that works consistently across providers. ## Supported Thinking Models Portkey currently supports thinking-enabled models from **Anthropic**, **Google Vertex AI**, **Amazon Bedrock**, **OpenAI**, **Together AI**, and other **OpenAI-compatible providers**. *Note: If a specific model is not supported for thinking on Portkey, please reach out to [Portkey support](https://support.portkey.ai/forms/customer-portal-ticket-form).* ## Using Thinking Mode 1. You must set `strict_open_ai_compliance=False` in your headers or client configuration 2. The thinking response is returned in a different format than standard completions 3. For streaming responses, the thinking content is in `response_chunk.choices[0].delta.content_blocks` > [!WARNING] > > Extended thinking API through Portkey is currently in beta. ### Basic Example > [tab] Python > > ```python theme={"system"} > from portkey_ai import Portkey > > # Initialize the Portkey client > portkey = Portkey( > api_key="PORTKEY_API_KEY", # Replace with your Portkey API key > provider="@PROVIDER", > strict_open_ai_compliance=False # Required for thinking mode > ) > > # Create the request > response = portkey.chat.completions.create( > model="claude-3-7-sonnet-latest", > max_tokens=3000, > thinking={ > "type": "enabled", > "budget_tokens": 2030 # Maximum tokens to use for thinking > }, > stream=False, > messages=[ > { > "role": "user", > "content": [ > { > "type": "text", > "text": "when does the flight from new york to bengaluru land tomorrow, what time, what is its flight number, and what is its baggage belt?" > } > ] > } > ] > ) > print(response) > > # For streaming responses, handle content_blocks differently > # response = portkey.chat.completions.create( > # ...same config as above but with stream=True > # ) > # for chunk in response: > # if chunk.choices[0].delta: > # content_blocks = chunk.choices[0].delta.get("content_blocks") > # if content_blocks is not None: > # for content_block in content_blocks: > # print(content_block) > ``` > [tab] Node.js > > ```javascript theme={"system"} > import Portkey from 'portkey-ai'; > > // Initialize the Portkey client > const portkey = new Portkey({ > apiKey: "PORTKEY_API_KEY", // Replace with your Portkey API key > provider:"@PROVIDER", > strictOpenAiCompliance: false // Required for thinking mode > }); > > // Generate a chat completion > async function getChatCompletionWithThinking() { > const response = await portkey.chat.completions.create({ > model: "claude-3-7-sonnet-latest", > max_tokens: 3000, > thinking: { > type: "enabled", > budget_tokens: 2030 > }, > stream: false, > messages: [ > { > role: "user", > content: [ > { > type: "text", > text: "when does the flight from new york to bengaluru land tomorrow, what time, what is its flight number, and what is its baggage belt?" > } > ] > } > ] > }); > console.log(response); > > // For streaming responses: > // const response = await portkey.chat.completions.create({ > // ...same config as above but with stream: true > // }); > // for await (const chunk of response) { > // if (chunk.choices[0].delta?.content_blocks) { > // for (const contentBlock of chunk.choices[0].delta.content_blocks) { > // console.log(contentBlock); > // } > // } > // } > } > > // Call the function > getChatCompletionWithThinking(); > ``` > [tab] OpenAI SDK (JS) > > ```javascript theme={"system"} > import OpenAI from 'openai'; // We're using the v4 SDK > import { PORTKEY_GATEWAY_URL, createHeaders } from 'portkey-ai' > > const openai = new OpenAI({ > apiKey: 'ANTHROPIC_API_KEY', // defaults to process.env["OPENAI_API_KEY"], > baseURL: PORTKEY_GATEWAY_URL, > defaultHeaders: createHeaders({ > provider: "anthropic", > apiKey: "PORTKEY_API_KEY", // defaults to process.env["PORTKEY_API_KEY"] > strictOpenAiCompliance: false > }) > }); > > // Generate a chat completion with thinking mode > async function getChatCompletionFunctions(){ > const response = await openai.chat.completions.create({ > model: "claude-3-7-sonnet-latest", > max_tokens: 3000, > thinking: { > type: "enabled", > budget_tokens: 2030 > }, > stream: false, > messages: [ > { > role: "user", > content: [ > { > type: "text", > text: "when does the flight from new york to bengaluru land tomorrow, what time, what is its flight number, and what is its baggage belt?" > } > ] > } > ], > }); > > console.log(response) > } > await getChatCompletionFunctions(); > ``` > [tab] OpenAI SDK (Python) > > ```python theme={"system"} > from openai import OpenAI > from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders > > openai = OpenAI( > api_key='ANTHROPIC_API_KEY', > base_url=PORTKEY_GATEWAY_URL, > default_headers=createHeaders( > provider="anthropic", > api_key="PORTKEY_API_KEY", > strict_open_ai_compliance=False > ) > ) > > response = openai.chat.completions.create( > model="claude-3-7-sonnet-latest", > max_tokens=3000, > thinking={ > "type": "enabled", > "budget_tokens": 2030 > }, > stream=False, > messages=[ > { > "role": "user", > "content": [ > { > "type": "text", > "text": "when does the flight from new york to bengaluru land tomorrow, what time, what is its flight number, and what is its baggage belt?" > } > ] > } > ] > ) > > print(response) > ``` > [tab] cURL > > ```sh theme={"system"} > curl "https://api.portkey.ai/v1/chat/completions" \ > -H "Content-Type: application/json" \ > -H "x-portkey-api-key: $PORTKEY_API_KEY" \ > -H "x-portkey-provider: anthropic" \ > -H "x-api-key: $ANTHROPIC_API_KEY" \ > -H "x-portkey-strict-open-ai-compliance: false" \ > -d '{ > "model": "claude-3-7-sonnet-latest", > "max_tokens": 3000, > "thinking": { > "type": "enabled", > "budget_tokens": 2030 > }, > "stream": false, > "messages": [ > { > "role": "user", > "content": [ > { > "type": "text", > "text": "when does the flight from new york to bengaluru land tomorrow, what time, what is its flight number, and what is its baggage belt?" > } > ] > } > ] > }' > ``` ## Multi-Turn Conversations For multi-turn conversations, include the previous thinking content in the conversation history: ```py Python theme={"system"} from portkey_ai import Portkey # Initialize the Portkey client portkey = Portkey( api_key="PORTKEY_API_KEY", # Replace with your Portkey API key provider="@PROVIDER", strict_open_ai_compliance=False ) # Create the request response = portkey.chat.completions.create( model="claude-3-7-sonnet-latest", max_tokens=3000, thinking={ "type": "enabled", "budget_tokens": 2030 }, stream=False, messages=[ { "role": "user", "content": [ { "type": "text", "text": "when does the flight from baroda to bangalore land tomorrow, what time, what is its flight number, and what is its baggage belt?" } ] }, { "role": "assistant", "content": [ { "type": "thinking", "thinking": "The user is asking several questions about a flight from Baroda (also known as Vadodara) to Bangalore:\n1. When does the flight land tomorrow\n2. What time does it land\n3. What is the flight number\n4. What is the baggage belt number at the arrival airport\n\nTo properly answer these questions, I would need access to airline flight schedules and airport information systems. However, I don't have:\n- Real-time or scheduled flight information\n- Access to airport baggage claim allocation systems\n- Information about specific flights between these cities\n- The ability to look up tomorrow's specific flight schedules\n\nThis question requires current, specific flight information that I don't have access to. Instead of guessing or providing potentially incorrect information, I should explain this limitation and suggest ways the user could find this information.", "signature": "EqoBCkgIARABGAIiQBVA7FBNLRtWarDSy9TAjwtOpcTSYHJ+2GYEoaorq3V+d3eapde04bvEfykD/66xZXjJ5yyqogJ8DEkNMotspRsSDKzuUJ9FKhSNt/3PdxoMaFZuH+1z1aLF8OeQIjCrA1+T2lsErrbgrve6eDWeMvP+1sqVqv/JcIn1jOmuzrPi2tNz5M0oqkOO9txJf7QqEPPw6RG3JLO2h7nV1BMN6wE=" } ] }, { "role": "user", "content": "thanks that's good to know, how about to chennai?" } ] ) print(response) ``` ```ts NodeJS theme={"system"} import Portkey from 'portkey-ai'; // Initialize the Portkey client const portkey = new Portkey({ apiKey: "PORTKEY_API_KEY", provider:"@PROVIDER", strictOpenAiCompliance: false }); // Generate a chat completion async function getChatCompletionFunctions() { const response = await portkey.chat.completions.create({ model: "@anthropic/claude-3-7-sonnet-latest", max_tokens: 3000, thinking: { type: "enabled", budget_tokens: 2030 }, stream: false, messages: [ { role: "user", content: [ { type: "text", text: "when does the flight from baroda to bangalore land tomorrow, what time, what is its flight number, and what is its baggage belt?" } ] }, { role: "assistant", content: [ { type: "thinking", thinking: "The user is asking several questions about a flight from Baroda (also known as Vadodara) to Bangalore:\n1. When does the flight land tomorrow\n2. What time does it land\n3. What is the flight number\n4. What is the baggage belt number at the arrival airport\n\nTo properly answer these questions, I would need access to airline flight schedules and airport information systems. However, I don't have:\n- Real-time or scheduled flight information\n- Access to airport baggage claim allocation systems\n- Information about specific flights between these cities\n- The ability to look up tomorrow's specific flight schedules\n\nThis question requires current, specific flight information that I don't have access to. Instead of guessing or providing potentially incorrect information, I should explain this limitation and suggest ways the user could find this information.", signature: "EqoBCkgIARABGAIiQBVA7FBNLRtWarDSy9TAjwtOpcTSYHJ+2GYEoaorq3V+d3eapde04bvEfykD/66xZXjJ5yyqogJ8DEkNMotspRsSDKzuUJ9FKhSNt/3PdxoMaFZuH+1z1aLF8OeQIjCrA1+T2lsErrbgrve6eDWeMvP+1sqVqv/JcIn1jOmuzrPi2tNz5M0oqkOO9txJf7QqEPPw6RG3JLO2h7nV1BMN6wE=" } ] }, { role: "user", content: "thanks that's good to know, how about to chennai?" } ] }); console.log(response); } // Call the function getChatCompletionFunctions(); ``` ```js OpenAI NodeJS theme={"system"} import OpenAI from 'openai'; // We're using the v4 SDK import { PORTKEY_GATEWAY_URL, createHeaders } from 'portkey-ai' const openai = new OpenAI({ apiKey: 'ANTHROPIC_API_KEY', // defaults to process.env["OPENAI_API_KEY"], baseURL: PORTKEY_GATEWAY_URL, defaultHeaders: createHeaders({ provider: "anthropic", apiKey: "PORTKEY_API_KEY", // defaults to process.env["PORTKEY_API_KEY"] strict_open_ai_compliance: false }) }); // Generate a chat completion with streaming async function getChatCompletionFunctions(){ const response = await openai.chat.completions.create({ model: "@anthropic/claude-3-7-sonnet-latest", max_tokens: 3000, thinking: { type: "enabled", budget_tokens: 2030 }, stream: false, messages: [ { role: "user", content: [ { type: "text", text: "when does the flight from baroda to bangalore land tomorrow, what time, what is its flight number, and what is its baggage belt?" } ] }, { role: "assistant", content: [ { type: "thinking", thinking: "The user is asking several questions about a flight from Baroda (also known as Vadodara) to Bangalore:\n1. When does the flight land tomorrow\n2. What time does it land\n3. What is the flight number\n4. What is the baggage belt number at the arrival airport\n\nTo properly answer these questions, I would need access to airline flight schedules and airport information systems. However, I don't have:\n- Real-time or scheduled flight information\n- Access to airport baggage claim allocation systems\n- Information about specific flights between these cities\n- The ability to look up tomorrow's specific flight schedules\n\nThis question requires current, specific flight information that I don't have access to. Instead of guessing or providing potentially incorrect information, I should explain this limitation and suggest ways the user could find this information.", signature: "EqoBCkgIARABGAIiQBVA7FBNLRtWarDSy9TAjwtOpcTSYHJ+2GYEoaorq3V+d3eapde04bvEfykD/66xZXjJ5yyqogJ8DEkNMotspRsSDKzuUJ9FKhSNt/3PdxoMaFZuH+1z1aLF8OeQIjCrA1+T2lsErrbgrve6eDWeMvP+1sqVqv/JcIn1jOmuzrPi2tNz5M0oqkOO9txJf7QqEPPw6RG3JLO2h7nV1BMN6wE=" } ] }, { role: "user", content: "thanks that's good to know, how about to chennai?" } ], }); console.log(response) } await getChatCompletionFunctions(); ``` ```py OpenAI Python theme={"system"} from openai import OpenAI from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders openai = OpenAI( api_key='PORTKEY_API_KEY', base_url=PORTKEY_GATEWAY_URL, default_headers=createHeaders( strict_open_ai_compliance=False ) ) response = openai.chat.completions.create( model="@anthropic/claude-3-7-sonnet-latest", max_tokens=3000, thinking={ "type": "enabled", "budget_tokens": 2030 }, stream=False, messages=[ { "role": "user", "content": [ { "type": "text", "text": "when does the flight from baroda to bangalore land tomorrow, what time, what is its flight number, and what is its baggage belt?" } ] }, { "role": "assistant", "content": [ { "type": "thinking", "thinking": "The user is asking several questions about a flight from Baroda (also known as Vadodara) to Bangalore:\n1. When does the flight land tomorrow\n2. What time does it land\n3. What is the flight number\n4. What is the baggage belt number at the arrival airport\n\nTo properly answer these questions, I would need access to airline flight schedules and airport information systems. However, I don't have:\n- Real-time or scheduled flight information\n- Access to airport baggage claim allocation systems\n- Information about specific flights between these cities\n- The ability to look up tomorrow's specific flight schedules\n\nThis question requires current, specific flight information that I don't have access to. Instead of guessing or providing potentially incorrect information, I should explain this limitation and suggest ways the user could find this information.", signature: "EqoBCkgIARABGAIiQBVA7FBNLRtWarDSy9TAjwtOpcTSYHJ+2GYEoaorq3V+d3eapde04bvEfykD/66xZXjJ5yyqogJ8DEkNMotspRsSDKzuUJ9FKhSNt/3PdxoMaFZuH+1z1aLF8OeQIjCrA1+T2lsErrbgrve6eDWeMvP+1sqVqv/JcIn1jOmuzrPi2tNz5M0oqkOO9txJf7QqEPPw6RG3JLO2h7nV1BMN6wE=" } ] }, { "role": "user", "content": "thanks that's good to know, how about to chennai?" } ] ) print(response) ``` ```sh cURL theme={"system"} curl "https://api.portkey.ai/v1/chat/completions" \ -H "Content-Type: application/json" \ -H "x-portkey-api-key: $PORTKEY_API_KEY" \ -H "x-portkey-strict-open-ai-compliance: false" \ -d '{ "model": "@anthropic/claude-3-7-sonnet-latest", "max_tokens": 3000, "thinking": { "type": "enabled", "budget_tokens": 2030 }, "stream": false, "messages": [ { "role": "user", "content": [ { "type": "text", "text": "when does the flight from baroda to bangalore land tomorrow, what time, what is its flight number, and what is its baggage belt?" } ] }, { "role": "assistant", "content": [ { "type": "thinking", "thinking": "The user is asking several questions about a flight from Baroda (also known as Vadodara) to Bangalore:\n1. When does the flight land tomorrow\n2. What time does it land\n3. What is the flight number\n4. What is the baggage belt number at the arrival airport\n\nTo properly answer these questions, I would need access to airline flight schedules and airport information systems. However, I don't have:\n- Real-time or scheduled flight information\n- Access to airport baggage claim allocation systems\n- Information about specific flights between these cities\n- The ability to look up tomorrow's specific flight schedules\n\nThis question requires current, specific flight information that I don't have access to. Instead of guessing or providing potentially incorrect information, I should explain this limitation and suggest ways the user could find this information.", "signature": "EqoBCkgIARABGAIiQBVA7FBNLRtWarDSy9TAjwtOpcTSYHJ+2GYEoaorq3V+d3eapde04bvEfykD/66xZXjJ5yyqogJ8DEkNMotspRsSDKzuUJ9FKhSNt/3PdxoMaFZuH+1z1aLF8OeQIjCrA1+T2lsErrbgrve6eDWeMvP+1sqVqv/JcIn1jOmuzrPi2tNz5M0oqkOO9txJf7QqEPPw6RG3JLO2h7nV1BMN6wE=" } ] }, { "role": "user", "content": "thanks that's good to know, how about to chennai?" } ] }' ``` ## Understanding Response Format When using thinking-enabled models, be aware of the special response format: > [!NOTE] > > The assistant's thinking response is returned in the `response_chunk.choices[0].delta.content_blocks` array, not the `response.choices[0].message.content` string. This is especially important for streaming responses, where you'll need to specifically parse and extract the thinking content from the content blocks. ## When to Use Thinking Models Thinking models are particularly valuable in specific use cases: * **Complex problem solving:** Break down multi-step tasks such as math, planning, debugging, or technical analysis where the model needs to reason through intermediate steps before answering. * **High-stakes decision support:** Review contracts, policies, research, or operational runbooks where showing the reasoning path helps users audit the answer and catch missed assumptions. * **Long-context analysis:** Compare multiple documents, synthesize evidence, or trace dependencies across large inputs where the model benefits from allocating tokens to structured reasoning. * **Agentic workflows:** Power agents that need to plan tool calls, evaluate options, recover from errors, or explain why they chose a particular next step. * **Quality-sensitive generation:** Produce answers that need stronger consistency, constraint following, and self-checking, such as code review, architecture recommendations, or data analysis. For simple classification, extraction, summarization, or low-latency chat flows, a standard chat model is usually more cost-effective. Thinking mode adds reasoning tokens, so tune `budget_tokens` based on the complexity of the task and the amount of reasoning you want returned. ## FAQs > [accordion] Can I use thinking mode with any model? > > No, thinking mode is only available on specific reasoning-optimized models. Currently, this includes Claude 3.7 Sonnet and will expand to other models as they become available. > [accordion] Does thinking mode increase token usage? > > Yes, enabling thinking mode will increase your token usage since the model is generating additional content for its reasoning process. The `budget_tokens` parameter lets you control the maximum tokens allocated to thinking. > [accordion] Do I need to handle the response differently for thinking mode? > > Yes, particularly for streaming responses. The thinking content is returned in the `content_blocks` array rather than the standard content field, so you'll need to adapt your response parsing logic. > [accordion] Why do I need to set strict\_open\_ai\_compliance to false? > > The thinking mode response format extends beyond the standard OpenAI completion schema. Setting `strict_open_ai_compliance` to false allows Portkey to return this extended format with the thinking content. *** > [card] [Portkey is now PRISMA AIRS AI Gateway. See it in action.](https://www.paloaltonetworks.in/ai-security/ai-gateway?utm_source=portkey\&utm_medium=referral\&utm_campaign=prisma_airs\&utm_content=docs_nav#contact) > > Contact Us ## Related topics - [Prompt Playground](https://docs.portkey.ai/docs/product/prompt-engineering-studio/prompt-playground.md) - [May](https://docs.portkey.ai/docs/changelog/2025/may.md) - [Together AI](https://docs.portkey.ai/docs/integrations/llms/together-ai.md) - [Messages](https://docs.portkey.ai/docs/product/ai-gateway/messages-api.md) - [Enterprise Gateway](https://docs.portkey.ai/docs/changelog/enterprise.md) This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.