You built a supervisor agent with eight sub-agents. It runs perfectly in dev. In production it throttles at step 4, loses all context, and restarts from scratch, wasting quota, wasting time, and giving users a blank error screen. Here’s the full breakdown. Image Generated Using AI Eight Agents, One Quota Hit, Everything Gone A few months into building a document processing pipeline on AWS Bedrock, a colleague on the platform team asked me a question that I didn’t have a good answer for. “If Agent 4 fails,” he said, “do Agents 1, 2, and 3 need to run again?” We had a supervisor agent coordinating eight sub-agents in sequence: ingestion, classification, extraction, validation, enrichment, anomaly detection, formatting, and final reporting. In our development environment the whole thing ran cleanly in about 40 seconds. In production, where actual document volume meant Bedrock quota limits were in play, we started seeing the flow die at Agent 4 (validation) with a ThrottlingException . And yes, we were restarting from Agent 1 every single time. That meant: every failure was consuming three agent-worth of tokens just to get back to where we’d broken. Under sustained load, this was making our throttling worse. We were burning quota to recover from quota exhaustion. The failure wasn’t just a Bedrock quota problem. It was an architecture problem that the quota hit exposed. The two are usually connected. This article is about both. The Bedrock-specific mechanics that cause these failures, which are stranger and less documented than you’d expect, and the architectural patterns that prevent a single agent failure from destroying an entire multi-step flow. Understanding the Quota Why You’re Getting Throttled Even Though Your Math Says You Shouldn’t Be This is the part that confused us for about two weeks. We checked our Bedrock service quotas. We were well under the listed limits. We were still getting throttled. The re:Post forums are full of people in the same situation. The issue is that Bedrock enforces two separate and independent quota dimensions simultaneously, RPM (Requests Per Minute) and TPM (Tokens Per Minute) and the TPM accounting works in a way most people don’t expect. When you make maxTokens: 4096 a Bedrock API call and set , Bedrock doesn't wait to see how many tokens your response actually uses. It reserves the full 4096 against your TPM quota upfront, the moment the request is received. If the response comes back using only 800 tokens, the reservation is released, but only after the response completes. While the request is in-flight, those 4096 tokens are locked. Now put eight parallel or near-parallel sub-agents in front of a supervisor, each with maxTokens: 4096 . That's up to 32,768 tokens reserved simultaneously, even if the actual outputs are a fraction of that. Add the 5x multiplier that Bedrock applies to output tokens in TPM calculations (one output token counts as five for quota purposes) and you can hit your TPM ceiling while your CloudWatch usage graphs look completely fine. The 5× output token multiplier, the thing nobody mentionsBedrock counts output tokens at 5× weight in TPM quota calculations. So if your response uses 500 output tokens, that consumes 2,500 TPM quota, not 500. This is not clearly documented in the quota page. You find out when your pipeline throttles at one-fifth of the load you expected it to handle. Analysis Image Generated Using AI So before blaming your retry logic or your agent architecture, check the Service Quotas console, but more importantly watch CloudWatch’s InputTokenCount , OutputTokenCount , and InvocationThrottles metrics at one-minute granularity. The quota console shows configured limits. CloudWatch shows actual burn rate. They're different views of different things. Fix #1: Before anything else: right-size your maxTokens The single highest-leverage change is reducing maxTokens to what your agents actually need. If your extraction agent outputs 300 tokens on average, setting maxTokens: 4096 is reserving 13× more quota than required per call. Set per-agent maxTokens based on observed output distributions, not a safe conservative ceiling. This alone can eliminate most throttling without any architectural changes. The Real Problem What Actually Happens When an Agent Mid-Chain Fails When your supervisor agent in Bedrock invokes a sub-agent and that sub-agent throws a ThrottlingException, the error bubbles up to the supervisor. The supervisor, unless you've explicitly handled this, has two bad options: fail the entire session and return an error to the caller, or retry the failed sub-agent (consuming more of the quota that just ran out). Neither option preserves the work that Agents 1, 2, and 3 already completed. Here’s what a typical 8-agent sequential pipeline looks like from a failure perspective: Agent Workflow Image Generated Using AI The frustrating part is that Agents 1, 2, and 3 did real, valid, expensive work. The ingestion and extraction steps are often the most token-heavy. When you restart the whole flow, you pay for them again. Under quota pressure, this creates a compounding problem: you’re burning tokens to recover from a token exhaustion error. Fix 1 Checkpointing Checkpoint After Every Agent Step, Resume From Where You Broke The architecture fix for this is checkpointing: after each sub-agent completes successfully, persist its output before calling the next agent. When a failure occurs, you know exactly which step failed and you have all prior outputs saved. On retry, the supervisor skips the completed steps and resumes from the failed one. AWS gives you two natural places to checkpoint inside a Bedrock Agents workflow: DynamoDB for fast key-value state, and S3 for larger agent outputs. The supervisor reads the checkpoint table at the start of each session and decides which step to begin from. AWS Console / CDK — DynamoDB checkpoint table structure // DynamoDB table: AgentWorkflowCheckpoints // Partition key: workflowId (String) // Sort key: stepName (String) // Sample item written after Agent 3 (Extraction) completes: { "workflowId": "workflow-2026-04-19-abc123", "stepName": "extraction", "status": "COMPLETE", "outputS3Key": "checkpoints/workflow-abc123/extraction-output.json", "completedAt": "2026-04-19T14:32:11Z", "tokensBurned": 1247, "ttl": 1713657600 // 24h expiry — clean up stale checkpoints } // When Agent 4 fails — this item is written before the error propagates: { "workflowId": "workflow-2026-04-19-abc123", "stepName": "validation", "status": "FAILED", "errorType": "ThrottlingException", "failedAt": "2026-04-19T14:32:54Z", "retryCount": 1 } The supervisor Lambda reads this table at invocation time and finds the last COMPLETE step. If extraction is complete but validation failed, it loads the extraction output from S3 and calls only the validation agent, passing the prior output as context without re-running ingestion, classification, or extraction. AWS Lambda (Python), supervisor checkpoint-aware orchestration import boto3, json, time from botocore.exceptions import ClientError dynamodb = boto3.resource('dynamodb') s3 = boto3.client('s3') bedrock = boto3.client('bedrock-agent-runtime') CHECKPOINT_TABLE = 'AgentWorkflowCheckpoints' BUCKET = 'my-agent-checkpoints' # Ordered pipeline — supervisor works through this list PIPELINE = [ 'ingestion', 'classification', 'extraction', 'validation', 'enrichment', 'anomaly_detection', 'formatting', 'reporting' ] AGENT_IDS = { 'ingestion': 'AGENT_ID_1', 'classification': 'AGENT_ID_2', 'extraction': 'AGENT_ID_3', 'validation': 'AGENT_ID_4', 'enrichment': 'AGENT_ID_5', 'anomaly_detection':'AGENT_ID_6', 'formatting': 'AGENT_ID_7', 'reporting': 'AGENT_ID_8', } def get_checkpoint(workflow_id, step): table = dynamodb.Table(CHECKPOINT_TABLE) resp = table.get_item(Key={'workflowId': workflow_id, 'stepName': step}) return resp.get('Item') def save_checkpoint(workflow_id, step, status, s3_key=None, error=None): table = dynamodb.Table(CHECKPOINT_TABLE) item = { 'workflowId': workflow_id, 'stepName': step, 'status': status, 'updatedAt': str(time.time()), 'ttl': int(time.time()) + 86400 } if s3_key: item['outputS3Key'] = s3_key if error: item['errorType'] = error table.put_item(Item=item) def load_step_output(s3_key): obj = s3.get_object(Bucket=BUCKET, Key=s3_key) return obj['Body'].read().decode() def invoke_agent_step(step, agent_id, input_text, workflow_id, session_id): try: resp = bedrock.invoke_agent( agentId=agent_id, agentAliasId='TSTALIASID', sessionId=session_id, inputText=input_text ) # Stream and collect response output = "" for event in resp['completion']: if 'chunk' in event: output += event['chunk']['bytes'].decode() # Save output to S3 then checkpoint s3_key = f"checkpoints/{workflow_id}/{step}-output.json" s3.put_object(Bucket=BUCKET, Key=s3_key, Body=output.encode()) save_checkpoint(workflow_id, step, 'COMPLETE', s3_key=s3_key) return output except ClientError as e: err_code = e.response['Error']['Code'] save_checkpoint(workflow_id, step, 'FAILED', error=err_code) raise # let supervisor handle the failure def run_pipeline(workflow_id, initial_input): session_id = f"session-{workflow_id}" prior_output = initial_input for step in PIPELINE: # Check: did this step already complete in a prior run? checkpoint = get_checkpoint(workflow_id, step) if checkpoint and checkpoint['status'] == 'COMPLETE': print(f"[{step}] Already complete — loading saved output") prior_output = load_step_output(checkpoint['outputS3Key']) continue # skip — don't re-invoke this agent print(f"[{step}] Invoking agent...") prior_output = invoke_agent_step( step, AGENT_IDS[step], prior_output, workflow_id, session_id ) return prior_output def lambda_handler(event, context): workflow_id = event.get('workflowId') initial_input = event.get('inputText') return run_pipeline(workflow_id, initial_input) The key lines are in the for step in PIPELINE loop. If the checkpoint table shows COMPLETE for a step, the supervisor loads the saved output and calls continue, the agent is never invoked again. When we retry a failed workflow, only the failed step and everything after it runs. Steps 1–3 never touch Bedrock again. On session IDs across retriesBedrock Agents use session IDs to maintain conversation context. When retrying a failed workflow, use the same sessionId as the original run. This preserves any memory or context that prior agents established in the session. If you generate a new session ID on retry, agents 5–8 won't have the conversation history from agents 1–3 and may behave incorrectly. Fix 2 Throttle Handling When the Throttle Hits, Don’t Make It Worse Checkpointing stops you from wasting quota on re-running completed steps. But you still need to handle the actual throttle exception at the point it occurs. This is where most teams make a second mistake: they retry immediately, or they retry uniformly, and end up amplifying the problem. When multiple instances of your pipeline are running (say, 10 concurrent workflows) and all of them hit a throttle at Agent 4 at the same time, they’ll all retry at the same time if your backoff is uniform. You get a “thundering herd”, a synchronized burst of retries that hits the quota ceiling again immediately. The fix is jitter: randomise the backoff delay so your retries spread across time. AWS explicitly recommends “full jitter”, pick a random delay between zero and the calculated backoff ceiling, rather than a fixed exponential value. Python: exponential backoff with full jitter for Bedrock throttles import time, random from botocore.exceptions import ClientError # Errors that are safe to retry RETRYABLE = {'ThrottlingException', 'ServiceUnavailableException', 'ModelTimeoutException', 'InternalServerException'} # Errors that will always fail — don't waste quota retrying these TERMINAL = {'ValidationException', 'AccessDeniedException', 'ResourceNotFoundException'} def invoke_with_backoff(fn, max_attempts=5, base_delay=1, cap=30): """ Retries fn() on retryable Bedrock errors. Uses full jitter: delay = random(0, min(cap, base * 2^attempt)) Raises immediately on terminal errors — no point retrying those. """ attempt = 0 while attempt < max_attempts: try: return fn() except ClientError as e: code = e.response['Error']['Code'] if code in TERMINAL: print(f"Terminal error [{code}] — not retrying") raise if code not in RETRYABLE: raise # unknown — don't swallow it attempt += 1 if attempt >= max_attempts: print(f"Max attempts reached for [{code}]") raise # Full jitter — random sleep between 0 and exponential ceiling ceiling = min(cap, base * (2 ** attempt)) sleep = random.uniform(0, ceiling) print(f"[{code}] attempt {attempt}/{max_attempts} — sleeping {sleep:.1f}s") time.sleep(sleep) # Usage inside invoke_agent_step: output = invoke_with_backoff( lambda: bedrock.invoke_agent(...), max_attempts=4, base_delay=2, cap=20 ) One thing worth being deliberate about: set max_attempts conservatively for agents mid-chain. In a pipeline where Agent 4 fails, you don't want to retry four times before giving up, that's four more Bedrock calls consuming quota while the quota is already exhausted. Two retries with increasing jitter is often the right call. Save the deeper retry budget for SQS-level redriving, which I'll cover next. Fix 3 SQS Queuing SQS as a Buffer, Stop Hammering Bedrock Directly The deeper structural fix for sustained quota pressure is not retry logic, it’s not hitting Bedrock directly from your caller at all. Putting SQS in front of your pipeline means incoming work queues up instead of hitting Bedrock simultaneously. Your Lambda consumers pull from the queue at a rate you control. This decouples user-facing request acceptance from actual Bedrock capacity. Users submit a document, get a job ID, and check status later. Meanwhile, your Lambda consumers process the queue at whatever rate doesn’t trigger throttling, say, 5 workflows per minute instead of 50. AWS CDK (Python): SQS pipeline with DLQ for failed workflows from aws_cdk import ( aws_sqs as sqs, aws_lambda as lambda_, aws_lambda_event_sources as eventsources, Duration ) # Dead-letter queue — workflows that exhausted all retries land here dlq = sqs.Queue(self, "AgentWorkflowDLQ", retention_period=Duration.days(14) ) # Main processing queue workflow_queue = sqs.Queue(self, "AgentWorkflowQueue", visibility_timeout=Duration.seconds(300), # > max pipeline execution time dead_letter_queue=sqs.DeadLetterQueue( max_receive_count=3, # retry 3 times before DLQ queue=dlq ) ) # Lambda processes queue — concurrency capped to control Bedrock call rate pipeline_fn = lambda_.Function(self, "AgentPipelineFn", runtime=lambda_.Runtime.PYTHON_3_12, handler="supervisor.lambda_handler", code=lambda_.Code.from_asset("lambda"), reserved_concurrent_executions=5, # hard cap — prevents quota storms timeout=Duration.seconds(290) ) # Wire queue to Lambda — batch size 1, no partial batch failures pipeline_fn.add_event_source(eventsources.SqsEventSource( workflow_queue, batch_size=1, report_batch_item_failures=True )) The reserved_concurrent_executions=5 is the number to tune. It's a hard ceiling on how many concurrent Bedrock pipelines you'll run at once. Too low and your throughput suffers. Too high and you're back to quota storms. Start conservative, watch your CloudWatch InvocationThrottles metric, and increase gradually. When a workflow hits the DLQ after three attempts, you have a persistent record of everything that failed, workflow ID, step, error type. When the throttling pressure eases (after quota resets or after a quota increase is approved), you can redrive from the DLQ and the checkpoint table picks up exactly where the failure happened. Fix 4 Cross-Region Cross-Region Inference Profiles: More Quota, Less Architecture All three fixes above are things you build yourself. This one is something Bedrock gives you. Cross-Region Inference Profiles are global model IDs that route your requests across multiple AWS regions automatically, pooling quota across those regions. Instead of a model ID like anthropic.claude-sonnet-4–20250514-v1:0 (us-east-1 only), you use the global variant and Bedrock routes to whichever region has capacity. Python, switching to cross-region inference profile # ❌ Single-region — exhausts one region's quota model_id = "anthropic.claude-sonnet-4-20250514-v1:0" # ✅ Global profile — Bedrock routes across regions automatically # Quota is pooled — effectively higher throughput model_id = "global.anthropic.claude-sonnet-4-20250514-v1:0" resp = bedrock_runtime.converse( modelId=model_id, messages=[{'role': 'user', 'content': [{'text': prompt}]}] ) The global profile is not a magic quota multiplier, you still need to request your cross-region quota allocation in the Service Quotas console. But that pooled quota is typically much higher than any single region’s default. For a pipeline with eight agents running sequentially, the reduced burst pressure on any one region makes throttling significantly less frequent. Cross-region quota is separate, request it explicitly Global inference profiles use a different quota entry from per-region quotas. Go to Service Quotas → Amazon Bedrock → find “Cross-region model inference tokens per minute” for your model. The default may be zero or very low. Request an increase before you need it, not after your pipeline starts throttling. What This Looks Like in Practice and What Still Hurts The checkpoint + SQS + backoff combination works. After we implemented it, the “Agent 4 fails, everything restarts” problem went away entirely. Workflows that fail now resume from the exact step they broke at. The DLQ gives us visibility into what’s failing and why. Quota pressure is real but manageable. But there are things this doesn’t fix, and I want to be upfront about them. Bedrock Agents sessions don’t persist indefinitely. The session that carries context across your eight agents has a maximum duration. If your pipeline takes longer than that or if a retry happens significantly after the original failure, the session context is gone and some agents may behave differently because they’re starting fresh. Design agents to be as stateless as possible and pass explicit outputs as input context rather than relying on session memory for critical information. The CloudWatch quota metrics are delayed. The Service Quotas console shows configured limits, not real-time consumption. CloudWatch shows actual burn rate but at one-minute granularity. When you’re debugging a burst throttle that happened over fifteen seconds, the CloudWatch data is often too coarse to tell you exactly which agent triggered it. We ended up adding explicit token count logging inside each agent Lambda to get per-call visibility. Provisioned Throughput is the real answer for sustained load. All of the above is sensible engineering for on-demand quota. But if your pipeline genuinely needs to run at high volume consistently, purchase Provisioned Throughput for the models your agents use. It removes the on-demand quota ceiling entirely for the provisioned capacity and makes your cost predictable. The on-demand approach is appropriate for variable workloads. For sustained production load, it’s fighting the architecture. Backoff, checkpointing, and SQS queuing will handle the failures gracefully. Provisioned Throughput will stop most of them from happening in the first place. Know which problem you’re solving. Bedrock throttles on two independent dimensions, RPM and TPM and TPM burns faster than you think because output tokens count 5× and maxTokens is reserved upfront. Right-size maxTokens per agent before anything else. Checkpoint step outputs to DynamoDB + S3 so failures at Agent 4 don't require re-running Agents 1–3. Use full-jitter exponential backoff on retryable errors and never retry terminal ones. Put SQS in front of your pipeline and cap Lambda concurrency so you're not hitting Bedrock from 50 concurrent callers at once. Switch to global cross-region inference profiles and request the quota increase explicitly, it doesn't happen automatically. For sustained high load, evaluate Provisioned Throughput. The architecture that survives production is the one where a single agent failure is a non-event-logged, checkpointed, retried from exactly that step, and invisible to the user except for slightly longer processing time. If you’ve hit a different failure mode in a Bedrock multi-agent flow, especially around session context loss, concurrent sub-agent quota collisions, or DLQ redrive strategies, share it in the comments. The real edge cases are always more interesting than the happy path. A message from our Founder Hey, Sunil here. I wanted to take a moment to thank you for reading until the end and for being a part of this community. Did you know that our team run these publications as a volunteer effort to over 3.5m monthly readers? We don’t receive any funding, we do this to support the community. If you want to show some love, please take a moment to follow me on LinkedIn , TikTok , Instagram . You can also subscribe to our weekly newsletter . And before you go, don’t forget to clap and follow the writer️! AWS Bedrock Agents Keep Crashing Mid-Flow, Here’s Why and How to Actually Fix It was originally published in Artificial Intelligence in Plain English on Medium, where people are continuing the conversation by highlighting and responding to this story.