Enable Prompt Caching on amazon bedrock Claude models using Autogen 0.4.7

[Alan Marquina (amarquinar)]

Hello experts,

I realised that Amazon implement Prompt caching on Claude models as they showed on the next blog: https://aws.amazon.com/blogs/machine-learning/effectively-use-prompt-caching-on-amazon-bedrock/

I would like to leverage that feature using Autogen 0.4.7.

Mi model client is configured using the follow code:

`
from autogen_ext.models.semantic_kernel import SKChatCompletionAdapter
from semantic_kernel import Kernel
from semantic_kernel.connectors.ai.bedrock import BedrockChatCompletion, BedrockPromptExecutionSettings
from semantic_kernel.memory.null_memory import NullMemory
import os
from enum import Enum

class LLMModel(str, Enum):
CLAUDE_3_5_SONNET = "eu.anthropic.claude-3-5-sonnet-20240620-v1:0"
CLAUDE_3_HAIKU = "anthropic.claude-3-haiku-20240307-v1:0"
CLAUDE_3_7_SONNET = "us.anthropic.claude-3-7-sonnet-20250219-v1:0"

def get_bedrock_model_client(
model_id=LLMModel.CLAUDE_3_5_SONNET,
region="us-east-1",
temperature=0.0,
price=[0.003, 0.015]
):
"""
model_id: eu.anthropic.claude-3-5-sonnet-20240620-v1:0 / anthropic.claude-3-haiku-20240307-v1:0
"""
os.environ["AWS_DEFAULT_REGION"] = region

sk_client = BedrockChatCompletion(
    model_id=model_id,
)
bedrock_settings = BedrockPromptExecutionSettings(
    options={
        "region": region,
        "temperature": temperature,
        "price": price
    }        
)

bedrock_model_client = SKChatCompletionAdapter(
    sk_client,
    kernel=Kernel(
        memory=NullMemory()
    ),
    prompt_settings=bedrock_settings,
    model_info={
        "vision": False,
        "function_calling": True,
        "json_output": True,
        "family": "unknown",
    },
)

return bedrock_model_client

`

I tried to use "cache_seed" but it doesn't work.

Could you please tell me how can I activate Prompt Caching on AWS bedrock Claude models, using Autogen 0.4.7?

Many thanks in advance.

Regards,,
Alan

[Alan Marquina (amarquinar)]

Hello Eric Zhu (@ekzhu) , could you please help me on that?

Many thanks in advance

[Lucas Kim (kimnamu)]

Thanks for pasting the full client setup and the blog link, that made this easy to reproduce end to end.

Short answer: nothing in that stack exposes prompt caching today, so there is no setting to flip. SK's Bedrock connector serialises the system prompt as {"text": ...} only and forwards just top_k for Claude, and SKChatCompletionAdapter passes settings through untouched. (cache_seed was the 0.2 local response cache; the 0.4 equivalent is ChatCompletionCache, a different feature.)

Because your setup already owns a boto3 client, a botocore event hook can append the cachePoint block without touching autogen or SK, so it works on 0.4.7 as well:

sk_client = BedrockChatCompletion(model_id=MODEL)

def add_cache_point(params, **kwargs):
    if params.get("system"):  # Converse wants a content block before the cache point
        params["system"].append({"cachePoint": {"type": "default"}})

for op in ("Converse", "ConverseStream"):
    sk_client.bedrock_runtime_client.meta.events.register(f"before-parameter-build.bedrock-runtime.{op}", add_cache_point)

# optional: the cache counters only exist in the raw Converse response
sk_client.bedrock_runtime_client.meta.events.register(
    "after-call.bedrock-runtime.Converse", lambda parsed, **kw: print("converse usage:", parsed["usage"]))

client = SKChatCompletionAdapter(sk_client, kernel=Kernel(), prompt_settings=..., model_info=...)

Measured on autogen main @ 027ecf0, semantic-kernel 1.36.0, boto3 1.40, us-east-1, us.anthropic.claude-haiku-4-5-20251001-v1:0, same ~21k-token system prompt each call:

Call	inputTokens	cacheWriteInputTokens	cacheReadInputTokens	autogen result.usage.prompt_tokens
1st create()	12	21710	0	12
2nd create()	12	0	21710	12

converse usage: {'inputTokens': 12, 'outputTokens': 4, 'totalTokens': 21726, 'cacheReadInputTokens': 0, 'cacheWriteInputTokens': 21710}
call 1: autogen usage=RequestUsage(prompt_tokens=12, completion_tokens=4) content='ok'
converse usage: {'inputTokens': 12, 'outputTokens': 4, 'totalTokens': 21726, 'cacheReadInputTokens': 21710, 'cacheWriteInputTokens': 0}
call 2: autogen usage=RequestUsage(prompt_tokens=12, completion_tokens=4) content='ok'

create_stream() goes through ConverseStream; with the second hook registered its request body ends with the same {'cachePoint': {'type': 'default'}} block.

Two things to watch: result.usage.prompt_tokens only reflects the uncached inputTokens, so autogen's total_usage() under-counts once caching is on, and a prompt below the model's minimum cacheable size silently reports cacheWriteInputTokens: 0 (a 1877-token prompt on Haiku 4.5 did, on both calls).

Where this lives, your model IDs, and the native client

Why there is no setting. bedrock_chat_completion.py _prepare_settings_for_request builds system via _prepare_system_messages_for_request, and utils._format_system_message returns {"text": message.content}. bedrock_anthropic_claude.get_chat_completion_additional_model_request_fields only forwards top_k. _sk_chat_completion_adapter.py hands the ChatHistory and settings to SK unchanged. before-parameter-build fires before botocore validates and serialises the call, which is why the appended block reaches the wire.

Your three model IDs. On my account the 3.5 Sonnet and 3.7 Sonnet IDs now return ResourceNotFoundException: This model version has reached the end of its life, and Claude 3 Haiku answers a cachePoint request with AccessDeniedException: You invoked an unsupported model or your request did not allow prompt caching, so I tested with Haiku 4.5. Check Bedrock's prompt caching model list before picking an ID.

AnthropicBedrockChatCompletionClient has the same gap: _anthropic_client.py:622 sets request_args["system"] to a plain string and :667 only forwards top_p/top_k/stop_sequences/metadata from extra_create_args, so a system list with cache_control passed there is dropped. Wrapping the SDK call works:

client = AnthropicBedrockChatCompletionClient(model=MODEL, bedrock_info=BedrockInfo(aws_region="us-east-1"), model_info=...)
_create = client._client.messages.create

async def create_with_cache(**kw):
    if isinstance(kw.get("system"), str):
        kw["system"] = [{"type": "text", "text": kw["system"], "cache_control": {"type": "ephemeral"}}]
    return await _create(**kw)

client._client.messages.create = create_with_cache

anthropic usage: {'cache_creation_input_tokens': 0, 'cache_read_input_tokens': 21710, 'input_tokens': 12, 'output_tokens': 4, ...}

That first call already read the entry written by the SK run above: same model, same prompt, within the TTL.