Anthropic recently introduced a new feature on its API called prompt caching, which aims to enhance user experience by remembering the context between API calls. This feature allows developers to avoid repeating prompts and maintain frequently used contexts in their sessions. While the prompt caching is currently available in public beta on Claude 3.5 Sonnet and Claude 3 Haiku models, the support for the largest Claude model, Opus, is still in the works.
Prompt caching, as described in a 2023 paper by Anthropic, offers several advantages to users. One of the key benefits is the ability to add additional background information without incurring extra costs. This feature becomes particularly useful when users need to send a significant amount of context in a prompt and refer back to it in different conversations with the model. Moreover, prompt caching enables developers and users to fine-tune model responses more effectively.
According to Anthropic, early users have reported substantial speed and cost improvements with prompt caching across various use cases. These include reducing costs and latency for long instructions and uploaded documents for conversational agents, achieving faster autocompletion of codes, providing multiple instructions to agentic search tools, and embedding entire documents in a prompt. One notable advantage of using cached prompts is the lower prices per token, with significant cost savings compared to the base input token price.
For instance, on the Claude 3.5 Sonnet model, writing a prompt to be cached will cost $3.75 per 1 million tokens (MTok), whereas using a cached prompt will only cost $0.30 per MTok. The base price of an input to the Claude 3.5 Sonnet model is $3/MTok, making the use of cached prompts a more cost-effective option in the long run. Similarly, Claude 3 Haiku users will pay $0.30/MTok to cache and $0.03/MTok when accessing stored prompts. However, prompt caching is not yet available for Claude 3 Opus, although pricing details have already been made public. Writing to cache on this model will cost $18.75/MTok, with accessing the cached prompt priced at $1.50/MTok.
It is worth noting that Anthropic has been strategically positioning itself against other AI platforms through competitive pricing. Before the launch of the Claude 3 models, the company had already reduced the prices of its tokens, signaling a competitive stance in the market. With the introduction of prompt caching, Anthropic is now facing a “race to the bottom” against competitors like Google and OpenAI in offering low-priced options for third-party developers leveraging its platform.
While Anthropic’s prompt caching feature brings significant benefits, other AI platforms also offer similar functionalities. For example, Lamina, an LLM inference system, utilizes KV caching to lower the cost of GPUs. Additionally, OpenAI provides a memory feature in its GPT-4o model where preferences or details can be remembered, albeit not in the same way as prompt caching.
Prompt caching on Anthropic’s API represents a significant step towards enhancing user experience, reducing costs, and improving model responses. The feature’s availability on different models and the cost-efficient pricing structure make it a valuable addition for developers and users looking to optimize their interactions with AI models. As the competition in the AI market intensifies, Anthropic’s prompt caching feature could prove to be a compelling option for those seeking cost-effective and efficient AI solutions.
Leave a Reply