Cloudflare AI Gateway: How to Cache, Rate-Limit, and Control Costs on LLM API Calls (2026 Guide)

If your SaaS shipped an LLM feature this year, your OpenAI or Anthropic bill is now a line item finance asks about, and it moves with usage you do not fully control. The reflex is to build a caching proxy and a spend cap in-house. Before you spend a sprint on that, Cloudflare AI Gateway gives you most of it by changing one line in your API base URL.

Cloudflare AI Gateway is a proxy that sits between your application and LLM providers like OpenAI, Anthropic, and Google. You route model calls through it to get caching, rate limiting, retries, spend limits, and per-request logging, without changing your provider SDK. The core features are free and add tens of milliseconds of latency at most.

This guide covers what it does, how caching and rate limits cut real cost, how to wire it up, how it compares to building your own proxy or using tools like LiteLLM and Helicone, and when it is the wrong fit.

What Is Cloudflare AI Gateway?

Cloudflare AI Gateway is a managed request proxy for AI model traffic. Instead of your app calling api.openai.com or api.anthropic.com directly, it calls a gateway endpoint on Cloudflare’s edge, which forwards the request to the provider and returns the response. Because every call passes through one place, you get a control point you did not have before.

That control point does five things worth caring about:

• Caching, so identical or similar requests are served without hitting the provider again.
• Rate limiting, so a runaway loop or an abusive client cannot rack up thousands of calls.
• Retries and fallbacks, so a provider timeout does not surface as a failed feature.
• Spend and request limits, so the gateway stops serving once it crosses a threshold you set.
• Logging and analytics, so you can see cost, latency, and token counts per request rather than reading a monthly invoice.

It supports the major providers, including OpenAI, Anthropic, Google AI Studio, Groq, and Mistral, plus Cloudflare’s own Workers AI. You do not need to host your app on Cloudflare to use it. The gateway is provider-agnostic and works with a standard backend anywhere.

How Does Caching Cut Your LLM Bill?

Caching is where most teams find the fastest savings, because a cache hit costs zero provider tokens and returns in tens of milliseconds instead of the one to several seconds a model call takes. The gateway stores a response keyed on the request, and when the same request arrives again it replays the stored answer.

How much this saves depends entirely on how repetitive your traffic is. A support assistant answering the same product questions, a classification endpoint scoring similar inputs, or a “summarize this document” feature run on the same documents will see high repeat rates. In workloads like these, 20 to 40 percent of requests are often near-duplicates, and caching removes that share of the bill outright.

Be honest about the opposite case. If every prompt carries long, user-specific context, such as a chat that includes the full conversation history, cache hit rates are low and exact-match caching barely helps. Cloudflare also offers similarity-based caching, which matches requests that are close but not identical. That lifts hit rates on paraphrased queries, but it can return an answer that was correct for a slightly different question, so test recall on real traffic before you turn it on for user-facing answers.

How Do Rate Limiting and Spend Caps Protect Your Budget?

The scariest LLM bills do not come from steady usage. They come from a retry loop that fires ten thousand calls overnight, a leaked API key, or a launch that goes further than forecast. Rate limiting and spend caps are the guardrails against that.

Rate limiting lets you cap requests per time window, per gateway. Set it just above your real peak and a runaway process hits the ceiling instead of the provider. Spend and request limits go further: once a gateway crosses the number of requests or the budget you configured, it stops forwarding calls until the window resets. That converts an open-ended risk into a known maximum.

The trade-off is real, so set these deliberately. A cap tuned too tight throttles legitimate users during a genuine spike, which reads to them as an outage. Base the numbers on your actual peak traffic plus headroom, and alert on approach rather than only on breach, so a human can react before real users are turned away.

How Do You Set Up Cloudflare AI Gateway?

Setup is deliberately small. You create a gateway in the Cloudflare dashboard, then point your existing provider SDK at the gateway URL. No SDK swap, no rewrite of your call sites.

The steps:

1. In the Cloudflare dashboard, open AI, then AI Gateway, and create a gateway. Note its account ID and gateway ID.
2. Change the base URL your SDK uses to the gateway endpoint for that provider.
3. Optionally enable caching, set a default TTL, and configure rate or spend limits in the gateway settings.
4. Deploy and watch the analytics tab for cache hit rate, cost, and latency.

Here is the one change that does the routing, using the OpenAI Node SDK as an example:

import OpenAI from "openai";

// Point the existing SDK at the gateway. Nothing else changes.
const client = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
  baseURL:
    "https://gateway.ai.cloudflare.com/v1/{account_id}/{gateway_id}/openai",
});

const completion = await client.chat.completions.create({
  model: "gpt-4o-mini",
  messages: [{ role: "user", content: "Summarize this ticket." }],
});

Caching can be turned on for the whole gateway, or controlled per request with a header, which is useful when some responses are safe to cache and others must stay fresh. You set the time to live in seconds:

const completion = await client.chat.completions.create(
  {
    model: "gpt-4o-mini",
    messages: [{ role: "user", content: "Summarize this ticket." }],
  },
  {
    headers: {
      "cf-aig-cache-ttl": "3600",
    },
  }
);

That is the whole integration for one provider. Anthropic, Google, and the rest follow the same pattern with their own path segment on the gateway URL. If your app already uses an inline baseURL override for staging, this fits the same place in your config.

Cloudflare AI Gateway vs. Building Your Own Proxy

You do not strictly need Cloudflare for any of this. A caching proxy is a known pattern, and tools like LiteLLM and Helicone cover overlapping ground. The right choice depends on how much routing logic you need and whether you want to run infrastructure yourself. Here is how the main options compare.

For most mid-market teams the honest answer is that Cloudflare AI Gateway covers the 80 percent case with the least work: caching, guardrails, and per-request visibility with a one-line change and no servers to run. Reach for LiteLLM when you need real multi-provider routing, model fallback chains, or a unified interface across many providers. Reach for a custom proxy only when you have a rule no product covers and the engineering time to own it. Building your own to save a managed-service fee usually costs more in maintenance than it saves.

When Cloudflare AI Gateway Is the Wrong Choice

No tool is free of trade-offs, and a gateway is another dependency in your request path. Skip it, or think twice, in these cases.

• Your prompts are almost all unique. If caching cannot help and you already have observability, the gateway adds a hop for little gain.
• You have strict data-residency or privacy rules. Request and response logging can capture prompts that contain personal data. Confirm what is logged and where before you enable full logging, and keep sensitive fields out of prompts.
• You need complex routing today. The gateway is not a full router. If your core need is weighted load balancing or fallback across five providers, a purpose-built router fits better.
• You cannot accept another external dependency. Cloudflare’s edge is reliable, but it is still a hop between you and the provider. For a system with hard single-vendor constraints, that may not pass review.

None of these are reasons to avoid it by default. They are reasons to decide on purpose rather than adding it because it is easy.

How Redwerk Controls LLM Costs in Production

Most LLM cost problems are not model problems. They are architecture decisions made before anyone was watching the bill: no caching layer, no spend ceiling, prompts carrying more context than the task needs, and no per-feature visibility into what each call actually costs.

Redwerk fields engineers who have already shipped on Cloudflare’s edge and the major LLM SDKs, so the gateway, a sensible cache-key strategy, and spend guardrails go in during onboarding rather than after a surprise invoice. That is the tech-match advantage: you are not paying a team to learn the stack on your budget. We also keep the projected cost curve visible from the start, so a product owner sees where spend goes before it is committed, not after.

If you are adding AI features and want them cheap to run as well as good, our AI and ML development team builds them with cost control designed in, and our custom web development engineers wire the gateway and guardrails into your existing app. For a deeper look at shipping production AI, see our guides on building LLM semantic search inside your SaaS product and cutting model cost with distillation.

Frequently Asked Questions

See how we trained a neural network on 1.5M+ texts to power relevance-first search inside a recruitment SaaS

Please enter your business email isn′t a business email

Get Case Study