An AI application requires a vector database for context, an orchestrator for workflows, and specialized observability tools to trace non-deterministic behavior. These components scale differently than tokens. Build a cost model that includes vector storage, tracing volume, and orchestration overhead before you launch, and set hard billing alarms for the entire stack.
Tokens Are Only One Dimension
The standard way to calculate the cost of an AI feature is to multiply the expected number of prompt and completion tokens by the provider's price per million tokens. This works for a simple chatbot script, but fails immediately for a production application.
A real AI feature is a distributed system. It retrieves documents from a vector database, chains multiple model calls through an orchestration layer, and generates telemetry for every step of that chain. When traffic spikes, the token bill goes up linearly, but the database read capacity and the observability ingestion costs might hit a step function that forces an expensive tier upgrade.
Many vector databases and AI observability platforms have generous free tiers but steep cliffs. You might pay pennies for tokens, but hundreds of dollars for exceeding your vector index size or tracing retention limit.
The Three Pillars of AI Infrastructure
To build an accurate budget, you need to model the three infrastructure pillars that support the model API.
| Infrastructure Component | What Drives the Cost | How to Model It |
|---|---|---|
| Vector Databases (Pinecone, Weaviate) | Storage (number of vectors/dimensions) and operations (reads/writes per second). | Estimate total documents, multiply by chunks per document, and check the pricing tier for that index size. |
| Observability & Tracing (LangSmith, Braintrust) | Ingestion volume (spans per trace) and data retention duration. | Calculate traces per user session. A single RAG query might generate 10+ spans across retrieval and generation. |
| Orchestration & Compute (Serverless functions) | Execution time. AI calls are slow; keeping a serverless function alive waiting for a stream adds up. | Model the average latency of the AI call, multiply by your cloud provider's gigabyte-second compute cost. |
If you don't model these, your "cheap" intelligence will be dragged down by the expensive plumbing required to use it safely.
Modeling the Vector Database
Vector database pricing usually has two parts that grow independently: what you store and how much you read and write. Pinecone’s serverless indexes, for example, bill storage per gigabyte and reads and writes per unit. The storage line grows with every vector you keep, whether anyone queries it or not, and the read line grows with traffic on top of that.
If you allow users to upload their own documents (a "bring your own data" RAG pattern), your vector count will grow unbounded.
Do not store vectors indefinitely if the user only needed them for a single session. Implement a Time-To-Live (TTL) or a background job to delete vectors associated with stale workspaces, keeping your index size within a predictable pricing tier.
The Cost of Observability
Classic application performance monitoring (APM) was built for deterministic code: it logs errors and latency. AI requires tracing the actual inputs and outputs of every step in a chain to debug hallucinations. This means you are logging enormous strings of text—sometimes tens of thousands of tokens per trace.
Dedicated AI observability platforms charge based on the number of traces or the sheer volume of data ingested. If you log every prompt, every retrieved document, and every completion for 100% of your production traffic, your logging bill will easily dwarf your model API bill.
Sample your traces in production. Log 100% of errors, but only sample 5% of successful requests. Only increase the sampling rate temporarily when debugging a specific incident or rolling out a new prompt version.
Setting Up the Alarms
A budget is only a suggestion unless it is enforced by alarms. Because AI infrastructure is distributed across multiple vendors (the model provider, the vector DB provider, the observability platform, and your cloud compute), a single dashboard won't catch everything.
- Model API: Set a hard billing limit in your provider's console. If you hit it, the API will reject requests, which is better than a bankruptcy event.
- Vector DB: Set an alert on index capacity (e.g., "80% of current tier limit reached").
- Observability: Set an alert on daily ingestion volume. If an infinite loop starts generating traces, you need to know before the month ends.
When you build your next AI feature, don't just ask "what model are we using?" Ask where the data lives, how long the functions wait, and where the traces are stored. That is your real architecture, and that is your real cost.
For a deep dive into the token side of the equation, read AI Cost Modeling: Tokens, Model Selection, and Budget Control. To understand how to monitor these systems once they are live, see AI Observability: Logging, Tracing, and Monitoring AI Features in Production.
Official Sources and Review Trigger
The model in this article is deliberately free of amounts: every figure belongs to your own workload and your own providers. What it does assume is that each pillar is billed the way these pages describe — per million tokens, per stored vector and index hour, and per ingested log volume.
- Anthropic: pricing — the per-million-token structure the first pillar multiplies against, checked Sep 9 2026.
- Pinecone: pricing — an example of a serverless vector database that bills storage per GB separately from read and write units, checked Sep 10 2026.
- Datadog: pricing — an example of ingestion-based observability billing, which is why sampling rates belong in the model, checked Sep 9 2026.
Review this page again by Dec 9 2026, or immediately if a provider changes its billing dimension rather than its rate — a move from per-token to per-request, or from ingested volume to retained volume, is what invalidates the model. A price change alone does not.