Fine-tuning vs prompting is the decision every CTO building product-grade AI faces when accuracy, cost, and control collide. The right path depends on three numbers you can measure this week: your monthly query volume, your latency budget, and the cost of errors for the feature you plan to ship.

Direct answer: For a feature with under 200k queries/month and a tolerance for 400–900ms end-to-end latency, a retrieval-augmented prompting strategy with a vector store and prompt templates will usually cost 3×–10× less in the first year than a full-model fine-tune and delivers comparable accuracy for many tasks; for features above 1M queries/month or with tight latency (<300ms) and strict correctness requirements, parameter-efficient fine-tuning or a private, hosted model becomes cost-justified. Measure cost per 1,000 queries and error cost per misclassification; if monthly spend on prompting-based inference exceeds the expected amortized fine-tune + hosting, move to fine-tune.

A 5‑engineer product team runs roughly $1.0M–$1.4M/yr fully loaded. That anchors build-vs-train math: you should prefer a SaaS or prompting-first approach when the incremental AI spend is under $200k/yr and there is no regulatory need for a private model. Conversely, a $120k–$600k one-time engineering and infra investment into fine-tuning and hosting becomes attractive when your incremental inference spend or error remediation exceeds that level.

Fine-tuning vs prompting: cost, latency, and control

Cost is where the two strategies separate fastest. A one-time parameter-efficient fine-tune (LoRA-style) on a 7B-13B model hosted via a managed inference provider typically costs $30k–$120k for training plus $1k–$6k/month for dedicated inference capacity. A full-model fine-tune on a 70B+ model commonly runs $200k–$600k for training and $8k–$30k/month to host with predictable latency SLAs.

Prompting with retrieval replaces that upfront cost with variable inference spend. Embeddings and vector search for a RAG pipeline at 100k queries/month often cost $15k–$45k/year when using hosted vector services (Pinecone, Weaviate, Qdrant on managed hosting) plus $12k–$36k/year in model inference depending on model choice. That means in year one the prompting route can cost as little as $27k while fine-tuning starts at $30k and often exceeds $120k.

Latency is the other decisive metric. A fine-tuned model colocated on GPUs can deliver 200–400ms median inference for a single-turn query on a 7B model. RAG pipelines add components: embedding generation (50–200ms), vector search (20–200ms depending on region and index type), and the final generation call (200–700ms). End-to-end RAG latency commonly sits at 600–1,200ms unless you aggressively cache or precompute.

Control and auditability favor fine-tuning. When you need deterministic, legally auditable outputs or want to eliminate the vector-store surface for GDPR-like requirements, a private fine-tuned model removes an entire attack plane. Companies with regulated data often accept a $150k–$400k upfront cost to avoid ongoing third-party data residency complexity.

Accuracy improvements are measurable. We commonly see domain pass-rate lifts of 8–18 percentage points on intent-resolution tasks after a targeted fine-tune versus a prompting baseline with retrieval—e.g., 62% → 75–80% pass rate in controlled evals. That delta matters when an error costs money: at a $10 average remediation cost per failed query and 500k queries/month, a 10% error-rate improvement saves $600k/year.

Pick prompting to get to product-market fit cheaply; pick fine-tuning when scale, latency, or auditability creates a predictable payback window.

When fine-tuning wins (and when it doesn't)

Fine-tuning wins when you cross predictable thresholds: sustained query volume above ~1M/month, latency budget below ~300ms, or an error cost that exceeds the amortized training plus hosting. If your product drives predictable traffic—billing, legal summarization, or automated decisions—finetuning reduces per-query marginal cost and reduces variance in outputs.

Prompting wins during discovery and for broad, low-stakes features. When you are iterating on product-market fit, the ability to change prompts, swap retrieval sources, or add rules without retraining lets you iterate 3×–10× faster. A prompting + RAG pipeline lets engineering teams ship new behavior using data engineering work (indexing documents) rather than model engineering cycles.

Hybrid patterns are common and practical: start with prompting, collect high-value failure cases (error logs, hallucination incidents, user corrections), then selectively fine-tune on a curated dataset of 10k–50k high-signal examples. That pathway compresses time-to-quality and keeps first-year costs under $60k in many implementations.

What this means for your CTO roadmap

You must instrument. Measure three operational KPIs every week: queries/month per feature, median end-to-end latency, and remediation cost-per-error. A feature that reaches 200k–400k queries/month with remediation costs above $6–$12 per error should trigger a project evaluation for fine-tuning and dedicated hosting.

Budget the switch. Expect a pragmatic parameter-efficient fine-tune project to require $30k–$120k upfront (training, data curation, eval infrastructure) and $1k–$6k/month ongoing. Put that number into a 24-month TCO model against your predicted inference spend and error-cost runway; only approve fine-tuning when NPV is positive or when non-financial constraints (compliance, latency) dictate.

Ship for observability and retrainability. Use tools like LangSmith or open telemetry collectors to capture prompts, retrieved context, model outputs, and user feedback. A 10k-example labeled failure corpus is enough to evaluate whether a fine-tune would move the needle materially; if it won't, stop before you spend $100k on training.

3 practical checks before you train

1) Run a 30-day prompting baseline and log every failed session. If failures cost >$300/day or you exceed 10k failures in 30 days, evaluate fine-tuning economics.

2) Calculate amortized training cost: training + infra + SRE overhead divided by projected months of service. If amortized cost per 1,000 queries is lower than your current inference cost per 1,000 queries, training is justified.

3) Confirm governance: if regulations require private-control of the model or you must remove all third-party data flows from production logs, treat prompt-first as a temporary tactic and plan a private-hosted fine-tune within 6–12 months.

Key takeaways:

1. Start with a prompting + RAG baseline to limit first-year spend to $20k–$60k and collect failure telemetry.
2. Move to parameter-efficient fine-tuning when sustained traffic exceeds ~1M queries/month, latency budget is <300ms, or error remediation costs exceed the projected amortized training cost.
3. Use a 10k–50k curated failure dataset to validate that fine-tuning delivers an 8–15pp improvement in domain pass rate before committing to a full training budget.
4. Budget $30k–$120k for pragmatic fine-tunes and $150k–$600k for enterprise-scale full-model retraining and private hosting.
5. Instrument prompts, retrieval, and user corrections; these logs are both your product roadmap and training corpus.

Choosing between fine-tuning and prompting is not a one-time binary. Treat the decision as a staged investment: prompt to learn, measure to decide, train to scale. That shifts the conversation from ideology to cashflow and latency: when you can show the finance team a 12–24 month payback on a $120k fine-tune, you stop debating philosophy and start shipping systems that meet real SLAs.