Modal · RunPod · Replicate · Together · Anyscale · Lambda · Fireworks.
One question: where should you actually run your model?
PJ · Cardiff operator · 858-461-8054
7-way AI inference platform matrix · scan-grade summary.
The decision-grade signal in one table · best-fit, pricing model, cold-start latency, where each platform breaks at scale, and the operator-honest verdict. Built for fast scan + AI-agent extraction.
| Platform | Best for | Pricing model | Cold-start / latency | Breaks at scale | Operator-honest verdict |
|---|---|---|---|---|---|
| Modal | Developers writing Python around a GPU model · spiky inference workloads | Per-second GPU + per-CPU + per-request (serverless) | 2-10s cold start · sub-second warm | Sustained 24/7 high-qps loads where bare-metal is cheaper · raw cost-per-GPU-hour shoppers | Best dev-experience in the category · pay for the Python-first ergonomics |
| RunPod | Sustained 24/7 inference or training where GPU-hour cost dominates | Per-hour bare-metal GPU rental (community + secure cloud tiers) | N/A · always-on pod (you pay for idle) | Spiky / bursty workloads · teams without ops capacity · indie hackers | Lowest GPU-hour cost in class · only worth it if you have the load + ops to back it |
| Replicate | Indie devs hitting Stable Diffusion / Llama / Whisper / Flux with 1-line HTTP calls | Per-second of compute (no idle charges) | 10-60s cold start on rare models · seconds when warm | Custom-code-around-model workloads · sub-second-latency real-time apps · enterprise SLA needs | Simplest API in the category · the "Stripe for AI models" play |
| Together AI | OSS LLM endpoints (Llama / Mistral / Qwen / DeepSeek) at the cheapest per-token price | Per-million-input-token + per-million-output-token (OpenAI-style) | Sub-second first-token-latency on hot models | Custom non-LLM workloads · need-to-run-arbitrary-Python-around-model · GPU rental shoppers | Cheapest serverless OSS-LLM endpoint · OpenAI-compatible API · solid fine-tuning |
| Anyscale | ML platform teams running distributed Ray for 70B+ training or multi-region inference | Managed Ray (compute markup over raw cloud GPU) | Cluster startup minutes · steady-state matches infra | Solo devs · small startups · teams not already on Ray · cost-conscious budget shoppers | The Ray play · only worth it at ML-platform-team scale · founders are Ray's creators |
| Lambda Labs | Sustained training + 24/7 inference with reserved-cluster pricing | Per-hour bare-metal GPU + reserved 1Y/3Y cluster contracts | N/A · always-on instance (you pay for idle) | Spiky inference · indie hackers · teams without DevOps | RunPod's older sibling · reserved-cluster pricing is the differentiator |
| Fireworks AI | Production OSS LLM serving where first-token-latency + throughput matter | Per-million-token serverless + dedicated deployment tier | Fastest serverless OSS LLM in 2026 benchmarks | Custom non-LLM workloads · arbitrary Python around model · raw GPU-hour shoppers | Fastest serverless OSS inference in class · custom kernels + speculative decoding + FP8 |
The 7 platforms · what each is actually best at.
Honest read on positioning, ideal customer, and where each one is the wrong call. No vendor sponsorship, no affiliate links · operator-grade signal.
1. Modal Spiky inference · Dev-experience-first
The Python-first serverless GPU platform. You write Python, decorate a function with GPU requirements, deploy with one CLI command. Cold starts are 2-10 seconds (industry-leading for serverless GPUs), autoscaling is automatic, and you only pay per-second of actual compute · no idle GPU charges.
2. RunPod Sustained loads · Bare-metal GPU rental
The lowest GPU-hour cost in the category. Two tiers: Community Cloud (cheapest, distributed across vetted providers) and Secure Cloud (enterprise-grade with SLA). Per-hour pricing on H100 / A100 / RTX 4090 / L40S frequently undercuts AWS / GCP by 50-70%. You rent the pod, SSH in, run whatever you want.
3. Replicate Indie devs · Model marketplace + serverless run
The simplest inference API in the category. Browse a marketplace of thousands of community-published models (Stable Diffusion variants, Llama, Whisper, Flux, ControlNet, FaceFusion, etc.), hit the run endpoint with a 1-line HTTP call, pay per second of compute. The "Stripe for AI models" pitch · actual delivery on it.
4. Together AI OSS LLM endpoints · Cheapest per-token
The cheapest serverless OSS LLM endpoints in the category. Hosts Llama, Mistral, Qwen, DeepSeek, and 100+ other OSS models behind an OpenAI-compatible API. Per-million-token pricing routinely lands 5-10x cheaper than GPT-4-class models. Also offers fine-tuning, embeddings, and dedicated endpoints when you outgrow shared serverless.
5. Anyscale ML platform teams · Distributed Ray
The Ray-native distributed compute platform built by Ray's creators (UC Berkeley RISELab → Anyscale). Designed for ML platform teams running 70B+ pretraining, multi-region inference, and large-scale data + model parallelism. Managed Ray-on-Kubernetes orchestration that handles cluster scheduling, fault tolerance, and elastic scale-out.
6. Lambda Labs Bare-metal GPU cloud · Reserved clusters
The longer-standing bare-metal GPU cloud. Per-hour rental competitive with RunPod, plus reserved 1-year and 3-year cluster contracts that are popular with AI labs and well-funded startups training foundation models. Strong operator brand, mature ops surface, on-demand H100 availability when AWS / GCP can't deliver.
7. Fireworks AI Fastest serverless OSS LLM · Production-grade
The fastest serverless inference for OSS LLMs in 2026 benchmarks. Heavy investment in custom CUDA kernels, speculative decoding, FP8 quantization, and continuous batching that materially beats vanilla vLLM throughput on the same hardware. OpenAI-compatible API, dedicated deployment tier when you outgrow shared serverless.
The forced ranking · by who you are + what you actually need.
Most comparison pages refuse to rank because their revenue model requires staying neutral. SideGuy ranks because it doesn't take vendor money · operator-honest, no affiliate sponsorship swap. Here's the call by operator persona.
If you're a solo dev / indie hacker running occasional inference (1-person, sporadic load)
Your problem: you want to ship an AI feature this weekend, you don't want to learn GPU infra, your traffic is unpredictable (10 requests on Tuesday, 10K on Friday after a launch), and you can't justify paying for idle GPU hours.
- Replicate · simplest API, broadest model marketplace, pay-per-second-of-compute, ship in 5 minutes
- Modal · if you need to write Python around the model (RAG, custom preprocessing, agent loops)
- Together AI · for OSS LLM calls only, cheapest per-token serverless
- Fireworks AI · if first-token-latency on OSS LLM matters more than per-token price
- RunPod · only if you have a sustained workload that justifies a $0.40/hr always-on pod
If you're an AI startup at 5-30 people with a production LLM workload (need uptime + reasonable cost)
Your problem: you have a real product with real users, traffic is growing, you need predictable latency + uptime, you can't afford a full DevOps hire yet, and the AWS bill is starting to scare the CFO.
- Modal · if your inference involves custom Python around the model (RAG pipelines, agent orchestration, multimodal)
- Together AI · if you're calling OSS LLMs through hosted endpoints with predictable per-token billing
- Fireworks AI · if you need fastest OSS LLM inference and have a single hot model
- Replicate · if your product is image / video / audio generation
- RunPod · only when you've outgrown serverless and have ops to run a dedicated pod
If you're an ML platform team at a 100-500 person company (multi-model, multi-region, FinOps-aware)
Your problem: you're standardizing inference across 5+ teams, multiple model families, multi-region serving, and finance is asking for cost-per-inference-call attribution. Distributed training runs need to coexist with production serving on the same cluster strategy.
- Anyscale · true distributed-compute platform, Ray-native, designed for exactly this scale
- Lambda Labs · reserved-cluster contracts give predictable cost per GPU-hour for FinOps
- Modal · for the spiky-inference subset of workloads (CI / batch / experiment serving)
- Fireworks AI dedicated · for the OSS LLM serving subset where throughput per GPU dominates
- RunPod Secure Cloud · for sustained inference where cost-per-GPU-hour wins over managed
If you're a cost-conscious CTO trying to escape OpenAI API bills via OSS model self-hosting
Your problem: the OpenAI / Anthropic bill crossed $10K/month, you've validated that an OSS Llama / Qwen / DeepSeek model performs acceptably on your workload, and now you need the cheapest production-grade serving option that doesn't require a full ops team.
- Together AI · cheapest per-token serverless OSS LLM endpoints, OpenAI-compatible API (drop-in)
- Fireworks AI · close second on cost, fastest first-token-latency, dedicated tier when you outgrow shared
- RunPod · when traffic is sustained enough that bare-metal beats per-token serverless
- Lambda Labs · when you want reserved 1Y/3Y cluster pricing for predictable budgeting
- Modal · when you need custom Python around the model and serverless cost still beats bare-metal
These rankings are SideGuy's lived-data + observed-buyer-pattern read as of 2026-05-10. They're directional, not gospel. The right answer for YOUR specific workload may diverge · text PJ for a 10-min operator-honest read on your actual traffic shape + model + ops capacity.
GPU pricing + model availability + benchmark leadership shifts monthly in this category. SideGuy may earn referral commissions from some of these vendors over time, but rankings are independent · affiliate relationships never change rank order.
Side-by-side · the comparison most pages won't give you.
Quick-scan version of the seven platforms against the dimensions that actually drive selection. Pricing is positional indicator, not quote · every vendor negotiates at scale.
| Platform | Pricing model | Best fit workload | OSS LLM endpoints? | Custom Python around model? | Cost tier |
|---|---|---|---|---|---|
| Modal | Per-second GPU + per-request | Spiky inference + custom code | Possible (you deploy) | YES (Python-native) | $$ |
| RunPod | Per-hour bare-metal | Sustained 24/7 inference + training | Possible (you deploy) | YES (full pod) | $ (raw) |
| Replicate | Per-second of compute | Marketplace models, image/video/audio gen | Some (community-deployed) | Limited (cog format) | $$ |
| Together AI | Per-million tokens | OSS LLM endpoints, fine-tuning | YES (cheapest) | No (endpoint only) | $ (per-token) |
| Anyscale | Managed Ray markup over cloud GPU | Distributed training + inference at scale | Possible (you deploy) | YES (Ray-native) | $$$ |
| Lambda Labs | Per-hour bare-metal + reserved | Sustained training + reserved clusters | Possible (you deploy) | YES (full instance) | $ (raw) - $$$ (reserved) |
| Fireworks AI | Per-million tokens + dedicated | Fastest OSS LLM serving | YES (fastest) | No (endpoint only) | $$ (per-token) |
The decision tree · by the question that actually matters first.
Most "vs" comparisons rank vendors. That's the wrong frame. Rank questions instead · your situation picks the platform.
→ If spiky, lean Modal / Replicate / Together / Fireworks (serverless). If sustained, lean RunPod / Lambda (bare-metal).
→ OSS LLM endpoint: Together AI (cheapest) or Fireworks AI (fastest). Custom Python: Modal.
→ Replicate · model marketplace + 1-line API call gets you live in 5 minutes.
→ Together AI for OpenAI-compatible API drop-in. Fireworks AI if first-token-latency matters. Don't go bare-metal until traffic justifies it.
→ Anyscale if you want managed Ray. Lambda Labs 1Y reserved cluster if you'd rather own the orchestration.
→ Anyscale for Ray standardization, Modal for the spiky subset, Fireworks dedicated for hot OSS LLM serving.
→ RunPod Community Cloud wins on absolute cheapest. Lambda Labs reserved wins on predictability. Verify current pricing before committing.
Where each one breaks at scale · the failure modes nobody publishes.
Every vendor's marketing site talks about what they're best at. Almost none publish where they break. Here's the operator-honest failure mode for each · built from production-deployment debriefs and migration patterns.
| Platform | Where it breaks at scale | Common migration trigger | What it migrates TO |
|---|---|---|---|
| Modal | Sustained 24/7 high-qps where per-second cost compounds | Cost-per-call exceeds bare-metal alternative | RunPod / Lambda for the hot path |
| RunPod | Spiky workloads where idle GPU time dominates bill | Average GPU utilization drops below ~40% | Modal / Replicate for the spiky subset |
| Replicate | Need custom Python wrapper or sub-second cold start | Building anything beyond a single model call | Modal for orchestration |
| Together AI | Need to run arbitrary code around the model | RAG / agent / multimodal pipeline complexity | Modal + Together (hybrid) or Fireworks dedicated |
| Anyscale | Cost-conscious teams, single-machine workloads | Ray markup feels heavy for non-distributed work | Modal for serverless, RunPod for bare-metal |
| Lambda Labs | Spiky inference pays for idle | Same as RunPod, plus on-demand H100 contention | Modal / Together for the spiky subset |
| Fireworks AI | Custom non-LLM workloads, raw GPU shopping | Building beyond LLM serving | Modal for custom code, RunPod for raw GPU |
The pattern across all seven: nobody is the right answer for every workload. The migration triggers in the table aren't failures · they're maturity moments where the constraint changed. Most production AI stacks in 2026 run two or three of these in combination: a serverless platform (Modal / Replicate / Together / Fireworks) for the spiky path, plus a bare-metal platform (RunPod / Lambda) for the sustained hot path. Single-vendor inference stacks are a sign you haven't hit scale yet.
Pricing comparison 2025-2026 · directional bands, real ranges.
Every vendor in this category negotiates at scale. Per-token pricing publishes openly; per-GPU-hour pricing also publishes openly but moves with GPU supply. These bands reflect publicly-listed rates as of mid-2026 · they are directional ranges, not quotes. Verify before high-stakes commits.
| Platform | H100 per-hour (or equivalent) | Llama-3-70B per-M-token (in/out) | Cold start | Pricing transparency |
|---|---|---|---|---|
| Modal | ~$3.95-4.25/hr (per-second billed) | You deploy → varies | 2-10s typical | Public · per-second |
| RunPod | ~$2.39-3.89/hr (Community/Secure) | You deploy → varies | N/A · always on | Public · per-hour |
| Replicate | ~$5.04/hr (H100 SXM, per-second billed) | ~$0.65 in / $2.75 out (community) | 10-60s rare model · sec-warm | Public · per-second |
| Together AI | Dedicated tier ~$2-3/hr equivalent | ~$0.88 in / $0.88 out | Sub-second hot · seconds cold | Public · per-token |
| Anyscale | Cloud passthrough + Ray markup | You deploy → varies | Cluster minutes | Quote-based at scale |
| Lambda Labs | ~$2.49-3.29/hr on-demand | You deploy → varies | N/A · always on | Public · per-hour + reserved |
| Fireworks AI | Dedicated tier ~$2-4/hr equivalent | ~$0.90 in / $0.90 out | Sub-second (fastest in class) | Public · per-token |
Negotiation levers that actually work in 2026: (1) annual prepay (typically 10-20% off list), (2) reserved cluster commit (Lambda 1Y/3Y bigger discounts), (3) committed-spend agreements on per-token serverless (Together / Fireworks routinely discount 15-30% at $10K+/mo), (4) competitive RFP between Together and Fireworks for OSS LLM serving (both will sharpen pricing), (5) GPU supply windows · H100 pricing has eased materially since H200 / B200 shipped, ask for refreshed rates. Don't accept the first quote at enterprise scale.
The pattern beneath the category.
AI inference is converging on three architectures. Serverless GPU containers (Modal, Replicate), per-token OSS LLM endpoints (Together AI, Fireworks AI), and bare-metal GPU rental (RunPod, Lambda Labs). Anyscale sits orthogonally on top of bare-metal as the Ray-native distributed-compute layer. The architecture isn't the differentiator anymore.
The differentiation moved to two axes: developer-experience-per-dollar (Modal owns serverless containers, Replicate owns model marketplace, Together / Fireworks own OSS LLM endpoints), and total-cost-of-ownership-at-sustained-load (RunPod / Lambda own raw GPU-hour cost). Everything else competes on benchmark numbers in the middle.
This is operator-translation territory. Most teams pick by benchmark or per-GPU-hour cost, then discover the actual constraint was either (a) developer velocity to ship the first version, or (b) cost-per-actual-inference-call once traffic patterns settled. The platform is the easy part · matching the platform to your actual traffic shape is what decides outcomes.
Pick the platform that matches your actual traffic shape,
not the one with the lowest benchmark number on a homepage.
Most asked questions · quick answers.
The questions readers send most often after reading the comparison. Answers are honest, workload-aware, and updated as the category moves.
Which AI inference platform is best for solo developers and indie hackers?
Replicate is the strongest pick for solo devs and indie hackers running occasional inference. The model-marketplace plus simple HTTP API means you can hit a Llama-3 or Stable Diffusion endpoint in 5 minutes with no GPU setup, and you only pay per second of compute. Modal is the close second if you also want to write custom Python · its serverless-container model gives you a real GPU runtime with zero infra. RunPod and Lambda Labs are the wrong fit for sporadic loads · you'll pay for idle GPU time.
What's the cheapest way to self-host OSS LLMs (Llama, Mistral, Qwen)?
For sustained throughput, RunPod and Lambda Labs win on raw GPU-hour cost · both rent bare-metal GPUs (H100, A100, RTX 4090) at the lowest per-hour rate. For per-token serverless pricing on OSS LLMs without managing the server, Together AI and Fireworks AI are the cheapest endpoints · both publish per-million-token pricing for Llama-3 / Mistral / Qwen / DeepSeek that frequently undercuts OpenAI's GPT-4-class pricing by 5-10x. The right answer depends on whether you have the load to justify reserved GPUs (RunPod / Lambda) or want pay-per-token (Together / Fireworks).
How is Modal different from RunPod?
Modal is serverless GPU containers with a Python-first developer experience · you write Python, decorate functions with GPU requirements, and Modal handles cold starts (typically 2-10 seconds), autoscaling, and infra. RunPod is bare-metal GPU rental · you pick a GPU, spin up a pod, SSH in, and run whatever you want. Modal optimizes for developer-velocity-and-elastic-load (great for spiky inference workloads). RunPod optimizes for raw-cost-per-GPU-hour (great for sustained training or long-running inference). If you'd rather write Python than configure Docker on a Linux box, Modal. If you want the lowest GPU-hour cost and don't mind ops, RunPod.
Which platform has the fastest serverless OSS LLM inference?
Fireworks AI consistently benchmarks at the top of serverless OSS-LLM inference latency in 2025-2026 · they've invested heavily in custom CUDA kernels, speculative decoding, and FP8 inference that materially beats vanilla vLLM throughput on the same hardware. Together AI is the close second with strong per-token pricing and a deeper model catalog. For absolute minimum cold-start latency on custom models, Modal's pre-warmed container pool wins. Replicate is good for breadth but typically slower on first-token-latency than Fireworks for the same OSS model.
Which platform is best for distributed training of large models?
Anyscale is purpose-built for distributed training and inference at scale · the platform is built on Ray (the open-source framework Anyscale's founders created at Berkeley), which is the de-facto standard for distributed Python ML workloads. If you're training a 70B+ parameter model across 32-256 GPUs, Anyscale's Ray-on-Kubernetes orchestration is the most mature option in this list. Lambda Labs is the close alternative if you'd rather rent the bare-metal cluster and run your own Ray / Slurm / Megatron stack. RunPod has multi-GPU pods but less mature distributed-training tooling. Modal supports multi-GPU but is positioned more for inference + lighter fine-tuning than full pretraining runs.
Can I escape OpenAI API bills by self-hosting OSS LLMs?
Yes, and this is the largest cost-arbitrage in 2026 inference. A Llama-3-70B or Qwen-2.5-72B endpoint on Together AI or Fireworks AI typically costs $0.50-0.90 per million input tokens and $0.80-1.50 per million output tokens · versus GPT-4-class pricing in the $5-15 / $15-30 per-million range. For 1M+ token/day workloads, that's a 10-30x bill reduction. The honest tradeoff: OSS models lag GPT-4 / Claude on hardest reasoning tasks but match or beat them on most production workloads (RAG, summarization, classification, structured extraction, agentic tool-use on well-defined schemas). Test on YOUR workload before migrating · quality parity is workload-specific.
What's the most common mistake teams make picking an inference platform?
Picking by per-GPU-hour price instead of by total-cost-per-inference-call. A bare-metal $1.99/hr H100 on RunPod looks cheap until you realize the GPU is idle 70% of the day waiting for traffic · your effective cost-per-call ends up higher than a $4/M-token serverless endpoint on Fireworks. The right framework: (1) measure your actual qps + variability, (2) if traffic is spiky, pay for serverless (Modal / Replicate / Together / Fireworks), (3) if traffic is sustained 24/7, rent bare-metal (RunPod / Lambda) and run your own vLLM / TensorRT-LLM, (4) only mix the two architectures if you have an ML platform team.
Which platform should an AI startup at 5-30 people pick for production LLM workload?
For an AI startup at 5-30 people running a production LLM workload, the honest pick is usually Modal or Together AI · Modal if you need to run custom code around the model (RAG pipelines, agent orchestration, multimodal preprocessing) on serverless GPUs, Together AI if you're calling OSS LLMs through a hosted endpoint and want predictable per-token billing. Fireworks AI is the close third for fastest serverless OSS inference. Avoid bare-metal (RunPod / Lambda) at this stage unless you have a dedicated infra engineer · the GPU-hour savings get eaten by ops time. Avoid Anyscale unless you're already deep in the Ray ecosystem.
Operator-honest · when to skip each platform.
No primary vendor will publish their own anti-fit list. Here's the honest "skip if X" guidance for each of the seven · built from operator decisions, production migration patterns, and the failure modes that actually show up after committing.
- Skip Modal if…
- …your traffic is sustained 24/7 high-qps where per-second serverless pricing compounds past bare-metal cost, you need sub-100ms cold start on every call (use a dedicated endpoint elsewhere), or you're shopping purely on raw cost-per-GPU-hour. Modal is the developer-velocity premium · wasted if developer velocity isn't the bottleneck.
- Skip RunPod if…
- …your traffic is spiky or unpredictable (you'll pay for idle GPU), nobody on your team is comfortable with Docker + CUDA on Linux, you need <2s cold start on every call, or you're a solo dev shipping a feature with sporadic load. RunPod wins on raw GPU-hour cost · wrong if your workload doesn't justify always-on rental.
- Skip Replicate if…
- …you need to run arbitrary Python around the model (use Modal), you need sub-second-latency real-time inference, you're serving an OSS LLM at high token throughput where Together / Fireworks per-token pricing wins, or you have enterprise SLA requirements. Replicate is the indie-hacker / model-marketplace play · wrong if your needs are beyond a model call.
- Skip Together AI if…
- …you need to run arbitrary code around the model (use Modal), your workload isn't an OSS LLM (use Replicate or Modal), you need first-token-latency leadership (Fireworks is faster), or you want raw GPU rental. Together AI is the cheapest per-token OSS LLM endpoint · wrong if "cheapest endpoint" isn't your actual constraint.
- Skip Anyscale if…
- …you're a solo dev or small startup (over-engineered), your team isn't already on Ray, you're cost-conscious and don't want a managed-Ray markup over raw cloud GPU, or you don't have distributed-training or multi-region serving as a real requirement. Anyscale is the ML-platform-team Ray standardization play · wrong if you don't have a platform team.
- Skip Lambda Labs if…
- …your workload is spiky inference (you'll pay for idle), you're a solo dev (RunPod Community is cheaper), you don't want to run your own training orchestration (use Anyscale), or you can't commit to a 1Y/3Y reserved cluster. Lambda Labs is RunPod's older sibling with reserved-cluster pricing · wrong if you don't want either bare-metal or reserved.
- Skip Fireworks AI if…
- …your workload isn't an OSS LLM (use Modal / Replicate), you need to run custom code around the model, you're shopping purely on per-token cost where Together AI is often slightly cheaper, or you want raw GPU rental. Fireworks is the fastest serverless OSS LLM in class · wrong if first-token-latency isn't the bottleneck.
Related operator guide:
⚖️ 6 New California AI Laws · Operator GuideStuck choosing?
If you're between two of these and the benchmark numbers aren't deciding it for you, text the actual constraint (traffic shape, model type, team ops capacity, monthly bill ceiling) and I'll send back which way I'd lean. Operator opinion, not vendor pitch.
Text PJ · 858-461-8054