Inference providers
An inference provider is a service that runs trained models on its own hardware and sells access to them through an API, so teams can use open-weight or proprietary models without operating GPUs. The category spans specialized clouds focused on open models, providers built on custom chips, and the AI platforms of the large public clouds.
Specialized providers such as Together AI, Fireworks AI, Baseten, and DeepInfra serve open models on GPUs, while Groq, Cerebras, and SambaNova run them on their own chips. Amazon Bedrock, Google Cloud's Gemini Enterprise Agent Platform, Microsoft Foundry, and OCI Enterprise AI add proprietary models from several labs, along with enterprise identity, networking, and billing. Hugging Face Inference Providers routes one API's traffic to several of these services.
Most providers offer serverless endpoints billed per token and dedicated deployments with reserved capacity. Many expose an OpenAI-compatible API, so moving a workload between them often means changing a base URL and a model name. Teams that want raw machines instead should compare GPU clouds.
Inference providers compared
| Platform | Open source | Deployment | Best for |
|---|---|---|---|
| Amazon Bedrock | No | Managed cloud, API | Teams standardized on AWS security and billing |
| Baseten | No | Managed cloud, API | Deploying custom and fine-tuned models on dedicated GPUs |
| Cerebras | No | Managed cloud, API, Self-hosted | High token throughput on wafer-scale hardware |
| Chutes | No | Managed cloud, API | Open models on decentralized compute |
| Cloudflare Workers AI | No | Managed cloud, API | Inference close to users inside Cloudflare's edge platform |
| DeepInfra | No | Managed cloud, API | Per-token access to a broad catalog of open models |
| Featherless AI | No | Managed cloud, API | Access to many community fine-tunes through one API |
| Fireworks AI | No | Managed cloud, API | Serving and customizing open models in production |
| FriendliAI | No | Managed cloud, API, Self-hosted | One inference engine across hosted and owned GPUs |
| Gemini Enterprise Agent Platform (formerly Vertex AI) | No | Managed cloud, API | Teams building on Google Cloud |
| Groq | No | Managed cloud, API | Low-latency inference for supported open models |
| Hugging Face Inference Providers | No | API | Trying many providers for Hub models with one token |
| Hyperbolic | No | Managed cloud, API | Mixing rented GPUs and hosted inference in one account |
| IBM watsonx.ai | No | Managed cloud, API, Self-hosted | Enterprises running AI across hybrid infrastructure |
| Inference.net | No | Managed cloud, API | Swapping a general model for a task-specific one |
| Microsoft Foundry (formerly Azure AI Foundry) | No | Managed cloud, API | Teams standardized on Azure |
| Modal | No | Managed cloud, CLI | Custom inference and batch jobs defined in Python |
| Nebius Token Factory (formerly Nebius AI Studio) | No | Managed cloud, API | Open-model inference with European hosting options |
| Novita AI | No | Managed cloud, API | Open models, agent sandboxes, and GPUs from one vendor |
| NVIDIA NIM | No | Self-hosted, API | Standardized model containers for NVIDIA GPU fleets |
| OCI Enterprise AI (formerly OCI Generative AI) | No | Managed cloud, API | Teams running workloads on Oracle Cloud |
| OVHcloud AI Endpoints | No | Managed cloud, API | Sovereign European inference for open models |
| Parasail | No | Managed cloud, API | Dedicated performance without GPU-hour commitments |
| Replicate | No | Managed cloud, API | Calling image, video, and audio models through one API |
| SambaNova | No | Managed cloud, API, Self-hosted | Open-model inference on non-GPU hardware, hosted or on premises |
| Scaleway Generative APIs | No | Managed cloud, API | EU-hosted serverless inference |
| SiliconFlow | No | Managed cloud, API | Serverless and dedicated access to open models |
| Together AI | No | Managed cloud, API | Running and fine-tuning open models without managing GPUs |
Profiles

Amazon Bedrock
Amazon Web Services · aws.amazon.com
Amazon Bedrock is an AWS service that offers foundation models from Anthropic, Meta, Mistral AI, Amazon, and other providers through a common API, with Knowledge Bases, Guardrails, fine-tuning, and AgentCore for running agents.

Baseten
baseten.co
Baseten is a platform for inference and training that deploys open-source, fine-tuned, and custom models on dedicated infrastructure, offers Model APIs for popular open models, and maintains Truss, an open-source model packaging tool.

Cerebras
Cerebras Systems · cerebras.ai · Sunnyvale, United States · founded 2015
Cerebras serves open models through its Inference Cloud, an OpenAI-compatible API that runs on the company's wafer-scale processors, and sells dedicated capacity, custom model hosting, and on-premises systems to enterprises.

Chutes
chutes.ai
Chutes is a decentralized serverless platform that runs open-source models on distributed GPU capacity behind an OpenAI-compatible API, including model variants served inside trusted execution environments for private inference.
Cloudflare Workers AI
Cloudflare · cloudflare.com · San Francisco, United States · founded 2009
Cloudflare Workers AI runs open models on serverless GPUs across Cloudflare's global network, callable from Workers or an OpenAI-compatible REST API, and pairs with Cloudflare AI Gateway for caching, rate limiting, and analytics.

DeepInfra
deepinfra.com
DeepInfra hosts open-weight language, embedding, image, and speech models behind an OpenAI-compatible API billed per token, and also rents dedicated NVIDIA GPU clusters that it procures and operates for a single customer.
Featherless AI
featherless.ai
Featherless AI is a serverless inference service that hosts tens of thousands of open models from Hugging Face, including fine-tunes and community models, behind one OpenAI-compatible API with subscription plans.
Fireworks AI
fireworks.ai
Fireworks AI is an inference and training platform for open models that offers serverless and dedicated deployments, supervised and reinforcement fine-tuning on a team's own data, and an OpenAI-compatible API.
FriendliAI
friendli.ai
FriendliAI serves open-weight models through serverless Model APIs and Dedicated Endpoints on its own inference engine, and offers a bring-your-own-GPU option that runs the same engine on a customer's hardware.

Gemini Enterprise Agent Platform (formerly Vertex AI)
Google Cloud · cloud.google.com
Google Cloud's Gemini Enterprise Agent Platform serves Gemini, Gemma, Anthropic's Claude, and other partner and open models through Model Garden, alongside Agent Studio, Agent Runtime, model tuning, and evaluation services.
Google replaced the Vertex AI brand with Gemini Enterprise Agent Platform in April 2026, and Vertex AI pages now redirect to it.
Groq
groq.com · Mountain View, United States · founded 2016
Groq runs GroqCloud, an inference service for open models on Groq's own LPU chips behind an OpenAI-compatible API; Groq licensed its inference technology to Nvidia in December 2025 and continues to operate GroqCloud.
Hugging Face Inference Providers
Hugging Face · huggingface.co · New York, United States · founded 2016
Hugging Face Inference Providers routes requests for Hub models to partner services such as Cerebras, Groq, Together AI, Fireworks, and Novita through one Hugging Face token and an OpenAI-compatible endpoint that picks a provider by speed, price, or preference.

Hyperbolic
hyperbolic.ai
Hyperbolic combines on-demand and reserved NVIDIA GPUs, managed Kubernetes GPU clusters, and a serverless inference API for open models on one platform, drawing on capacity from a network of data center operators.

IBM watsonx.ai
IBM · ibm.com
IBM watsonx.ai is an AI development platform for building, tuning, and deploying IBM Granite, OpenAI gpt-oss, and other third-party models across IBM Cloud and hybrid environments, next to predictive and optimization workloads.
Inference.net
inference.net
Inference.net deploys open models from its catalog or task-specific custom models it trains on a customer's data, and adds monitoring, tracing, and evaluation of production traffic across model providers.
Microsoft Foundry (formerly Azure AI Foundry)
Microsoft · azure.microsoft.com
Microsoft Foundry is Azure's platform for models and agents, with a catalog that includes models from OpenAI, Anthropic, Meta, Mistral AI, DeepSeek, xAI, and Microsoft, plus Foundry Agent Service, evaluations, and tracing.
Microsoft renamed Azure AI Foundry to Microsoft Foundry in November 2025.
Modal
Modal Labs · modal.com
Modal is a serverless cloud for AI workloads where teams define GPU functions, model endpoints, training jobs, batch processing, and sandboxes for agent code in Python, with capacity that scales with demand.
Nebius Token Factory (formerly Nebius AI Studio)
Nebius · nebius.com · Amsterdam, Netherlands
Nebius Token Factory is an inference service for open-weight language and image models on Nebius AI Cloud, offering per-token API access, dedicated endpoints, and fine-tuning from data centers in Europe and the United States.
Nebius renamed its AI Studio inference service to Token Factory, and studio.nebius.com now redirects to tokenfactory.nebius.com.
Novita AI
novita.ai
Novita AI provides one API for open-weight language, image, video, and audio models, along with secure sandboxes for running AI agents and on-demand GPU instances for custom workloads.

NVIDIA NIM
NVIDIA · nvidia.com
NVIDIA NIM packages open and partner models with optimized inference engines into containers that teams self-host on NVIDIA GPUs, and offers hosted NIM APIs on build.nvidia.com for prototyping.

OCI Enterprise AI (formerly OCI Generative AI)
Oracle · oracle.com
OCI Enterprise AI is Oracle Cloud's managed service for models and agents, offering models from Cohere, Meta, xAI, OpenAI, and others, plus Dedicated AI Clusters, guardrails, observability, and access control.
The former OCI Generative AI service page now redirects to OCI Enterprise AI, which keeps the Dedicated AI Clusters feature.
OVHcloud AI Endpoints
OVHcloud · ovhcloud.com · Roubaix, France · founded 1999
OVHcloud AI Endpoints is a serverless inference service from the French cloud provider OVHcloud that serves open-weight language, speech, and vision models through an OpenAI-compatible API on European infrastructure.
Parasail
parasail.io
Parasail is an inference cloud for open models that offers serverless endpoints, dedicated deployments, and elastic endpoints that give eligible models dedicated GPU performance while still billing per token.
Replicate
replicate.com
Replicate runs open-source and community models, many of them for images, video, and audio, through a single API, and packages custom models as containers with Cog, its open-source tool.
Cloudflare acquired Replicate, which continues to operate under its own brand while its model catalog comes to Cloudflare Workers AI.

SambaNova
SambaNova Systems · sambanova.ai · founded 2017
SambaNova runs SambaCloud, an inference service for open models on the company's own reconfigurable dataflow unit (RDU) chips, and sells SambaStack and SambaRack systems for running the same stack in customers' data centers.
Scaleway Generative APIs
Scaleway · scaleway.com · Paris, France
Scaleway Generative APIs is a serverless, OpenAI-compatible inference service from the French cloud provider Scaleway that serves open models with data kept in Europe and a Batch API for non-realtime jobs.

SiliconFlow
siliconflow.com
SiliconFlow is an AI infrastructure platform that serves open language and multimodal models through serverless APIs, dedicated endpoints, and reserved GPUs, with fine-tuning and one-click deployment of customized models.

Together AI
together.ai · San Francisco, United States
Together AI runs open-weight models on serverless and dedicated inference, and also offers fine-tuning, batch inference, and NVIDIA GPU clusters that range from self-serve instant clusters to large reserved deployments.
How to choose
- 01Model catalog
- Which open-weight and proprietary models are available, and how quickly new releases appear.
- 02Serving mode
- Serverless per-token endpoints, dedicated deployments with reserved capacity, or batch processing for offline jobs.
- 03Performance
- Time to first token and output tokens per second for your models and prompt lengths, measured on your own traffic.
- 04Customization
- Support for fine-tuning, LoRA adapters, and deploying custom weights or containers.
- 05Compliance and regions
- Data retention terms, regional hosting, and the certifications your organization requires.
- 06API compatibility
- Whether the API follows the OpenAI format, which determines how easily a gateway or SDK can switch providers.
Frequently asked questions
What is an inference provider?
- An inference provider runs AI models on its own infrastructure and charges for access through an API, usually per input and output token or per hour of reserved capacity. Developers send requests and receive generated output without managing GPUs, drivers, or serving software. Together AI, Fireworks AI, Groq, and the AI services of AWS, Google Cloud, and Microsoft Azure are examples.
Serverless vs dedicated inference: what is the difference?
- Serverless inference shares capacity across customers and bills per token, which suits variable or low traffic and requires no capacity planning. Dedicated inference reserves GPUs for one customer and bills by time, which gives more predictable latency, supports custom or fine-tuned weights, and can lower unit cost at steady high volume. Many providers offer both modes.
What are Hugging Face Inference Providers?
- Hugging Face Inference Providers is a Hugging Face service that routes requests for Hub models to partner inference companies, including Cerebras, Groq, Together AI, and Fireworks, through one API and one Hugging Face token. It offers an OpenAI-compatible chat endpoint and picks the provider with the highest throughput by default, or picks by price or by name when the request asks for it.
What is the fastest LLM inference provider?
- No provider is fastest for every model and prompt. Groq, Cerebras, and SambaNova run models on their own chips and target high output speed for the models they support, while GPU providers tune engines such as vLLM, SGLang, and TensorRT-LLM. Speed also changes with model size, context length, and load, so measure time to first token and tokens per second on your own traffic.
What is an OpenAI-compatible API?
- An OpenAI-compatible API accepts the same request and response format as OpenAI's Chat Completions endpoint, so code written for the OpenAI SDK can call another provider by changing the base URL, API key, and model name. Most inference providers and serving engines offer one. Compatibility usually covers chat, streaming, and tool calling, while structured outputs, reasoning parameters, and newer endpoints vary by provider.
LLM inference vs training: what is the difference?
- Training adjusts a model's weights by processing large datasets over many GPU hours, and it happens before deployment. Inference uses the finished weights to generate output for each request, so its cost scales with traffic and its performance is measured in latency and throughput. Inference providers handle the second stage, and several also offer fine-tuning.
Related guides
LLM inference and serving hubOther categories
- Model providers
- GPU clouds
- Serving engines
- Vector databases
- Embedding and reranking models
- Document parsing and data ingestion
- Fine-tuning and post-training
- Agent frameworks
- Agent infrastructure
- Web search APIs
- MCP platforms
- Voice AI
- AI gateways
- LLM observability and evaluation
- Guardrails and AI security
- AI coding tools
Product details change quickly; check each vendor’s site for current features, licensing and pricing.