Inference providers

An inference provider is a service that runs trained models on its own hardware and sells access to them through an API, so teams can use open-weight or proprietary models without operating GPUs. The category spans specialized clouds focused on open models, providers built on custom chips, and the AI platforms of the large public clouds.

Specialized providers such as Together AI, Fireworks AI, Baseten, and DeepInfra serve open models on GPUs, while Groq, Cerebras, and SambaNova run them on their own chips. Amazon Bedrock, Google Cloud's Gemini Enterprise Agent Platform, Microsoft Foundry, and OCI Enterprise AI add proprietary models from several labs, along with enterprise identity, networking, and billing. Hugging Face Inference Providers routes one API's traffic to several of these services.

Most providers offer serverless endpoints billed per token and dedicated deployments with reserved capacity. Many expose an OpenAI-compatible API, so moving a workload between them often means changing a base URL and a model name. Teams that want raw machines instead should compare GPU clouds.

Inference providers compared

PlatformOpen sourceDeploymentBest for
Amazon BedrockNoManaged cloud, APITeams standardized on AWS security and billing
BasetenNoManaged cloud, APIDeploying custom and fine-tuned models on dedicated GPUs
CerebrasNoManaged cloud, API, Self-hostedHigh token throughput on wafer-scale hardware
ChutesNoManaged cloud, APIOpen models on decentralized compute
Cloudflare Workers AINoManaged cloud, APIInference close to users inside Cloudflare's edge platform
DeepInfraNoManaged cloud, APIPer-token access to a broad catalog of open models
Featherless AINoManaged cloud, APIAccess to many community fine-tunes through one API
Fireworks AINoManaged cloud, APIServing and customizing open models in production
FriendliAINoManaged cloud, API, Self-hostedOne inference engine across hosted and owned GPUs
Gemini Enterprise Agent Platform (formerly Vertex AI)NoManaged cloud, APITeams building on Google Cloud
GroqNoManaged cloud, APILow-latency inference for supported open models
Hugging Face Inference ProvidersNoAPITrying many providers for Hub models with one token
HyperbolicNoManaged cloud, APIMixing rented GPUs and hosted inference in one account
IBM watsonx.aiNoManaged cloud, API, Self-hostedEnterprises running AI across hybrid infrastructure
Inference.netNoManaged cloud, APISwapping a general model for a task-specific one
Microsoft Foundry (formerly Azure AI Foundry)NoManaged cloud, APITeams standardized on Azure
ModalNoManaged cloud, CLICustom inference and batch jobs defined in Python
Nebius Token Factory (formerly Nebius AI Studio)NoManaged cloud, APIOpen-model inference with European hosting options
Novita AINoManaged cloud, APIOpen models, agent sandboxes, and GPUs from one vendor
NVIDIA NIMNoSelf-hosted, APIStandardized model containers for NVIDIA GPU fleets
OCI Enterprise AI (formerly OCI Generative AI)NoManaged cloud, APITeams running workloads on Oracle Cloud
OVHcloud AI EndpointsNoManaged cloud, APISovereign European inference for open models
ParasailNoManaged cloud, APIDedicated performance without GPU-hour commitments
ReplicateNoManaged cloud, APICalling image, video, and audio models through one API
SambaNovaNoManaged cloud, API, Self-hostedOpen-model inference on non-GPU hardware, hosted or on premises
Scaleway Generative APIsNoManaged cloud, APIEU-hosted serverless inference
SiliconFlowNoManaged cloud, APIServerless and dedicated access to open models
Together AINoManaged cloud, APIRunning and fine-tuning open models without managing GPUs

Profiles

  • Amazon Bedrock logo

    Amazon Bedrock

    Amazon Web Services · aws.amazon.com

    Amazon Bedrock is an AWS service that offers foundation models from Anthropic, Meta, Mistral AI, Amazon, and other providers through a common API, with Knowledge Bases, Guardrails, fine-tuning, and AgentCore for running agents.

    • Managed cloud
    • API
  • Baseten logo

    Baseten

    baseten.co

    Baseten is a platform for inference and training that deploys open-source, fine-tuned, and custom models on dedicated infrastructure, offers Model APIs for popular open models, and maintains Truss, an open-source model packaging tool.

    • Managed cloud
    • API
    • Python
  • Cerebras logo

    Cerebras

    Cerebras Systems · cerebras.ai · Sunnyvale, United States · founded 2015

    Cerebras serves open models through its Inference Cloud, an OpenAI-compatible API that runs on the company's wafer-scale processors, and sells dedicated capacity, custom model hosting, and on-premises systems to enterprises.

    • Managed cloud
    • API
    • Self-hosted
  • Chutes logo

    Chutes

    chutes.ai

    Chutes is a decentralized serverless platform that runs open-source models on distributed GPU capacity behind an OpenAI-compatible API, including model variants served inside trusted execution environments for private inference.

    • Managed cloud
    • API
  • Cloudflare Workers AI

    Cloudflare · cloudflare.com · San Francisco, United States · founded 2009

    Cloudflare Workers AI runs open models on serverless GPUs across Cloudflare's global network, callable from Workers or an OpenAI-compatible REST API, and pairs with Cloudflare AI Gateway for caching, rate limiting, and analytics.

    • Managed cloud
    • API
  • DeepInfra logo

    DeepInfra

    deepinfra.com

    DeepInfra hosts open-weight language, embedding, image, and speech models behind an OpenAI-compatible API billed per token, and also rents dedicated NVIDIA GPU clusters that it procures and operates for a single customer.

    • Managed cloud
    • API
  • Featherless AI logo

    Featherless AI

    featherless.ai

    Featherless AI is a serverless inference service that hosts tens of thousands of open models from Hugging Face, including fine-tunes and community models, behind one OpenAI-compatible API with subscription plans.

    • Managed cloud
    • API
  • Fireworks AI logo

    Fireworks AI

    fireworks.ai

    Fireworks AI is an inference and training platform for open models that offers serverless and dedicated deployments, supervised and reinforcement fine-tuning on a team's own data, and an OpenAI-compatible API.

    • Managed cloud
    • API
  • FriendliAI logo

    FriendliAI

    friendli.ai

    FriendliAI serves open-weight models through serverless Model APIs and Dedicated Endpoints on its own inference engine, and offers a bring-your-own-GPU option that runs the same engine on a customer's hardware.

    • Managed cloud
    • API
    • Self-hosted
  • Gemini Enterprise Agent Platform (formerly Vertex AI) logo

    Gemini Enterprise Agent Platform (formerly Vertex AI)

    Google Cloud · cloud.google.com

    Google Cloud's Gemini Enterprise Agent Platform serves Gemini, Gemma, Anthropic's Claude, and other partner and open models through Model Garden, alongside Agent Studio, Agent Runtime, model tuning, and evaluation services.

    Google replaced the Vertex AI brand with Gemini Enterprise Agent Platform in April 2026, and Vertex AI pages now redirect to it.

    • Managed cloud
    • API
  • Groq logo

    Groq

    groq.com · Mountain View, United States · founded 2016

    Groq runs GroqCloud, an inference service for open models on Groq's own LPU chips behind an OpenAI-compatible API; Groq licensed its inference technology to Nvidia in December 2025 and continues to operate GroqCloud.

    • Managed cloud
    • API
    • Python
    • TypeScript
  • Hugging Face Inference Providers logo

    Hugging Face Inference Providers

    Hugging Face · huggingface.co · New York, United States · founded 2016

    Hugging Face Inference Providers routes requests for Hub models to partner services such as Cerebras, Groq, Together AI, Fireworks, and Novita through one Hugging Face token and an OpenAI-compatible endpoint that picks a provider by speed, price, or preference.

    • API
    • Python
    • TypeScript
  • Hyperbolic logo

    Hyperbolic

    hyperbolic.ai

    Hyperbolic combines on-demand and reserved NVIDIA GPUs, managed Kubernetes GPU clusters, and a serverless inference API for open models on one platform, drawing on capacity from a network of data center operators.

    • Managed cloud
    • API
  • IBM watsonx.ai logo

    IBM watsonx.ai

    IBM · ibm.com

    IBM watsonx.ai is an AI development platform for building, tuning, and deploying IBM Granite, OpenAI gpt-oss, and other third-party models across IBM Cloud and hybrid environments, next to predictive and optimization workloads.

    • Managed cloud
    • API
    • Self-hosted
  • Inference.net logo

    Inference.net

    inference.net

    Inference.net deploys open models from its catalog or task-specific custom models it trains on a customer's data, and adds monitoring, tracing, and evaluation of production traffic across model providers.

    • Managed cloud
    • API
  • Microsoft Foundry (formerly Azure AI Foundry) logo

    Microsoft Foundry (formerly Azure AI Foundry)

    Microsoft · azure.microsoft.com

    Microsoft Foundry is Azure's platform for models and agents, with a catalog that includes models from OpenAI, Anthropic, Meta, Mistral AI, DeepSeek, xAI, and Microsoft, plus Foundry Agent Service, evaluations, and tracing.

    Microsoft renamed Azure AI Foundry to Microsoft Foundry in November 2025.

    • Managed cloud
    • API
  • Nebius Token Factory (formerly Nebius AI Studio) logo

    Nebius Token Factory (formerly Nebius AI Studio)

    Nebius · nebius.com · Amsterdam, Netherlands

    Nebius Token Factory is an inference service for open-weight language and image models on Nebius AI Cloud, offering per-token API access, dedicated endpoints, and fine-tuning from data centers in Europe and the United States.

    Nebius renamed its AI Studio inference service to Token Factory, and studio.nebius.com now redirects to tokenfactory.nebius.com.

    • Managed cloud
    • API
  • Novita AI logo

    Novita AI

    novita.ai

    Novita AI provides one API for open-weight language, image, video, and audio models, along with secure sandboxes for running AI agents and on-demand GPU instances for custom workloads.

    • Managed cloud
    • API
  • NVIDIA NIM logo

    NVIDIA NIM

    NVIDIA · nvidia.com

    NVIDIA NIM packages open and partner models with optimized inference engines into containers that teams self-host on NVIDIA GPUs, and offers hosted NIM APIs on build.nvidia.com for prototyping.

    • Self-hosted
    • API
  • OCI Enterprise AI (formerly OCI Generative AI) logo

    OCI Enterprise AI (formerly OCI Generative AI)

    Oracle · oracle.com

    OCI Enterprise AI is Oracle Cloud's managed service for models and agents, offering models from Cohere, Meta, xAI, OpenAI, and others, plus Dedicated AI Clusters, guardrails, observability, and access control.

    The former OCI Generative AI service page now redirects to OCI Enterprise AI, which keeps the Dedicated AI Clusters feature.

    • Managed cloud
    • API
  • OVHcloud AI Endpoints logo

    OVHcloud AI Endpoints

    OVHcloud · ovhcloud.com · Roubaix, France · founded 1999

    OVHcloud AI Endpoints is a serverless inference service from the French cloud provider OVHcloud that serves open-weight language, speech, and vision models through an OpenAI-compatible API on European infrastructure.

    • Managed cloud
    • API
  • Parasail logo

    Parasail

    parasail.io

    Parasail is an inference cloud for open models that offers serverless endpoints, dedicated deployments, and elastic endpoints that give eligible models dedicated GPU performance while still billing per token.

    • Managed cloud
    • API
  • Replicate

    replicate.com

    Replicate runs open-source and community models, many of them for images, video, and audio, through a single API, and packages custom models as containers with Cog, its open-source tool.

    Cloudflare acquired Replicate, which continues to operate under its own brand while its model catalog comes to Cloudflare Workers AI.

    • Managed cloud
    • API
    • Python
    • TypeScript
  • SambaNova logo

    SambaNova

    SambaNova Systems · sambanova.ai · founded 2017

    SambaNova runs SambaCloud, an inference service for open models on the company's own reconfigurable dataflow unit (RDU) chips, and sells SambaStack and SambaRack systems for running the same stack in customers' data centers.

    • Managed cloud
    • API
    • Self-hosted
  • Scaleway Generative APIs logo

    Scaleway Generative APIs

    Scaleway · scaleway.com · Paris, France

    Scaleway Generative APIs is a serverless, OpenAI-compatible inference service from the French cloud provider Scaleway that serves open models with data kept in Europe and a Batch API for non-realtime jobs.

    • Managed cloud
    • API
  • SiliconFlow logo

    SiliconFlow

    siliconflow.com

    SiliconFlow is an AI infrastructure platform that serves open language and multimodal models through serverless APIs, dedicated endpoints, and reserved GPUs, with fine-tuning and one-click deployment of customized models.

    • Managed cloud
    • API
  • Together AI logo

    Together AI

    together.ai · San Francisco, United States

    Together AI runs open-weight models on serverless and dedicated inference, and also offers fine-tuning, batch inference, and NVIDIA GPU clusters that range from self-serve instant clusters to large reserved deployments.

    • Managed cloud
    • API
    • Python
    • TypeScript

How to choose

01Model catalog
Which open-weight and proprietary models are available, and how quickly new releases appear.
02Serving mode
Serverless per-token endpoints, dedicated deployments with reserved capacity, or batch processing for offline jobs.
03Performance
Time to first token and output tokens per second for your models and prompt lengths, measured on your own traffic.
04Customization
Support for fine-tuning, LoRA adapters, and deploying custom weights or containers.
05Compliance and regions
Data retention terms, regional hosting, and the certifications your organization requires.
06API compatibility
Whether the API follows the OpenAI format, which determines how easily a gateway or SDK can switch providers.

Frequently asked questions

What is an inference provider?

An inference provider runs AI models on its own infrastructure and charges for access through an API, usually per input and output token or per hour of reserved capacity. Developers send requests and receive generated output without managing GPUs, drivers, or serving software. Together AI, Fireworks AI, Groq, and the AI services of AWS, Google Cloud, and Microsoft Azure are examples.

Serverless vs dedicated inference: what is the difference?

Serverless inference shares capacity across customers and bills per token, which suits variable or low traffic and requires no capacity planning. Dedicated inference reserves GPUs for one customer and bills by time, which gives more predictable latency, supports custom or fine-tuned weights, and can lower unit cost at steady high volume. Many providers offer both modes.

What are Hugging Face Inference Providers?

Hugging Face Inference Providers is a Hugging Face service that routes requests for Hub models to partner inference companies, including Cerebras, Groq, Together AI, and Fireworks, through one API and one Hugging Face token. It offers an OpenAI-compatible chat endpoint and picks the provider with the highest throughput by default, or picks by price or by name when the request asks for it.

What is the fastest LLM inference provider?

No provider is fastest for every model and prompt. Groq, Cerebras, and SambaNova run models on their own chips and target high output speed for the models they support, while GPU providers tune engines such as vLLM, SGLang, and TensorRT-LLM. Speed also changes with model size, context length, and load, so measure time to first token and tokens per second on your own traffic.

What is an OpenAI-compatible API?

An OpenAI-compatible API accepts the same request and response format as OpenAI's Chat Completions endpoint, so code written for the OpenAI SDK can call another provider by changing the base URL, API key, and model name. Most inference providers and serving engines offer one. Compatibility usually covers chat, streaming, and tool calling, while structured outputs, reasoning parameters, and newer endpoints vary by provider.

LLM inference vs training: what is the difference?

Training adjusts a model's weights by processing large datasets over many GPU hours, and it happens before deployment. Inference uses the finished weights to generate output for each request, so its cost scales with traffic and its performance is measured in latency and throughput. Inference providers handle the second stage, and several also offer fine-tuning.

Other categories

Product details change quickly; check each vendor’s site for current features, licensing and pricing.