---
title: "Inference providers"
description: "Compare LLM inference providers, from Together AI, Fireworks, Groq, and Cerebras to Amazon Bedrock and Microsoft Foundry, by serving mode, models, and APIs."
type: "platform-category"
canonical_url: "https://theaiengineering.com/platforms/inference-providers"
markdown_url: "https://theaiengineering.com/platforms/inference-providers.md"
topic: "LLM inference and serving"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
citation: "The AI Engineering Editors. \"Inference providers.\" The AI Engineering. https://theaiengineering.com/platforms/inference-providers"
---

# Inference providers

An inference provider is a service that runs trained models on its own hardware and sells access to them through an API, so teams can use open-weight or proprietary models without operating GPUs. The category spans specialized clouds focused on open models, providers built on custom chips, and the AI platforms of the large public clouds.

Specialized providers such as Together AI, Fireworks AI, Baseten, and DeepInfra serve open models on GPUs, while Groq, Cerebras, and SambaNova run them on their own chips. Amazon Bedrock, Google Cloud's Gemini Enterprise Agent Platform, Microsoft Foundry, and OCI Enterprise AI add proprietary models from several labs, along with enterprise identity, networking, and billing. Hugging Face Inference Providers routes one API's traffic to several of these services.

Most providers offer serverless endpoints billed per token and dedicated deployments with reserved capacity. Many expose an OpenAI-compatible API, so moving a workload between them often means changing a base URL and a model name. Teams that want raw machines instead should compare GPU clouds.

## Inference providers compared

| Platform | Open source | Deployment | Best for |
| --- | --- | --- | --- |
| [Amazon Bedrock](https://aws.amazon.com/bedrock/) | No | Managed cloud, API | Teams standardized on AWS security and billing |
| [Baseten](https://www.baseten.co) | No | Managed cloud, API | Deploying custom and fine-tuned models on dedicated GPUs |
| [Cerebras](https://www.cerebras.ai) | No | Managed cloud, API, Self-hosted | High token throughput on wafer-scale hardware |
| [Chutes](https://chutes.ai) | No | Managed cloud, API | Open models on decentralized compute |
| [Cloudflare Workers AI](https://www.cloudflare.com/developer-platform/products/workers-ai/) | No | Managed cloud, API | Inference close to users inside Cloudflare's edge platform |
| [DeepInfra](https://deepinfra.com) | No | Managed cloud, API | Per-token access to a broad catalog of open models |
| [Featherless AI](https://featherless.ai) | No | Managed cloud, API | Access to many community fine-tunes through one API |
| [Fireworks AI](https://fireworks.ai) | No | Managed cloud, API | Serving and customizing open models in production |
| [FriendliAI](https://friendli.ai) | No | Managed cloud, API, Self-hosted | One inference engine across hosted and owned GPUs |
| [Gemini Enterprise Agent Platform (formerly Vertex AI)](https://cloud.google.com/products/gemini-enterprise-agent-platform) | No | Managed cloud, API | Teams building on Google Cloud |
| [Groq](https://groq.com) | No | Managed cloud, API | Low-latency inference for supported open models |
| [Hugging Face Inference Providers](https://huggingface.co/docs/inference-providers/index) | No | API | Trying many providers for Hub models with one token |
| [Hyperbolic](https://hyperbolic.ai) | No | Managed cloud, API | Mixing rented GPUs and hosted inference in one account |
| [IBM watsonx.ai](https://www.ibm.com/products/watsonx-ai) | No | Managed cloud, API, Self-hosted | Enterprises running AI across hybrid infrastructure |
| [Inference.net](https://inference.net) | No | Managed cloud, API | Swapping a general model for a task-specific one |
| [Microsoft Foundry (formerly Azure AI Foundry)](https://azure.microsoft.com/en-us/products/ai-foundry) | No | Managed cloud, API | Teams standardized on Azure |
| [Modal](https://modal.com) | No | Managed cloud, CLI | Custom inference and batch jobs defined in Python |
| [Nebius Token Factory (formerly Nebius AI Studio)](https://nebius.com/token-factory) | No | Managed cloud, API | Open-model inference with European hosting options |
| [Novita AI](https://novita.ai) | No | Managed cloud, API | Open models, agent sandboxes, and GPUs from one vendor |
| [NVIDIA NIM](https://www.nvidia.com/en-us/ai-data-science/products/nim-microservices/) | No | Self-hosted, API | Standardized model containers for NVIDIA GPU fleets |
| [OCI Enterprise AI (formerly OCI Generative AI)](https://www.oracle.com/artificial-intelligence/enterprise-ai/) | No | Managed cloud, API | Teams running workloads on Oracle Cloud |
| [OVHcloud AI Endpoints](https://www.ovhcloud.com/en/public-cloud/ai-endpoints/) | No | Managed cloud, API | Sovereign European inference for open models |
| [Parasail](https://www.parasail.io) | No | Managed cloud, API | Dedicated performance without GPU-hour commitments |
| [Replicate](https://replicate.com) | No | Managed cloud, API | Calling image, video, and audio models through one API |
| [SambaNova](https://sambanova.ai) | No | Managed cloud, API, Self-hosted | Open-model inference on non-GPU hardware, hosted or on premises |
| [Scaleway Generative APIs](https://www.scaleway.com/en/generative-apis/) | No | Managed cloud, API | EU-hosted serverless inference |
| [SiliconFlow](https://www.siliconflow.com) | No | Managed cloud, API | Serverless and dedicated access to open models |
| [Together AI](https://www.together.ai) | No | Managed cloud, API | Running and fine-tuning open models without managing GPUs |

## Profiles

### Amazon Bedrock

Amazon Bedrock is an AWS service that offers foundation models from Anthropic, Meta, Mistral AI, Amazon, and other providers through a common API, with Knowledge Bases, Guardrails, fine-tuning, and AgentCore for running agents.

- Company: Amazon Web Services
- Website: https://aws.amazon.com/bedrock/
- Open source: No
- Deployment: Managed cloud, API
- Best for: Teams standardized on AWS security and billing
- Permalink: https://theaiengineering.com/platforms/inference-providers#amazon-bedrock

### Baseten

Baseten is a platform for inference and training that deploys open-source, fine-tuned, and custom models on dedicated infrastructure, offers Model APIs for popular open models, and maintains Truss, an open-source model packaging tool.

- Website: https://www.baseten.co
- GitHub: https://github.com/basetenlabs
- Open source: No
- Deployment: Managed cloud, API
- Languages: Python
- Best for: Deploying custom and fine-tuned models on dedicated GPUs
- Permalink: https://theaiengineering.com/platforms/inference-providers#baseten

### Cerebras

Cerebras serves open models through its Inference Cloud, an OpenAI-compatible API that runs on the company's wafer-scale processors, and sells dedicated capacity, custom model hosting, and on-premises systems to enterprises.

- Company: Cerebras Systems
- Website: https://www.cerebras.ai
- GitHub: https://github.com/Cerebras
- Headquarters: Sunnyvale, United States
- Founded: 2015
- Open source: No
- Deployment: Managed cloud, API, Self-hosted
- Best for: High token throughput on wafer-scale hardware
- Permalink: https://theaiengineering.com/platforms/inference-providers#cerebras

### Chutes

Chutes is a decentralized serverless platform that runs open-source models on distributed GPU capacity behind an OpenAI-compatible API, including model variants served inside trusted execution environments for private inference.

- Website: https://chutes.ai
- Open source: No
- Deployment: Managed cloud, API
- Best for: Open models on decentralized compute
- Permalink: https://theaiengineering.com/platforms/inference-providers#chutes

### Cloudflare Workers AI

Cloudflare Workers AI runs open models on serverless GPUs across Cloudflare's global network, callable from Workers or an OpenAI-compatible REST API, and pairs with Cloudflare AI Gateway for caching, rate limiting, and analytics.

- Company: Cloudflare
- Website: https://www.cloudflare.com/developer-platform/products/workers-ai/
- GitHub: https://github.com/cloudflare
- Headquarters: San Francisco, United States
- Founded: 2009
- Open source: No
- Deployment: Managed cloud, API
- Best for: Inference close to users inside Cloudflare's edge platform
- Permalink: https://theaiengineering.com/platforms/inference-providers#cloudflare-workers-ai

### DeepInfra

DeepInfra hosts open-weight language, embedding, image, and speech models behind an OpenAI-compatible API billed per token, and also rents dedicated NVIDIA GPU clusters that it procures and operates for a single customer.

- Website: https://deepinfra.com
- GitHub: https://github.com/deepinfra
- Open source: No
- Deployment: Managed cloud, API
- Best for: Per-token access to a broad catalog of open models
- Permalink: https://theaiengineering.com/platforms/inference-providers#deepinfra

### Featherless AI

Featherless AI is a serverless inference service that hosts tens of thousands of open models from Hugging Face, including fine-tunes and community models, behind one OpenAI-compatible API with subscription plans.

- Website: https://featherless.ai
- Open source: No
- Deployment: Managed cloud, API
- Best for: Access to many community fine-tunes through one API
- Permalink: https://theaiengineering.com/platforms/inference-providers#featherless-ai

### Fireworks AI

Fireworks AI is an inference and training platform for open models that offers serverless and dedicated deployments, supervised and reinforcement fine-tuning on a team's own data, and an OpenAI-compatible API.

- Website: https://fireworks.ai
- GitHub: https://github.com/fw-ai
- Open source: No
- Deployment: Managed cloud, API
- Best for: Serving and customizing open models in production
- Permalink: https://theaiengineering.com/platforms/inference-providers#fireworks-ai

### FriendliAI

FriendliAI serves open-weight models through serverless Model APIs and Dedicated Endpoints on its own inference engine, and offers a bring-your-own-GPU option that runs the same engine on a customer's hardware.

- Website: https://friendli.ai
- Open source: No
- Deployment: Managed cloud, API, Self-hosted
- Best for: One inference engine across hosted and owned GPUs
- Permalink: https://theaiengineering.com/platforms/inference-providers#friendliai

### Gemini Enterprise Agent Platform (formerly Vertex AI)

Google Cloud's Gemini Enterprise Agent Platform serves Gemini, Gemma, Anthropic's Claude, and other partner and open models through Model Garden, alongside Agent Studio, Agent Runtime, model tuning, and evaluation services.

Google replaced the Vertex AI brand with Gemini Enterprise Agent Platform in April 2026, and Vertex AI pages now redirect to it.

- Company: Google Cloud
- Website: https://cloud.google.com/products/gemini-enterprise-agent-platform
- Open source: No
- Deployment: Managed cloud, API
- Best for: Teams building on Google Cloud
- Permalink: https://theaiengineering.com/platforms/inference-providers#gemini-enterprise-agent-platform

### Groq

Groq runs GroqCloud, an inference service for open models on Groq's own LPU chips behind an OpenAI-compatible API; Groq licensed its inference technology to Nvidia in December 2025 and continues to operate GroqCloud.

- Website: https://groq.com
- GitHub: https://github.com/groq
- Headquarters: Mountain View, United States
- Founded: 2016
- Open source: No
- Deployment: Managed cloud, API
- Languages: Python, TypeScript
- Best for: Low-latency inference for supported open models
- Permalink: https://theaiengineering.com/platforms/inference-providers#groq

### Hugging Face Inference Providers

Hugging Face Inference Providers routes requests for Hub models to partner services such as Cerebras, Groq, Together AI, Fireworks, and Novita through one Hugging Face token and an OpenAI-compatible endpoint that picks a provider by speed, price, or preference.

- Company: Hugging Face
- Website: https://huggingface.co/docs/inference-providers/index
- GitHub: https://github.com/huggingface
- Headquarters: New York, United States
- Founded: 2016
- Open source: No
- Deployment: API
- Languages: Python, TypeScript
- Best for: Trying many providers for Hub models with one token
- Permalink: https://theaiengineering.com/platforms/inference-providers#hugging-face-inference-providers

### Hyperbolic

Hyperbolic combines on-demand and reserved NVIDIA GPUs, managed Kubernetes GPU clusters, and a serverless inference API for open models on one platform, drawing on capacity from a network of data center operators.

- Website: https://hyperbolic.ai
- Open source: No
- Deployment: Managed cloud, API
- Best for: Mixing rented GPUs and hosted inference in one account
- Permalink: https://theaiengineering.com/platforms/inference-providers#hyperbolic

### IBM watsonx.ai

IBM watsonx.ai is an AI development platform for building, tuning, and deploying IBM Granite, OpenAI gpt-oss, and other third-party models across IBM Cloud and hybrid environments, next to predictive and optimization workloads.

- Company: IBM
- Website: https://www.ibm.com/products/watsonx-ai
- Open source: No
- Deployment: Managed cloud, API, Self-hosted
- Best for: Enterprises running AI across hybrid infrastructure
- Permalink: https://theaiengineering.com/platforms/inference-providers#ibm-watsonx-ai

### Inference.net

Inference.net deploys open models from its catalog or task-specific custom models it trains on a customer's data, and adds monitoring, tracing, and evaluation of production traffic across model providers.

- Website: https://inference.net
- Open source: No
- Deployment: Managed cloud, API
- Best for: Swapping a general model for a task-specific one
- Permalink: https://theaiengineering.com/platforms/inference-providers#inference-net

### Microsoft Foundry (formerly Azure AI Foundry)

Microsoft Foundry is Azure's platform for models and agents, with a catalog that includes models from OpenAI, Anthropic, Meta, Mistral AI, DeepSeek, xAI, and Microsoft, plus Foundry Agent Service, evaluations, and tracing.

Microsoft renamed Azure AI Foundry to Microsoft Foundry in November 2025.

- Company: Microsoft
- Website: https://azure.microsoft.com/en-us/products/ai-foundry
- Open source: No
- Deployment: Managed cloud, API
- Best for: Teams standardized on Azure
- Permalink: https://theaiengineering.com/platforms/inference-providers#microsoft-foundry

### Modal

Modal is a serverless cloud for AI workloads where teams define GPU functions, model endpoints, training jobs, batch processing, and sandboxes for agent code in Python, with capacity that scales with demand.

- Company: Modal Labs
- Website: https://modal.com
- GitHub: https://github.com/modal-labs
- Open source: No
- Deployment: Managed cloud, CLI
- Languages: Python
- Best for: Custom inference and batch jobs defined in Python
- Permalink: https://theaiengineering.com/platforms/inference-providers#modal

### Nebius Token Factory (formerly Nebius AI Studio)

Nebius Token Factory is an inference service for open-weight language and image models on Nebius AI Cloud, offering per-token API access, dedicated endpoints, and fine-tuning from data centers in Europe and the United States.

Nebius renamed its AI Studio inference service to Token Factory, and studio.nebius.com now redirects to tokenfactory.nebius.com.

- Company: Nebius
- Website: https://nebius.com/token-factory
- Headquarters: Amsterdam, Netherlands
- Open source: No
- Deployment: Managed cloud, API
- Best for: Open-model inference with European hosting options
- Permalink: https://theaiengineering.com/platforms/inference-providers#nebius-token-factory

### Novita AI

Novita AI provides one API for open-weight language, image, video, and audio models, along with secure sandboxes for running AI agents and on-demand GPU instances for custom workloads.

- Website: https://novita.ai
- Open source: No
- Deployment: Managed cloud, API
- Best for: Open models, agent sandboxes, and GPUs from one vendor
- Permalink: https://theaiengineering.com/platforms/inference-providers#novita-ai

### NVIDIA NIM

NVIDIA NIM packages open and partner models with optimized inference engines into containers that teams self-host on NVIDIA GPUs, and offers hosted NIM APIs on build.nvidia.com for prototyping.

- Company: NVIDIA
- Website: https://www.nvidia.com/en-us/ai-data-science/products/nim-microservices/
- Open source: No
- Deployment: Self-hosted, API
- Best for: Standardized model containers for NVIDIA GPU fleets
- Permalink: https://theaiengineering.com/platforms/inference-providers#nvidia-nim

### OCI Enterprise AI (formerly OCI Generative AI)

OCI Enterprise AI is Oracle Cloud's managed service for models and agents, offering models from Cohere, Meta, xAI, OpenAI, and others, plus Dedicated AI Clusters, guardrails, observability, and access control.

The former OCI Generative AI service page now redirects to OCI Enterprise AI, which keeps the Dedicated AI Clusters feature.

- Company: Oracle
- Website: https://www.oracle.com/artificial-intelligence/enterprise-ai/
- Open source: No
- Deployment: Managed cloud, API
- Best for: Teams running workloads on Oracle Cloud
- Permalink: https://theaiengineering.com/platforms/inference-providers#oci-enterprise-ai

### OVHcloud AI Endpoints

OVHcloud AI Endpoints is a serverless inference service from the French cloud provider OVHcloud that serves open-weight language, speech, and vision models through an OpenAI-compatible API on European infrastructure.

- Company: OVHcloud
- Website: https://www.ovhcloud.com/en/public-cloud/ai-endpoints/
- Headquarters: Roubaix, France
- Founded: 1999
- Open source: No
- Deployment: Managed cloud, API
- Best for: Sovereign European inference for open models
- Permalink: https://theaiengineering.com/platforms/inference-providers#ovhcloud-ai-endpoints

### Parasail

Parasail is an inference cloud for open models that offers serverless endpoints, dedicated deployments, and elastic endpoints that give eligible models dedicated GPU performance while still billing per token.

- Website: https://www.parasail.io
- Open source: No
- Deployment: Managed cloud, API
- Best for: Dedicated performance without GPU-hour commitments
- Permalink: https://theaiengineering.com/platforms/inference-providers#parasail

### Replicate

Replicate runs open-source and community models, many of them for images, video, and audio, through a single API, and packages custom models as containers with Cog, its open-source tool.

Cloudflare acquired Replicate, which continues to operate under its own brand while its model catalog comes to Cloudflare Workers AI.

- Website: https://replicate.com
- GitHub: https://github.com/replicate
- Open source: No
- Deployment: Managed cloud, API
- Languages: Python, TypeScript
- Best for: Calling image, video, and audio models through one API
- Permalink: https://theaiengineering.com/platforms/inference-providers#replicate

### SambaNova

SambaNova runs SambaCloud, an inference service for open models on the company's own reconfigurable dataflow unit (RDU) chips, and sells SambaStack and SambaRack systems for running the same stack in customers' data centers.

- Company: SambaNova Systems
- Website: https://sambanova.ai
- GitHub: https://github.com/sambanova
- Founded: 2017
- Open source: No
- Deployment: Managed cloud, API, Self-hosted
- Best for: Open-model inference on non-GPU hardware, hosted or on premises
- Permalink: https://theaiengineering.com/platforms/inference-providers#sambanova

### Scaleway Generative APIs

Scaleway Generative APIs is a serverless, OpenAI-compatible inference service from the French cloud provider Scaleway that serves open models with data kept in Europe and a Batch API for non-realtime jobs.

- Company: Scaleway
- Website: https://www.scaleway.com/en/generative-apis/
- Headquarters: Paris, France
- Open source: No
- Deployment: Managed cloud, API
- Best for: EU-hosted serverless inference
- Permalink: https://theaiengineering.com/platforms/inference-providers#scaleway-generative-apis

### SiliconFlow

SiliconFlow is an AI infrastructure platform that serves open language and multimodal models through serverless APIs, dedicated endpoints, and reserved GPUs, with fine-tuning and one-click deployment of customized models.

- Website: https://www.siliconflow.com
- Open source: No
- Deployment: Managed cloud, API
- Best for: Serverless and dedicated access to open models
- Permalink: https://theaiengineering.com/platforms/inference-providers#siliconflow

### Together AI

Together AI runs open-weight models on serverless and dedicated inference, and also offers fine-tuning, batch inference, and NVIDIA GPU clusters that range from self-serve instant clusters to large reserved deployments.

- Website: https://www.together.ai
- GitHub: https://github.com/togethercomputer
- Headquarters: San Francisco, United States
- Open source: No
- Deployment: Managed cloud, API
- Languages: Python, TypeScript
- Best for: Running and fine-tuning open models without managing GPUs
- Permalink: https://theaiengineering.com/platforms/inference-providers#together-ai

## How to choose

1. **Model catalog**: Which open-weight and proprietary models are available, and how quickly new releases appear.
2. **Serving mode**: Serverless per-token endpoints, dedicated deployments with reserved capacity, or batch processing for offline jobs.
3. **Performance**: Time to first token and output tokens per second for your models and prompt lengths, measured on your own traffic.
4. **Customization**: Support for fine-tuning, LoRA adapters, and deploying custom weights or containers.
5. **Compliance and regions**: Data retention terms, regional hosting, and the certifications your organization requires.
6. **API compatibility**: Whether the API follows the OpenAI format, which determines how easily a gateway or SDK can switch providers.

## Frequently asked questions

### What is an inference provider?

An inference provider runs AI models on its own infrastructure and charges for access through an API, usually per input and output token or per hour of reserved capacity. Developers send requests and receive generated output without managing GPUs, drivers, or serving software. Together AI, Fireworks AI, Groq, and the AI services of AWS, Google Cloud, and Microsoft Azure are examples.

### Serverless vs dedicated inference: what is the difference?

Serverless inference shares capacity across customers and bills per token, which suits variable or low traffic and requires no capacity planning. Dedicated inference reserves GPUs for one customer and bills by time, which gives more predictable latency, supports custom or fine-tuned weights, and can lower unit cost at steady high volume. Many providers offer both modes.

### What are Hugging Face Inference Providers?

Hugging Face Inference Providers is a Hugging Face service that routes requests for Hub models to partner inference companies, including Cerebras, Groq, Together AI, and Fireworks, through one API and one Hugging Face token. It offers an OpenAI-compatible chat endpoint and picks the provider with the highest throughput by default, or picks by price or by name when the request asks for it.

### What is the fastest LLM inference provider?

No provider is fastest for every model and prompt. Groq, Cerebras, and SambaNova run models on their own chips and target high output speed for the models they support, while GPU providers tune engines such as vLLM, SGLang, and TensorRT-LLM. Speed also changes with model size, context length, and load, so measure time to first token and tokens per second on your own traffic.

### What is an OpenAI-compatible API?

An OpenAI-compatible API accepts the same request and response format as OpenAI's Chat Completions endpoint, so code written for the OpenAI SDK can call another provider by changing the base URL, API key, and model name. Most inference providers and serving engines offer one. Compatibility usually covers chat, streaming, and tool calling, while structured outputs, reasoning parameters, and newer endpoints vary by provider.

### LLM inference vs training: what is the difference?

Training adjusts a model's weights by processing large datasets over many GPU hours, and it happens before deployment. Inference uses the finished weights to generate output for each request, so its cost scales with traffic and its performance is measured in latency and throughput. Inference providers handle the second stage, and several also offer fine-tuning.

## Related guides

- [LLM inference optimization: a practical guide for engineers](https://theaiengineering.com/articles/llm-inference-optimization): LLM inference optimization guide: prefill vs decode, TTFT, KV cache and PagedAttention, continuous batching, quantization, speculative decoding, and GPUs.
- [What is an AI gateway? How it works and when you need one](https://theaiengineering.com/articles/what-is-an-ai-gateway): An AI gateway is a proxy between your apps and model providers. Learn how it works, its core features, AI gateway vs API gateway, and open source options.
- [LLM routing: strategies, fallbacks, and learned routers](https://theaiengineering.com/articles/llm-routing): LLM routing guide: rule-based and cost-aware strategies, fallbacks, load balancing, semantic and learned routers like RouteLLM, router evals, and pitfalls.

Product details change quickly; check each vendor's site for current features, licensing and pricing.

## Cite this page

The AI Engineering Editors. "Inference providers." The AI Engineering. https://theaiengineering.com/platforms/inference-providers

- Title: Inference providers
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- URL: https://theaiengineering.com/platforms/inference-providers
- Markdown: https://theaiengineering.com/platforms/inference-providers.md
