Browse notes
Deployment Models
When choosing how to use a model, two questions clarify most of the map: where inference runs and which parts of the system you operate.
| Option | Example | What you manage |
|---|---|---|
| SaaS | ChatGPT or Claude | You use the application; the provider manages the model, runtime, and infrastructure |
| Hosted API | Direct API from OpenAI, Anthropic, or another provider | You build the application; the provider serves the model |
| Cloud API | Azure AI Foundry or Amazon Bedrock | You build the application and configure cloud resources; the platform manages inference |
| Gateway | OpenRouter or an internal gateway | Your application talks to a layer that routes to models or providers |
| Self-hosted | vLLM on your infrastructure | You manage runtime, model, capacity, and infrastructure |
| Local | LM Studio on a PC | You manage the model and runtime directly on your hardware |
Finished product: SaaS
With SaaS, you use an application that is already built. The provider decides how the model is served, which features and tools are exposed, and much of how requests are assembled.
That gives up some control in exchange for not having to build the client or operate inference infrastructure. For interactive work, it is often the option with the fewest moving parts to manage.
Managed inference: APIs and gateways
With a direct API, your application decides what input to send, how to preserve state, and what to do with the response; the provider loads the model and operates the runtime and hardware. A cloud platform such as Azure AI Foundry or Amazon Bedrock adds its own identity, quotas, regions, and configuration around managed inference.
A gateway sits in front of one or more providers. It can centralize credentials, logging, limits, or routing and present a common interface to the application. OpenRouter is one example of this layer.
A shared interface reduces client changes, while model capabilities can still differ in tools, reasoning, usage, and supported parameters. Model APIs covers the integration at that level.
Your own inference: self-hosted and local
With self-hosted inference, you operate the runtime and infrastructure where the model is loaded. That gives control over the model and configuration while making capacity, memory, concurrency, upgrades, and availability your responsibility.
Local inference brings the same responsibility onto a nearby machine, usually a workstation or laptop. LM Studio or llama.cpp can load the model and expose an API for other applications. Local Inference covers the quantization, context, VRAM, and offload trade-offs that become important here.
Choosing an option
For a practical decision, I separate four concerns:
- Control: how much you need to choose the model, parameters, context, and tools.
- Operations: how much runtime, capacity, and availability you want to manage.
- Data: what restrictions apply to where information may be processed or stored.
- Cost and usage: whether subscription access, API consumption, or your own infrastructure fits the expected workload.
These options can coexist. The same team may use SaaS for interactive work, an API for an application, and a local model for testing without turning them into one homogeneous deployment.