Managing enterprise AI at scale: Hosting, deployment patterns, and Day 2 operations

In our previous article, Enterprise AI model selection: Balancing performance, privacy, and operational fit, we described the 4 layers that make up an effective enterprise AI architecture. In this article, we'll talk about who operates each layer, where workloads run (managed APIs, self-hosting, or hybrid), and how Red Hat AI Enterprise provides all 4 layers as a tightly integrated production AI system. We'll also cover some deployment implications for RAG, fine-tuning, and agents, and some considerations around Day 2 operations.Managed APIs versus self-hostingWhen you're building out your AI infrastructure, one fundamental choice is whether to use managed AI models via APIs or host the models yourself. That decision directly impacts the architecture we've already discussed, and affects cost structure, data privacy, operational complexity, and how quickly you can scale.Managed AI modelsManaged AI models are accessed through APIs from providers such as OpenAI, Anthropic, and Google. Requests go to their cloud infrastructure and responses come back, usually billed per token or per request. The provider operates infrastructure layers 1 through 3. Your team chooses region, model name, tier, and quotas, but doesn't manage GPUs, weight storage, inference engines, or serving. The integration point is the vendor endpoint and API contract.That arrangement avoids capital expense and Day 2 operations for serving, but per-token cost can rise quickly at scale, and prompts and outputs typically leave your network unless the vendor offers private or regional terms. Availability, pricing, throttling, and model updates remain under provider control.Self-hosted modelsSelf-hosted models run on infrastructure you control, on premise or in a private cloud. Your team operates layers 1 through 3, or delegates them to a managed Kubernetes or AI platform on your estate. Artifacts are downloaded or mirrored, inference engines are deployed behind a serving layer, and monitoring, scaling, patching, and model promotion stay in-house. Data and prompts can remain inside your boundary, which helps with regulated workloads. Infrastructure cost is often more predictable at high volume, but the trade-off is GPU capacity, platform skills, and incident response.Many organizations use a hybrid strategy: managed APIs for lower-risk or exploratory workloads, and self-hosted models where data must stay in their environment. A hybrid design works best when each workload has an explicit hosting choice and a clear traffic path, such as egress to a vendor API versus calls to an internal inference route, private links where needed, and a decision about whether the RAG retrieval store and embeddings stay on your side even when inference uses a managed model.Running the AI stack with Red Hat AI EnterpriseWhen you self-host—whether for compliance, latency, cost, or sovereignty—you still need a platform that implements the 4 layer system reliably. For organizations that have already standardized on Red Hat, Red Hat AI Enterprise is a fully-integrated AI platform that provides a production-grade way to stand up layers 1 through 4 on infrastructure you already have, rather than having to build something new.In Red Hat AI Enterprise, Red Hat OpenShift is the Kubernetes substrate for GPU scheduling and multi-team isolation, and Red Hat OpenShift AI is where model lifecycle, serving, and agentic integration land. Inference capabilities for production are built into OpenShift AI, you don't need a separate serving product to get a callable model endpoint.Compute and hardwareRed Hat OpenShift solves operational problems that are hard to get right when self-hosting models: GPU quotas and isolation across teams, hybrid accelerator support, and AI components as standard container workloads. That includes sharing individual GPUs through partitioning or time-slicing, so smaller workloads don't each reserve a full device. OpenShift AI runs inference and serving, Model Context Protocol (MCP) servers, agent frameworks, RAG pipelines, and related services on the same cluster or connected clusters.Model storage and lifecycleOn OpenShift AI under Red Hat AI Enterprise, weight files and related artifacts typically live in durable storage the inference engine can pull from, commonly an Open Container Initiative (OCI) registry packaged as a ModelCar image (model files packaged like a container image), or another configured object-store location. A model registry sits on top of that to record versions, metadata, and promotion state so teams can register, track, and deploy approved builds rather than sharing informal folders.Separately, a centralized model catalog helps teams discover validated and optimized models and experiment in a controlled way before those artifacts are registered and served. Training Hub supports customization with private data, including fine-tuning and reinforcement-style workflows (training that improves behavior from preference or reward signals), so domain adaptation stays on a platform you govern.Inference servingThis is where Layer 3 of the AI infrastructure stack, inference and model serving, becomes concrete, and OpenShift AI supplies both. Its model serving deploys an inference engine behind routing, health checks, replicas, and autoscaling aligned to GPU capacity so applications get a stable, OpenAI-compatible endpoint.The inference engine in that path is typically vLLM; for demanding scale-out workloads, llm-d (a Kubernetes-native stack for distributed LLM inference) extends that pattern. Red Hat packages these together in Red Hat AI Inference. On OpenShift AI you don't deploy it separately, because the serving platform ships these engines as built-in runtimes; the standalone offering exists for estates that run inference outside OpenShift AI, such as on Red Hat Enterprise Linux (RHEL) or another Kubernetes platform.Together, model serving on OpenShift AI with the vLLM engine underneath is the usual boundary between AI infrastructure and your applications. Everything above that boundary, such as prompts, RAG orchestration, and agents, calls the model as a service.Integration, agents, and governanceFor connecting models to enterprise data and systems, Red Hat AI Enterprise covers both directions of the integration layer, layer 4 in our infrastructure stack, with OpenShift AI.On the systems-to-AI path, we made the case for an AI-aware front door on shared model endpoints. Under Red Hat AI Enterprise that front door is not an additional component to operate: OpenShift AI includes AI gateway capabilities with Red Hat Connectivity Link, so platform teams can set per-team access, quotas, and token budgets on the serving path, Models-as-a-Service (MaaS) style. Where inference runs outside OpenShift AI, Connectivity Link is available as its own gateway.On the AI-to-systems path, the portfolio emphasizes agentic workflows with governance. MCP is made operational through a catalog of curated servers you can deploy on-cluster, so agents can discover and call approved tools through a shared protocol. An MCP lifecycle operator deploys and manages those servers as workloads.For a single governed entry point across those servers, Connectivity Link's MCP gateway (currently in technology preview) centralizes authentication, routing, federated tool discovery, and tool-level access control, distinct from the Connectivity Link AI gateway that fronts model inference. Teams can also expose custom tools and OpenShift AI resources, such as approved models, workbenches (interactive development environments), and pipeline runs, through scaffolding and platform MCP servers. AI guardrails add monitoring, performance tracking, and drift detection (watching for shifts in model behavior or data over time) so models and outputs remain reliable in production.Application patterns in deploymentThe first article in this series, Enterprise AI model selection: Balancing performance, privacy, and operational fit, covers how RAG, fine-tuning, and agentic workflows align a model with your domain. In production, the question is less what those patterns are and more what else you must run and promote alongside the model.RAG is the clearest example. Inference may use a managed API or a self-hosted model, but organizations usually own the embedding pipeline and retrieval index, including ingest capacity, secured and backed-up storage, and access control on retrieved sources. End-to-end latency includes retrieval, not inference alone, and promotion checks typically cover retrieval quality as well as the final answer.Fine-tuning adds a different burden. Training usually runs as a burst on GPUs kept separate from inference unless demand is steady enough to share; teams govern the training data, approve checkpoints per environment, and once live, operate the tuned artifact like any other served model.Agents go further still. Beyond inference they need an orchestration runtime and governed access to your systems, typically an allowlist of approved APIs or MCP tools, with timeouts, audited invocations, and human approval or policy controls for high-impact actions. Before putting a system into live production, teams confirm that the allowlist holds and that permitted tools fail safely when they time out or error.Most production designs combine these patterns, for example an instruct-tuned model with RAG and a small tool set. That is why model identifiers, prompt templates, retrieval index versions, and agent tool configurations are usually pinned together and promoted as a bundle across development, test, and production. Rollback then undoes one unit of change instead of leaving you to guess which component moved.Day 2 operationsDay 2 operations are what keep a production AI system stable after it goes live. Like any enterprise service, the workload needs clear targets for latency, availability, and error rate on inference and on critical integration paths. Those targets shape capacity planning. Peak concurrency and token throughput matter more than average load, and self-hosted serving adds constraints that managed APIs hide, such as GPU memory limits, replica cold starts (the delay when a new replica loads the model into memory), and queues when demand exceeds what the fleet can serve.ObservabilityObservability is how teams know whether those targets are being met. Inference latency, error rates, token usage, and queue depth show whether serving itself is healthy. RAG adds retrieval hit rate and latency to that picture; agents add tool-call logs tied to a correlation ID (a shared identifier that links related events to one user or session). Cost broken down by application, team, and model makes the financial and capacity implications of that traffic visible rather than leaving them buried in a single provider bill.Failures and continuity planningFailures still happen, and they are often outside the model. Managed APIs can time out or throttle, GPU nodes can disappear, a bad rollout can regress quality, and an upstream tool an agent depends on can break. Teams that plan for those cases define degraded modes in advance, such as falling back to a smaller model, serving cached responses, running agents read-only without write tools, or temporarily disabling agent features. Continuity planning differs by hosting model. Self-hosted estates usually need registry replication and secondary clusters or regions; managed APIs shift uptime responsibility to the vendor, while application-level fallbacks and backups for indexes and configuration remain on your side.Security and governanceSecurity and governance extend the same operational discipline to who and what can act once the system is live. Agents typically run under service accounts rather than personal credentials, with least privilege on each tool and validation that limits injection and abuse, including prompt injection, where crafted input tries to override instructions or exfiltrate data. An approved catalog of models and tools, risk tiers by use case such as internal copilots versus customer-facing features versus automated remediation, and review gates for high-impact automation keep those capabilities aligned with existing change management and, where stakes require it, human approval.Final thoughtsChoosing a capable model is only the first architectural move. What matters next is whether that choice survives contact with your infrastructure, hosting model, and the teams who will operate it. A model that fits on paper may be impractical if your estate cannot store, serve, or integrate it. A managed API may be the right start for one workload and the wrong long-term home for another where data must stay in your boundary. Hybrid designs are common; what matters is being explicit about traffic paths, data boundaries, and who owns each layer.Treat production AI as a system, not a model endpoint. Integration and steady-state operations usually carry more complexity than model selection. Promote model identifiers, prompts, retrieval indexes, and tool configurations as a bundle, and plan observability, failure modes, and governance before you put your system into production.Learn moreRed Hat AI EnterpriseNo-cost Red Hat AI Enterprise trial