# Chapter 87 — Secure AI Platform Deployment & Runtime Delivery

— Secure AI Platform Deployment & Runtime Delivery: Kubernetes Security, Workload Identity, Admission Control, Image Verification, Progressive Delivery, Runtime Policy Enforcement & Production Integrity 87.1 Introduction A secure build pipeline is only one part of the software supply chain. After an application, model, container, configuration, and policy have been built and verified, they must be delivered into a production runtime. That deployment process creates another major security boundary. A secure AI deployment architecture must answer: Which artifact is allowed to run? Who is allowed to deploy it? Where can it run? Which identity does the workload use? What permissions does it receive? Can an untrusted image enter the cluster? Can a compromised workload access another workload? Can a deployment bypass security policies? How is a bad release stopped? How is a release safely rolled back? The fundamental principle is: Deployment should be a controlled transition from a verified artifact to a verified runtime state. 87.2 Deployment Security Model A simplified secure delivery chain is: Verified Artifact ↓ Release Policy ↓ Deployment Authorization ↓ Admission Validation ↓ Image / Artifact Verification ↓ Runtime Configuration Validation ↓ Workload Creation ↓ Runtime Security Controls ↓ Health Verification ↓ Progressive Traffic ↓ Production The deployment system should not simply execute: deploy latest Instead, it should establish exactly what is being deployed and why it is authorized. 87.3 Kubernetes as a Security Boundary For AI platforms using Kubernetes, the cluster becomes an important security boundary. Kubernetes may manage: application services API services AI inference workers model-serving systems background workers media-processing jobs RAG services databases or supporting services monitoring components queues scheduled workloads Because many components share the same cluster, isolation becomes critical. A secure design should distinguish: Cluster ├── Application Namespace ├── AI Inference Namespace ├── Worker Namespace ├── Media Processing Namespace ├── Monitoring Namespace └── Platform Services Namespace Namespaces alone are not complete isolation. They should be combined with identity, authorization, network controls, resource policies, and workload restrictions. 87.4 Workload Identity Every production workload should have a clear identity. Instead of relying on shared credentials, a workload can receive an identity associated with its specific service account or workload. Conceptually: AI Worker ↓ Workload Identity ↓ Authorized Cloud Resources This allows policies such as: Image Processor → Object Storage: Read/Write Approved Bucket RAG Service → Vector Database: Query Only Billing Service → Payment Data: Restricted Access Monitoring Service → Metrics: Read The identity should correspond to the workload's actual responsibilities. 87.5 Service Account Separation A common mistake is assigning one powerful service identity to many workloads. For example: Application AI Worker Media Worker Admin API Background Worker ↓ Shared Powerful Identity If one workload is compromised, the attacker may inherit permissions belonging to all of them. A safer model is: Application → Identity A AI Worker → Identity B Media Worker → Identity C Admin API → Identity D This reduces privilege concentration. 87.6 Short-Lived Credentials Long-lived credentials increase the impact of compromise. Where supported, deployment and runtime systems should prefer: workload identity short-lived tokens automatic credential rotation narrowly scoped permissions The goal is to avoid situations where a stolen credential remains useful indefinitely. 87.7 Image Verification A production cluster should verify that container images originate from an approved source. A conceptual policy is: IF image is unsigned → reject IF image provenance is invalid → reject IF image source is unapproved → reject IF critical security policy fails → reject This prevents arbitrary container images from entering production. 87.8 Admission Control Admission control provides a security checkpoint before workloads are accepted by the cluster. A deployment request can be evaluated against policies such as: approved image registry image signature provenance required security context resource limits namespace restrictions privileged-mode restrictions host-access restrictions required labels required network policies allowed service accounts Conceptually: Deployment Request ↓ Admission Policy ↓ ┌─────┴─────┐ │ │ PASS DENY │ │ ▼ ▼ Create Reject Workload Request Admission policies should fail closed for critical security requirements. 87.9 Preventing Privileged Workloads A workload should not receive powerful host-level privileges unless there is a documented and justified requirement. Security-sensitive restrictions can address: privileged execution host filesystem access host networking unnecessary Linux capabilities unsafe device access root execution unrestricted privilege escalation The principle is simple: Give workloads the minimum runtime authority necessary to perform their function. 87.10 Non-Root Execution Where technically practical, applications should run without unnecessary root privileges. For example: Container ↓ Application User ↓ Limited Permissions If the application is compromised, non-root execution can reduce the attacker's ability to affect the underlying environment. However, non-root execution should be tested carefully because some applications or media-processing libraries may require specific filesystem or operating-system permissions. 87.11 Runtime Filesystem Security AI workloads often create temporary data. Examples include: uploaded images generated video frames audio fragments model caches temporary documents extracted archives Temporary storage should have appropriate: permissions size limits lifecycle controls isolation cleanup A media-processing worker should not automatically have unrestricted access to the host filesystem. 87.12 Network Policies Workloads should not automatically communicate with every other workload. For example: API ↓ AI Gateway ↓ Inference Service ↓ Model Worker The model worker may not need direct access to: Billing Database Admin API User Authentication Database Network policies can restrict communication to required paths. A useful model is: Default deny, explicitly allow required communication. 87.13 Service-to-Service Authentication Network reachability should not automatically mean authorization. Even when two services can communicate, they should establish identity and authorization where appropriate. Conceptually: Service A ↓ Authenticated Request ↓ Service B ↓ Authorization Check This creates defense in depth. 87.14 Runtime Configuration Security Deployment systems often inject configuration into workloads. Configuration may include: model selection API endpoints feature flags resource limits safety policies RAG settings processing limits timeout values Sensitive credentials should remain separate from ordinary configuration. Production configuration should be: versioned validated authorized auditable environment-specific protected against unauthorized modification 87.15 Secrets During Deployment Deployment pipelines should avoid embedding secrets inside: container images source code deployment manifests public logs static configuration files Instead, secrets should be retrieved through controlled secret-management mechanisms at runtime or deployment time. A strong separation is: Artifact ↓ No Production Secrets Runtime ↓ Authorized Secret Retrieval This means the same artifact can often be promoted through environments without embedding environment-specific credentials. 87.16 Resource Limits AI workloads can consume substantial resources. Examples include: CPU memory GPU storage network bandwidth Without appropriate limits, one workload can consume resources needed by others. Production deployment policies should define appropriate: requests limits quotas concurrency limits job limits queue limits timeout limits This is both a reliability and security control. 87.17 GPU Security AI inference and media processing may use GPUs. GPU workloads introduce additional concerns around: resource allocation isolation driver compatibility memory consumption scheduling noisy-neighbor effects privileged runtime components GPU-enabled workloads should receive only the resources they require. A GPU worker should not automatically gain unrelated administrative permissions. 87.18 Namespace and Tenant Boundaries Multi-tenant AI platforms require especially careful deployment isolation. A tenant may have: application data uploaded files generated media RAG documents embeddings configuration usage information A deployment mistake must not allow one tenant's workload to access another tenant's data. Tenant boundaries should therefore exist at multiple layers: Identity ↓ Authorization ↓ Application ↓ Database ↓ Object Storage ↓ Vector Store ↓ Cache ↓ Network ↓ Runtime This is defense in depth. 87.19 Progressive Delivery Production deployment does not need to expose a new release to every user immediately. Progressive delivery can gradually increase exposure. For example: New Release ↓ 1% Traffic ↓ 5% ↓ 25% ↓ 50% ↓ 100% At each stage, the platform can evaluate: errors latency resource usage AI quality safety behavior user-impact indicators unexpected output patterns If the release behaves incorrectly, promotion can stop. 87.20 Canary Deployment Canary deployment sends a small amount of traffic to a new version. Example: ┌── Old Version ── 99% Users ──────────────┤ └── New Version ── 1% The new version can then be evaluated before broader rollout. For AI systems, canary evaluation should include behavioral metrics rather than only HTTP health. For example: Canary ├── API Error Rate ├── Latency ├── Cost ├── Hallucination Metrics ├── Safety Metrics └── Task Quality 87.21 Blue-Green Deployment Another strategy is blue-green deployment. Conceptually: Blue = Current Production Green = New Release Traffic initially goes to Blue. After Green passes validation: Traffic ↓ Green If a problem appears: Traffic ↓ Blue This can simplify rollback when sufficient infrastructure capacity exists. 87.22 Rolling Deployment A rolling deployment replaces workloads gradually. For example: Old Old Old Old ↓ New Old Old Old ↓ New New Old Old ↓ New New New Old ↓ New New New New This reduces sudden capacity changes. However, compatibility between old and new versions must be considered. 87.23 Database Compatibility During Deployment Application deployments often interact with database migrations. A new application may expect a schema that the old application cannot understand. A safer strategy is often: Expand ↓ Deploy Compatible Application ↓ Migrate Data ↓ Remove Old Schema Elements This is safer than making an irreversible database change before the application is ready. 87.24 Health Checks Deployment systems need reliable health checks. Common categories include: Startup check Determines whether the application initialized successfully. Readiness check Determines whether the service is ready to receive traffic. Liveness check Helps identify a process that is no longer functioning correctly. Health checks should test meaningful conditions without creating unnecessary load. 87.25 AI-Specific Health Checks AI services need more than process-level health checks. A model server may be running while the AI functionality is broken. Additional checks may include: model loaded successfully expected model version inference endpoint operational tokenizer available vector database reachable policy engine reachable required tools available safety middleware functioning A successful process status should not automatically imply successful AI service behavior. 87.26 Runtime Policy Enforcement Security policies should remain active after deployment. Examples: Maximum Request Size Maximum File Size Maximum Generation Duration Maximum Token Budget Maximum Tool Calls Maximum Concurrent Jobs Maximum Model Cost Runtime policies provide protection even when application-level assumptions fail. 87.27 Runtime AI Safety Controls An AI runtime may apply controls around: prompt validation input filtering output validation content policy tool authorization data-access policy rate limiting abuse detection model routing The deployment should ensure that required safety services cannot silently disappear. For example: Request ↓ Authentication ↓ Authorization ↓ Safety Policy ↓ Model ↓ Output Validation ↓ User A deployment that accidentally bypasses the safety layer should be rejected. 87.28 Runtime Policy Drift Deployment security is not finished after release. Runtime configuration can change later. Therefore, the platform should monitor: Desired State vs. Actual State Examples of drift include: unexpected image unexpected service account changed network policy modified configuration missing security policy unexpected exposed service Drift detection helps identify unauthorized or accidental changes. 87.29 Deployment Integrity Monitoring The platform should continuously verify important deployment properties. Useful signals include: running image digest deployed version configuration version policy version workload identity resource configuration exposed endpoints deployment timestamp deployment owner The goal is to make the runtime state observable and verifiable. 87.30 Runtime Detection Production monitoring should detect abnormal behavior. Potential signals include: unexpected outbound traffic unusual resource consumption repeated crashes abnormal authentication failures unexpected privilege use suspicious file activity unusual model invocation patterns abnormal generation volume Detection should connect to the platform's broader incident-response process. 87.31 Automatic Rollback Automatic rollback can be useful when a release violates predefined conditions. For example: IF error rate > threshold → stop rollout IF critical health check fails → rollback IF security policy violation occurs → stop deployment IF severe AI safety regression occurs → stop promotion Thresholds should be carefully designed to avoid both missed incidents and unnecessary rollbacks. 87.32 Deployment Freeze During major incidents, a deployment freeze can prevent additional changes from increasing uncertainty. A freeze can apply to: production deployments model promotion configuration changes infrastructure modifications nonessential releases Emergency changes can use a separate controlled process. 87.33 Secure Production Access Production deployment should not require broad interactive access for ordinary developers. A preferred model is: Developer ↓ Approved Change ↓ CI/CD ↓ Controlled Deployment rather than: Developer ↓ Production Administrator ↓ Manual Modification Interactive production access should be restricted and audited. 87.34 Break-Glass Access Emergency administrative access may still be necessary. A break-glass mechanism should be: strongly authenticated tightly restricted time-limited logged monitored reviewed afterward Emergency access should not become the normal deployment mechanism. 87.35 Deployment Audit Trail Every production deployment should have a traceable identity. A useful deployment record includes: Release ID Source Revision Artifact Digest Image Digest Model Version Configuration Version Policy Version Deployment Identity Approval Timestamp Environment Rollout Strategy Result Rollback Information This creates a strong forensic trail. 87.36 Secure Deployment Architecture A mature deployment architecture can look like: Verified Artifact │ ▼ ┌────────────────────┐ │ Deployment Control │ └─────────┬──────────┘ │ ▼ ┌────────────────────┐ │ Policy / Admission │ └─────────┬──────────┘ │ ▼ ┌────────────────────┐ │ Artifact Verification│ └─────────┬──────────┘ │ ▼ ┌────────────────────┐ │ Kubernetes / Runtime│ └─────────┬──────────┘ │ ┌─────────────┼─────────────┐ ▼ ▼ ▼ Identity Network Policy Resource Policy │ │ │ └─────────────┼─────────────┘ ▼ Running Workload │ ▼ Health + Monitoring │ ▼ Progressive Traffic │ ▼ Production The deployment system acts as the bridge between trusted artifacts and trusted runtime state. 87.37 Production Deployment Checklist Artifact [ ] Approved artifact [ ] Verified digest [ ] Valid signature [ ] Valid provenance [ ] Approved model version [ ] Required AI evaluation passed Kubernetes / Runtime [ ] Approved namespace [ ] Restricted service account [ ] Least-privilege workload identity [ ] Non-root execution where practical [ ] No unnecessary privileged mode [ ] Resource requests and limits [ ] Appropriate filesystem restrictions Network [ ] Network policies configured [ ] Required service paths allowed [ ] Unnecessary paths blocked [ ] External communication controlled Admission [ ] Image policy [ ] Signature verification [ ] Security-context policy [ ] Resource policy [ ] Namespace policy [ ] Required metadata Secrets [ ] No embedded secrets [ ] Short-lived credentials [ ] Runtime secret retrieval [ ] Environment separation Deployment [ ] Approval recorded [ ] Health checks configured [ ] Progressive rollout where appropriate [ ] Rollback available [ ] Database compatibility verified Monitoring [ ] Deployment telemetry [ ] Runtime telemetry [ ] Security monitoring [ ] AI quality monitoring [ ] Safety monitoring [ ] Drift detection 87.38 Deployment Security Maturity Model Level 1 — Basic manual deployment basic health checks basic access control Level 2 — Controlled CI/CD deployment environment separation approvals resource limits basic runtime policies Level 3 — Verified signed images admission policies workload identities network policies progressive delivery Level 4 — Advanced automated artifact verification automated rollback continuous drift detection AI-specific deployment evaluation strong workload isolation Level 5 — Resilient continuously verified runtime state strong identity boundaries automated containment independently auditable deployment lineage tested emergency and recovery procedures 87.39 Final Security Principle Secure deployment is not simply the act of starting a container. It is the controlled transformation of: Verified Artifact into: Verified Runtime State A production AI platform should continuously establish that: the artifact is trusted the image is authentic the workload has the correct identity permissions are minimal network access is restricted resources are controlled secrets are protected policies are enforced tenant boundaries remain intact rollout behavior is observable runtime state matches the intended state failures can trigger containment or rollback The strongest architecture therefore follows this principle: Do not trust deployment merely because an authorized pipeline initiated it. Verify the artifact, verify the deployment request, verify the workload identity, enforce runtime policy, monitor the resulting state, and retain the ability to stop or reverse the release. Chapter 87 Summary Secure AI deployment requires: Kubernetes/runtime isolation workload identity least-privilege service accounts short-lived credentials image verification admission control secure workload configuration non-root execution where practical network segmentation resource limits GPU/resource isolation tenant isolation progressive delivery canary deployment blue-green or rolling strategies AI-specific health checks runtime policy enforcement drift detection deployment monitoring automatic or controlled rollback audited production access break-glass procedures complete deployment lineage disaster and recovery readiness Core principle: Production integrity depends on continuously verifying both what was deployed and how that deployed workload behaves after deployment.