DevSecOps for AI keeps the core idea of classic DevSecOps: security checks run automatically at every stage of delivery, owned by the team that ships, not bolted on as a review at the end. What changes is the inventory of things that can be compromised. A secure AI pipeline treats prompts, model artefacts, datasets, vector indexes and agent tool permissions as release artefacts, with provenance, scanning, review and signing, in the same pipeline that already scans code and containers. This guide walks through each stage, the controls that belong there, and how a team runs it.
What is new compared with classic DevSecOps
If your team already runs SAST, dependency scanning and image scanning, you have most of the foundation. Enterprise AI applications add five kinds of artefact that classic pipelines never had to see:
- Prompts as code. System prompts, examples and tool descriptions decide behaviour. A careless edit can remove a refusal rule or widen what an agent attempts.
- Model and dataset artefacts. Open-weight models, adapters, embedding models and datasets are large files pulled from hubs and buckets. Some model formats can execute code when loaded.
- Third-party model APIs. Part of your system runs on a provider's infrastructure and changes on their schedule. Pin what you can (model identifiers, region, retention settings) and detect what you cannot.
- Vector stores. An index built from internal documents is a copy of sensitive data in a new shape. It can leak through retrieval and be poisoned through ingestion.
- Agents with tool permissions. An agent's blast radius is defined by its credentials and tool list. Those are configuration, and configuration can be reviewed and tested before release.
The runtime threats themselves (prompt injection, data disclosure, excessive agency, poisoning) are covered in our guide to AI security for enterprises. This article is about the delivery side. You will also hear MLSecOps for the model-and-data half of this work; in practice it is most effective as one pipeline, not two.
The secure AI pipeline at a glance
General release mechanics for LLM applications (eval gates, canaries, rollback) are in CI/CD for AI applications. Here is the same flow with security controls marked.
PR: code + prompts + config + IaC |-- SAST, secrets scan, prompt review |-- SCA on lockfiles, IaC scan v Artefact intake |-- models: source, checksum, format, | licence -> model registry |-- data: lineage, PII scan -> index v Build |-- container image scan + SBOM |-- sign image, attest provenance v AI security tests (CI) |-- injection suite, jailbreak set, | tool-misuse cases, eval gate v Deploy |-- policy-as-code admission: | signed? scanned? approved? v Runtime |-- guardrails, vault secrets, | least-privilege agents, monitoring '-- findings feed back into test sets
The last line matters most. Every incident, red-team finding or blocked attack should become a test case on the next pull request. That loop is what turns a set of scanners into a security programme.
Code: SAST and secrets scanning
Semgrep, CodeQL and similar tools work on AI code as they do elsewhere. Add custom rules for AI-specific patterns:
- Model output passed into
eval, a shell, a SQL string or an HTML template without validation (insecure output handling). - User input concatenated into system prompts instead of a delimited user section.
- Tool functions accepting a raw URL or raw query where a typed schema or allow-list belongs.
- Full prompts and responses, often containing customer data, written to general-purpose logs.
Secrets scanning (gitleaks or your Git platform's native scanning) needs extra attention: provider API keys turn up in notebooks, sample .env files, eval scripts and pasted traces. Scan full history on adoption, block on push, and include notebook output cells. Treat a leaked provider key as a live incident: revoke first, investigate second.
Dependencies: SCA and lockfiles
Orchestration frameworks, SDKs, vector store clients and MCP servers release often and pull deep transitive trees.
- Hashed lockfiles (uv, Poetry or pip-tools) so builds install exactly what was reviewed; fail on a stale lockfile.
- Software composition analysis on every pull request with an agreed severity threshold, plus a scheduled scan of the main branch for new advisories.
- A second approver for new direct dependencies, because typosquatted names around popular AI libraries are a known pattern.
- MCP servers and agent plugins as dependencies. They run with the credentials you give them: pin versions, record them in the SBOM, review them like any third-party code touching production data.
Models: provenance, integrity and a registry
This is the stage most teams are missing. Whether you self-host an open-weight model, fine-tune one or run an embedding model locally, model files should enter through one controlled path.
- Provenance. Record publisher, repository, exact revision and approver. Pull only from approved sources and mirror approved models into your own storage, so production never pulls from the public internet.
- Checksums verified at intake and again at deployment.
- Safe serialisation formats. Pickle-based formats can execute arbitrary code on load. Prefer tensor-only formats such as safetensors, and scan any pickle-based file before it nears a production host.
- Licence review. Open-weight licences differ on commercial use and restrictions. Review at intake and record the outcome against the model.
- A model registry such as MLflow, Amazon SageMaker Model Registry, Azure Machine Learning or Vertex AI Model Registry. Deployments reference a registry version, never a file path.
For hosted models on Amazon Bedrock or Azure OpenAI, the equivalent control is an approved-model list in configuration (identifiers, regions, settings), checked in CI and enforced by cloud policy.
Data: lineage and PII scanning
- Lineage. For every index or dataset version, record the sources, extraction run and access labels. When a document must be removed, lineage tells you which indexes to rebuild.
- PII scanning before embedding. Decide per source whether to redact, mask, exclude or keep with stricter labels. Removing PII from a live vector index is far harder than stopping it at ingestion.
- Ingestion as untrusted input. Documents can carry hidden instructions; flag them and keep examples in the injection suite.
- Access labels on chunks so retrieval filters by user identity, with that filter tested in CI.
For teams in India, this evidence also supports data-protection obligations; see our sibling article on India's DPDP Act for AI teams.
Prompts and configuration: review and versioning
Keep prompts, tool definitions, guardrail settings and model parameters in the repository, not in an admin screen. Then:
- Use CODEOWNERS so system-prompt and tool-definition changes need the owning team and, for regulated flows, a security reviewer.
- Review prompt diffs. A deleted line such as "never reveal another customer's account details" should be visible and questioned.
- Version prompts with the release so any response traces to the exact prompt, model and configuration.
- Lint configuration: no secrets in prompts, no tool without a matching permission entry, no model outside the approved list.
Containers and infrastructure as code
AI images are large, bundling ML libraries and sometimes CUDA runtimes, so they carry more vulnerable packages. Use minimal base images, scan with Trivy or Grype, and generate an SBOM (for example with Syft) per build that also lists model and MCP server versions. Image structure is covered in Docker for AI applications.
Scan Terraform with Checkov, tfsec or Trivy, and add AI-specific policies: model-artefact buckets must be encrypted and versioned, vector databases must not be internet-reachable, model endpoints must use private networking, agent roles must not use wildcards. Module patterns are in Terraform for AI infrastructure.
AI-specific security tests in CI
Scanners find known-bad code and packages. They cannot tell you whether the assembled application, with its prompts, model and tools, can be talked into misbehaving. That needs behavioural tests against a preview environment:
- Prompt-injection suite. Direct attacks from users and indirect attacks planted in documents or emails the system retrieves. Each case asserts an outcome: tool not called, data not returned, response in scope.
- Jailbreak and red-team sets. A growing domain-specific corpus in the repository, plus periodic runs of open-source probing tools such as garak or PyRIT, or promptfoo's red-team features. Generic sets alone miss what matters to your business.
- Tool-misuse and permission tests. As a low-privilege test user, request an admin-only action or another user's records, and assert refusal at the tool or retrieval layer.
- Leakage tests. Seed canary strings in the system prompt and restricted documents; assert they never appear in output.
- Evaluation gate. Security cases are must-pass; quality metrics use agreed thresholds.
Because outputs vary, run each adversarial case several times and fail on any success, not on an average. Guardrail design is in AI guardrails explained; the point here is that guardrails get tested in the pipeline, not just configured.
Want to build this pipeline hands-on? Cloudsoft's DevSecOps training in Hyderabad covers SAST, SCA, image scanning, SBOMs, IaC scanning and policy enforcement in real CI pipelines, which you can then extend with the AI stages above.
Deployment: policy-as-code and signed artefacts
Deployment is where earlier evidence gets enforced. SLSA (Supply-chain Levels for Software Artifacts) gives the vocabulary: build on a trusted platform, produce provenance for each artefact, verify it before deploying.
- Sign images with Sigstore cosign or a cloud signing service, with attestations for the SBOM, scan results and eval report.
- Verify at admission with Kyverno or OPA Gatekeeper, rejecting unsigned, unscanned or out-of-pipeline images.
- Encode AI release rules as policy: model version approved in the registry, passing security eval attached, no production write tools without an approval step.
- Separate identities: the deploy identity cannot call models or read data; the eval identity cannot deploy.
Runtime: guardrails, secrets and least privilege
- Guardrails on input, retrieval, output and actions, using the configuration tested in CI.
- Secrets via a vault or cloud secrets manager, injected at the tool layer, rotated, never in prompts or images.
- Least privilege for agents: a workload identity per agent, scoped to its tools and data, with user-delegated tokens when acting for a person. See identity and access for AI agents.
- Monitoring of guardrail blocks, tool denials, unusual token spend and refusal-rate drift, routed to security operations as well as the app team.
Risk, control and pipeline stage
| Risk | Control | Pipeline stage |
|---|---|---|
| Leaked model provider key | Secrets scanning on push and history; vault at runtime | Code, runtime |
| Vulnerable or typosquatted package | Hashed lockfiles, SCA, new-dependency review | Dependencies |
| Malicious model file | Approved sources, checksums, safe formats, model scanning | Models |
| Model licence violation | Licence review recorded in the registry | Models |
| PII exposed through retrieval | PII scan before embedding, access labels, permission tests | Data, AI tests |
| Poisoned ingested documents | Lineage, ingestion flags, indirect-injection cases | Data, AI tests |
| Safety rule removed from a prompt | CODEOWNERS, prompt diffs, versioning | Prompts and config |
| Prompt injection and jailbreaks | Injection suite, red-team set, must-pass gate | AI tests, runtime |
| Insecure output handling | SAST rules on output sinks, schema validation | Code, runtime |
| Excessive agency | Permission lint, tool-misuse tests, scoped identities | Config, AI tests, runtime |
| Vulnerable base image | Image scanning, SBOM, minimal images | Containers |
| Public vector database | IaC scanning with AI-specific policies | IaC |
| Unreviewed artefact in production | Signing, provenance, admission policy | Deploy |
Frameworks to anchor the programme
- OWASP Top 10 for LLM Applications for application risks. Map each test to a category so coverage gaps are visible.
- NIST AI Risk Management Framework (AI RMF) for the organisational side. Its functions, Govern, Map, Measure and Manage, show risks were identified, tested and owned; pipeline evidence sits naturally under Measure and Manage.
- SLSA for build integrity and provenance, applied to images and increasingly to model artefacts.
How this evidence feeds approval boards and model inventories is covered in enterprise AI governance.
Illustrative example: a bank's contact-centre assistant
Consider a bank building an assistant for contact-centre staff. It answers policy questions from internal documents and looks up a customer's recent transactions through an internal API. Engineering sits in a GCC in Hyderabad; risk and compliance sit at head office. The first version was a notebook demo. Before go-live, the team rebuilt the delivery path:
- Prompts and tool definitions moved into the repository, with CODEOWNERS requiring application security approval for system-prompt or transaction-tool changes.
- Secrets scanning found a provider key in an old notebook output cell. It was revoked the same day.
- PII scanning at ingestion found customer names in complaint-handling examples, which were masked before embedding; each index build got a lineage record.
- A self-hosted embedding model came through intake: approved source, checksum, safetensors, licence review, registry entry.
- CI ran indirect-injection cases planted in policy documents, attempts to fetch another customer's transactions, and system-prompt canaries, each repeated and must-pass.
- Images were scanned, signed and admitted to EKS only with a passing eval attestation. The agent's role could call the read-only transaction endpoint and nothing else.
- Guardrail blocks and tool denials flowed to security operations. A phrasing that slipped past a filter early on became a regression case that week.
None of this is exotic. Every artefact had an owner, a check and a record, which is what made the risk team's sign-off possible. Building this kind of system end to end, security review included, is the Secure Banking AI Assistant project in Cloudsoft's FDE PRO program.
The team operating model
- Platform or DevOps team: shared pipeline templates, scanners, signing, admission policies and the model registry, so every AI team gets controls by default.
- Application security: rule sets, severity thresholds, the shared red-team corpus, and review of high-risk prompt and tool changes.
- AI product teams: domain test cases, prompt changes, agent permission requests and fixing their own findings.
- Data owners and privacy: approving sources for indexing and PII handling per source.
- Risk and compliance: consuming the evidence instead of running separate manual reviews.
A security champion in each AI team triages findings and adds test cases. Exceptions, such as shipping with a known medium finding, are time-boxed and recorded, not settled in a chat thread. Forward Deployed Engineers inside a customer often build the first version of this model, because they are the ones who must get the system through the customer's security review.
FAQ
What is DevSecOps for AI?
It is building automated security checks into every stage of delivering an AI application: code, dependencies, models, data, prompts, containers, infrastructure, AI-specific tests, deployment and runtime, owned by the delivery team.
How is DevSecOps for AI different from classic DevSecOps?
Classic DevSecOps secures code, packages, images and infrastructure. AI adds prompts, model files, datasets, vector indexes, model APIs and agent permissions as artefacts, plus behavioural tests such as prompt-injection suites that scanners cannot replace.
What is MLSecOps?
MLSecOps applies security practice to the machine learning lifecycle, especially models and data: provenance, integrity, safe formats, lineage and testing. In enterprise AI teams it works well as part of one shared pipeline.
How do you run LLM security testing in CI?
Deploy a preview environment and run a versioned suite of injection, jailbreak, tool-misuse and leakage cases against it. Repeat each case because outputs vary, and fail the build if any attack succeeds.
Why are model files a supply chain risk?
Pickle-based model formats can execute code when loaded, and models can be tampered with or swapped. Use approved sources, verify checksums, prefer safetensors and register approved versions.
Do I need an SBOM for an AI application?
Yes. Generate one per build and include model versions and MCP servers alongside OS and Python packages, so affected services are easy to find when an issue is announced.
Which frameworks should a secure AI pipeline follow?
The OWASP Top 10 for LLM Applications for application tests, the NIST AI Risk Management Framework for organisational risk management, and SLSA for build integrity and provenance.
Who owns AI pipeline security in an enterprise?
It is shared: the platform team provides pipelines and controls, application security sets rules and reviews high-risk changes, AI product teams own test cases and fixes, and risk teams consume the evidence.
Ready to build pipelines a bank's security team would approve? Cloudsoft's DevSecOps course teaches scanning, SBOMs, signing and policy-as-code with GitHub Actions, Kubernetes and Terraform, in classroom sessions in Ameerpet or live online. Call +91 96660 19191 for a free demo.
