Everyone is talking about using AI in manufacturing, but almost nobody is talking about how to validate it.

Someday, that gap is going to cost some companies dearly. Not because AI is bad for regulated manufacturing, but because the people deploying these tools are treating AI like any other software upgrade, and it isn’t that either.

I’ve been building electronic software solutions for regulated industries for over 30 years. We started with AI by adapting it as a tool to improve our customer experience with electronic software solutions and to improve our quality management systems. While doing this work, I keep running into the problem that the AI validation question doesn’t have a clean answer yet. Regulators are just catching up. The industry is moving faster than the guidance which has generally been the case with all new introductions.  We went through when electronic signatures were first introduced. Manufacturers who don’t understand the difference between a validated AI tool and a useful-but-unqualified one are going to find out the hard way when FDA inspectors are given final guidance on AI validation.

What Makes AI Different From the Software You Already Validate

If you’ve done Computer System Validation (CSV) for a standard electronic batch record system or a LIMS, you know the process per GAMP 5: you write user requirements, functional specifications, detailed design specifications, run IQ/OQ/PQ protocols, document the results, lock the system, and manage changes. It’s tediaus, but it’s straightforward. The system does what you tested it to do, and it keeps doing it.

However, AI doesn’t work that way. An AI model produces outputs based on probabilities and what its large language model learns. Then its behavior can shift as it learns more. So if you’re using a large language model or a machine learning system to make decisions that affect your GMP records, the question “does this system do what we validated it to do?” becomes genuinely hard to answer.

Traditional CSV assumes the system is deterministic. Each time you put in an input, you get the same output, every time. AI doesn’t work that way. Each input is treated individually, and you may get a different answer and sometimes you get more than you asked for. That’s what makes AI powerful, and that’s what makes validation so much more complex.

What the Regulators Are Saying Now

The starting point in the US is that existing 21 CFR Part 211 and Part 11 requirements apply to AI systems used in drug manufacturing — there is no AI carve-out. If your AI tool generates, modifies, or archives electronic records in a GMP context, it must meet Part 11’s audit trail, electronic signature, and data integrity requirements. That makes sense in a regulated environment.

The FDA sharpened its thinking in January 2025 with a draft guidance, Considerations for the Use of Artificial Intelligence to Support Regulatory Decision-Making for Drug and Biological Products. Rather than creating new exemptions, it lays out a risk-based framework for establishing the credibility of AI models used to support decisions about safety, efficacy, and product quality across the drug lifecycle, including manufacturing.

A year later, the FDA and EMA issued joint Guiding Principles of Good AI Practice in Drug Development (January 2026) — ten principles spanning research through manufacturing and safety monitoring — signaling convergence between US and European regulators. On the industry side, ISPE released its GAMP Guide: Artificial Intelligence in July 2025, the first comprehensive framework dedicated to validating AI-enabled systems in GxP environments.

Europe is moving in parallel. The European Commission opened consultation in 2025 on a new Annex 22 on Artificial Intelligence (drafted by the EMA GMDP Inspectors’ Working Group with PIC/S), with final guidance expected in 2026. If you sell into European markets, this is not a future concern — it’s a current one.

What the Industry Is Getting Wrong

The most common mistake I see is treating AI as a black box you plug into an existing quality system. You add an AI-powered deviation detection tool, or an AI module in your quality management system, and you assume the vendor has handled the validation. Sometimes they have. Often they haven’t. Or they have documented validation that doesn’t fully address your specific use case, your data, or your operating environment.

The second mistake: validating the algorithm without validating the data feeding it. An AI system is only as reliable as the data it was trained on and the data it receives in production. If your batch records are inconsistent, your inventory inputs are manual and error-prone, or your SOPs haven’t been standardized, the AI will confidently produce outputs from bad inputs. Garbage in, garbage out. That sentence has been true since the first computer was built. AI doesn’t change it. It amplifies it.

We learned that we can use AI to improve the way we write and work with our standard operating procedures. But that only works when the underlying documentation is already traceable and controlled. You can’t automate your way out of a messy quality system. You have to clean it up first.

What Good AI Validation Actually Looks Like

The clearest current framework is the ISPE GAMP Guide: Artificial Intelligence, released in July 2025 — the first guide dedicated entirely to validating AI-enabled systems in GxP environments. Its central shift mirrors what the FDA-EMA joint principles signal at the regulatory level: move from static validation to continuous validation. Because AI models can change over time, you need ongoing monitoring of model performance, drift detection, and defined criteria for when a change triggers a new validation event.

On the FDA side, the January 2025 draft guidance introduced a risk-based credibility assessment framework for AI models — the idea being that not every AI tool carries the same risk, and your validation effort should be proportional to how consequential the model’s output is. A model that flags anomalies for human review carries different stakes than one that drives autonomous release decisions. The FDA has also introduced the Predetermined Change Control Plan (PCCP) concept, which lets manufacturers pre-define acceptable categories of model changes and get them approved upfront, rather than filing a supplement every time the model updates. Think of it as a validated change corridor. It’s a pragmatic acknowledgment that AI systems are supposed to improve over time — and that regulators need a framework that doesn’t punish improvement.

For InstantGMP users, this means the foundation matters more than ever. Traceable electronic batch records, controlled document management, audit trails that meet 21 CFR Part 11: these aren’t just compliance checkboxes. They’re the data infrastructure that makes AI validation possible. A global manufacturing platform that already maintains clean, structured, auditable records is the starting point for any AI deployment that regulators will accept.

What InstantGMP AI Validation Looks Like

We don’t just say our AI features are validated. We built a protocol to demonstrate it, and the approach is more rigorous than most of what I see in the industry right now.

The foundation is the MCP server that connects InstantGMP to large language models such as Claude. The first validation step is straightforward: every API called by the MCP server is validated as part of our standard operational qualification. That step confirms the data pipelines are working correctly before the AI ever touches them.

The validation itself works as a reference standard comparison. We download data directly from the application not through the APIs or the MCP server and draw from each module: batch records, inventory, quality events, documents. That direct download is the reference standard. We then run the same queries through the MCP server connected to the LLM and compare the AI-generated report to the reference data. Acceptance criteria are defined before we start: factual accuracy must be exact. Every data value, count, and record in the AI output must match the reference while presentation and formatting are permitted to vary. An LLM can phrase things differently on different runs. The underlying data cannot.

Because LLMs are non-deterministic, we run each test query multiple times and confirm the factual content is consistent across runs. We also include negative test cases. These are  queries designed to return no results to confirm the system responds with “no data found” rather than generating plausible-sounding fabrications. That hallucination risk has no analog in traditional software validation, and we test for it explicitly.

Access control parity is validated as a separate step: we confirm that a user with restricted permissions in the application receives the same restricted view when querying through the AI. 21 CFR Part 11 does not make exceptions for AI-generated reports.

Finally, we document change control triggers in advance: which changes to the LLM model version, the MCP server code, or if the InstantGMP application require a re-validation run. This is the AI equivalent of a Predetermined Change Control Plan. We define the boundaries before we need them, not after something changes.

The result is a validation package that stands on its own at an inspection: documented methodology, explicit acceptance criteria, reproducibility evidence, negative test results, and a defined change control framework. That is what validated AI means in a GMP environment. It’s not a vendor attestation, it’s a protocol with results.

The Question Worth Asking Before You Deploy Anything

Before you add any AI tool to a GMP process, ask this: if an FDA investigator walked in tomorrow and asked me to show the validation documentation for this system, could I hand it to them?

If the answer is no, or not sure, or we’re working on it: you’re not ready. And “the vendor said it’s validated” is not an answer. It’s a starting point for a conversation.

The AI tools coming to regulated manufacturing are genuinely useful. I believe that. We are learning more every day about how to use them well. But the manufacturers who will come out ahead are the ones who treat validation as a prerequisite, not an afterthought. They’re the ones who won’t have an uncomfortable conversation at their next inspection.

 

Sources

Federal Register — FDA draft guidance, Jan 7 2025: federalregister.gov/documents/2025/01/07/2024-31542/

FDA — Guiding Principles of Good AI Practice in Drug Development (Jan 2026): fda.gov/about-fda/artificial-intelligence-drug-development/

ISPE — GAMP Guide: Artificial Intelligence (July 2025): ispe.org/news/ispe-releases-gampr-guide-artificial-intelligence

ECA Academy — EU GMP Annex 22 (Draft 2025): gmp-compliance.org/guidelines/gmp-guideline/eu-gmp-annex-22-draft-2025-artificial-intelligence