Most of what the market publishes as an AI factory reference architecture describes a server room: GPU nodes, network fabric, storage tiers. An enterprise already running on a public cloud needs the layers every use case passes through on its way to production, and the contracts that let any product inside a layer be replaced without redrawing the rest. If swapping a model forces a new diagram, the diagram described products, not an architecture.
In their ICSE-SEIP 2019 case study, Amershi and colleagues surveyed 551 Microsoft software engineers (13.6% of the 4,195 invited) and ran 14 interviews. One of three differences they found between AI and earlier software: AI components are "more difficult to handle as distinct modules," because models can be entangled. It is one company's data, yet the lesson is general: in AI systems, layer boundaries have to be designed. A 2024 review by Nazir, Bucaioni and Pelliccione (41 primary studies selected from 3,038, plus 12 expert interviews across 9 countries) identified 35 design challenges, 42 best practices and 27 design decisions for ML-enabled systems. A hardware order settles none of them.
What an AI Factory Reference Architecture Describes
An AI factory reference architecture is a logical map of the layers every production AI system passes through, the single responsibility of each layer, and the contract each layer exposes to the others, independent of the products that fill them.
Most documents sold under that name are mappings of such a map: hardware validated designs, or one cloud's services arranged in boxes. The vendors say so. The AWS Machine Learning Lens calls its guidance "cloud- and technology-agnostic" and adds separate advice for implementing it on AWS. Databricks says its reference architectures "focus only on AWS services and the Databricks platform" and publishes separate versions for Azure and Google Cloud.
This article is about the logical architecture of an AI factory: the layers every production AI system needs, the contract each layer exposes, and how one design maps onto AWS, Azure or Google Cloud without being redrawn. It does not size GPU clusters or data center hardware, it does not choose a data platform or a vendor, and it does not catalog reusable components or explain how to run the platform as an internal product. It expands the first building block of the operating model this architecture serves.
Why AI Systems Break Ordinary Architecture Habits
Three findings explain why AI factory architecture needs explicit layers.
Models do not respect module boundaries. The Microsoft study found data harder to version than code and models entangled, with non-monotonic error behavior.
Old problems stay. Serban and Visser (SANER 2022) validated 20 architecture challenges and solutions and 20 architectural tactics. Traditional challenges such as component coupling still matter next to new ones such as continuous retraining, while ML-specific factors such as privacy play a marginal role next to scalability.
Most of the system is ordinary code. Sens and colleagues (ICSE 2025) studied 2,928 open-source ML-enabled systems: they "still mainly consist of traditional source code," and model reuse through duplicated code or pretrained models is common.
AI architecture is therefore mostly software engineering with three new problems: data that changes, models that age, and components that do not hold their boundaries. A reference architecture gives each of them a fixed place.
Five Layers and Two Planes: What Each One Owns
The AI factory reference architecture proposed here has five horizontal layers that every use case passes through and two cross-cutting planes that touch all of them. Each part owns one job, shown in the table.
The data foundation treats ingestion, storage and transformation as one block, the scope of governed data foundations. The knowledge and features layer, in Azure's words, "enforces data access policies and authorization." The gateway in the inference layer is the one place for authentication, quotas, routing and filtering, and the client stays thin.
The split is not one vendor's frame. Azure's AI application design guidance names five layers (client, intelligence, inferencing, knowledge, tools), each of which "enforces its own policies, identities, and caching strategies." Databricks draws seven swim lanes, from Source to Storage, under a cross-cutting governance system. Lu and colleagues at CSIRO's Data61 describe a reference architecture for foundation model based systems in three layers: supply chain, system and operation. The five-plus-two split is our synthesis; the replacement tests are our recommendation.
Contracts Between Layers Matter More Than the Products Inside Them
A design is technology-agnostic when every layer talks to its neighbors through a documented contract, so any product inside a layer can change without the neighbors noticing. That is what technology-agnostic AI architecture means in practice.
Azure's guidance spells it out: abstract your models and tools; block direct access to data stores, so AI code reaches data through an API that enforces authorization and carries the user's context; design around capabilities rather than specific technologies. Its examples of open interfaces are OpenAPI for tools, ONNX for model portability and OpenTelemetry for telemetry.
Google Cloud's AI and ML performance guidance applies this to pipelines under "Promote modular design": each step is a containerized component that works "like a function, with input parameters and output artifacts." Data can carry a format-level contract too: an open table format such as the Apache Iceberg spec lets the query engine change without rewriting consumers.
Technology-agnostic does not mean the architecture works with every product. It means no product is written into a contract. A managed service inside a layer is fine; inside a contract, it is a problem. Azure itself says to prioritize prebuilt SaaS and PaaS solutions, with compensating controls at the gateway.
Design for Replacement: Models Retire, the Architecture Should Not
AI components live shorter lives than the architecture around them. Azure's design methodology for AI workloads warns that "what you build today might become obsolete quickly" and that "some components might have a limited lifespan."
The Azure Architecture Center guide to foundation model life cycles treats models as versioned dependencies: deployments of retired models in model-as-a-service offerings "return HTTP errors," and deprecated models accept no new deployments. A model change can touch six areas: the model, its configuration, the prompt, orchestration logic, grounding data and hardware. The guide puts a router in front of the orchestrator and a gateway in front of the models, and keeps tested combinations "pinned together." Hardware ages too; the guide cites a retiring GPU virtual machine series as a reason to move hosting.
The test question: if your provider retired your model next quarter, how many layers would change? One, inference and serving, plus the pinned configuration. Anything more means the model is written into a contract.
One Logical Design, Three Cloud Mappings
An AI factory reference architecture reaches a specific cloud through a mapping, and the design itself stays unchanged.
The logical diagram carries no product names; the mapping is a separate table, layer to service, that can change while the diagram stays put.
The providers already work this way. AWS keeps agnostic guidance beside its AWS implementation, Databricks publishes one swim-lane logic in a version per cloud, and Google Cloud treats AI and ML as a perspective on its Well-Architected Framework, organized into five pillars. The providers differ in the mapping.
Record each product choice as a decision with its reasoning, because the question that returns later is why this product, not why this layer; a lightweight architecture decision record is enough. Which product fills the data and knowledge layers is a selection question with its own scorecard, covered in choosing the platform for these layers.
Five Tests Before You Approve an AI Factory Design
Before approving an AI factory reference architecture from a vendor or your own team, run five design review tests.
- Replacement: can a model, a storage engine or a cloud change by editing one layer and one mapping row? Expected: yes.
- Contract: does every layer have a written contract (API, format, telemetry schema) free of product names? Expected: yes.
- Data access: does any AI code read a data store directly, bypassing authorization? Expected: no.
- Pinning: do model, prompt, configuration and orchestration ship to production as one tested combination? Expected: yes.
- Second use case: does the next one add only its own application logic, or start again from data and hosting? Expected: the former. What gets built once and shared across teams is a separate question.
A reference architecture is a document. The factory starts when the layers run on the cloud you already pay for, and that build is what we do at DS Stream. Our service page describes the AI platform reference architecture block as a "cloud-native, technology-agnostic architecture covering data foundations, MLOps/LLMOps, AIOps monitoring, and agentic AI infrastructure across GCP, Azure, and AWS." If the architecture on your desk is a cluster specification, talk to our architects.
Frequently Asked Questions
What is an AI factory reference architecture?
An AI factory reference architecture is a logical map of the layers every production AI system passes through, with one responsibility and one contract per layer, independent of the products inside them. It describes how use cases reach production, where a hardware design describes a cluster.
How is a logical AI architecture different from a hardware reference design?
A hardware reference design specifies servers, networking and storage for one platform. A logical architecture specifies layers and contracts, and treats hardware and cloud as one possible mapping. AWS keeps its Machine Learning Lens guidance "cloud- and technology-agnostic" and adds AWS specifics separately.
What layers does an AI factory need?
We propose five layers (data foundation; knowledge and features; model development and registry; inference and serving; application and orchestration) and two cross-cutting planes (governance and security; observability and delivery). It is our proposal, drawn from the layered splits of Azure, Databricks and Lu and colleagues.
Does technology-agnostic mean avoiding managed cloud services?
No. Azure's Well-Architected guidance says to prioritize prebuilt SaaS and PaaS solutions that meet your requirements, adding compensating controls at the gateway. Agnostic describes the contracts between layers; the hosting inside a layer can be fully managed.
Sources
- Amershi S. et al., Software Engineering for Machine Learning: A Case Study, ICSE-SEIP 2019.
- Nazir R. et al., Architecting ML-enabled systems: Challenges, best practices, and design decisions, Journal of Systems and Software, 2024.
- Serban A., Visser J., Adapting Software Architectures to Machine Learning Challenges, SANER 2022.
- Sens Y. et al., A Large-Scale Study of Model Integration in ML-Enabled Software Systems, ICSE 2025.
- Lu Q. et al., A Reference Architecture for Designing Foundation Model based Systems.
- Microsoft, Application design for AI workloads on Azure.
- Microsoft, Design methodology for AI workloads on Azure.
- Microsoft, Design to support foundation model life cycles.
- AWS, Machine Learning Lens.
- Databricks, Databricks reference architectures.
- Google Cloud, AI and ML perspective; Performance optimization.
- Apache Iceberg, Table Spec.
.webp)
