AI Team Structure: Roles That Make an AI Factory Work

An AI team structure is the set of roles and ownership rules that lets an organization take a use case from intake to a monitored production system without renegotiating who does what each time. The roles are stable; what changes with scale is how many people hold them and where they report. The NIST AI Risk Management Framework asks for this in GOVERN 2.1: "Roles and responsibilities and lines of communication related to mapping, measuring, and managing AI risks are documented and are clear to individuals and teams throughout the organization." Kreuzberger, Kühl and Hirschl, in their 2022 MLOps paper, call MLOps "an interdisciplinary group process." The AI team structure that works gives every production system one named owner.

What an AI Team Structure Is For, and Why Titles Are the Wrong Place to Start

Two companies can hire the same titles and run very different systems. The difference is who answers for a model the Tuesday after go-live.

Melvin Conway put it in How Do Committees Invent? in 1968: organizations that design systems "are constrained to produce designs which are copies of the communication structures of these organizations." Split model builders and model operators across two departments, and the system gets a seam where they meet.

Google's MLOps guidance describes that seam as the usual starting point. At level 0, "The process separates data scientists who create the model and engineers who serve the model as a prediction service." Once it serves traffic, nobody owns it.

Organization design is one of the five building blocks of an AI factory. This article spells that block out as roles and the accountability each one carries.

The Seven Roles of an AI Factory Team, and What Each One Owns

The seven roles below are the stable part of any AI team structure. They come from the Kreuzberger paper.

The paper's R1, the Business Stakeholder ("similar roles: Product Owner, Project Manager"), "defines the business goal to be achieved with ML." R2, the Solution Architect, "designs the architecture and defines the technologies to be used"; what we call the AI lead holds it once per program. R3, the Data Scientist, "translates the business problem into an ML problem and takes care of the model engineering." R4, the Data Engineer, "builds up and manages data and feature engineering pipelines." R5, the Software Engineer, works "to turn the raw ML problem into a well-engineered product." R6, the DevOps Engineer, owns CI/CD and deployment. R7, the ML Engineer/MLOps Engineer, "combines aspects of several roles and thus has cross-domain knowledge."

Governance is an eighth accountability, held at first by an existing role. NIST GOVERN 3.2 asks for policies that "define and differentiate roles and responsibilities for human-AI configurations and oversight of AI systems," and ISO/IEC 42001 expects the same assignment. Databricks' MLOps guidance notes that governance requirements "may come from a data governance officer or business stakeholders." Small programs give it to the AI lead.

Roles are also a different count from people. Microsoft's Azure guidance on workload team personas says "An individual can embody one or multiple personas, depending on context," and suggests starting "with between 5 and 10 personas."

Centralized CoE, Hub and Spoke, or Embedded: Where Each Role Sits

The same AI team structure can be drawn three ways, and the business roles never move. Which of the four organizational shapes fits your enterprise, and when to change it, is covered in AI center of excellence vs AI factory.

In a centralized CoE, every technical role reports to one team and product owners join per use case. Microsoft's Cloud Adoption Framework guidance on an AI Center of Excellence notes that such a CoE "commonly builds on existing teams rather than being a standalone team."

In hub and spoke, the AI lead, the platform engineer, the MLOps engineer and the governance owner sit in the hub. Data scientists and application engineers sit in the spokes with their product owner. Microsoft's advice for this stage is to "Distribute AI expertise into product teams, platform teams, and enabling teams." In Team Topologies vocabulary, a spoke is a stream-aligned team, and the hub adds an enabling team that "helps a Stream-aligned team to overcome obstacles."

In the embedded model, sometimes called federated, every role lives in product teams and the center keeps standards. At this stage, the CoE Microsoft describes "focuses on guidance and policy rather than direct control." Across the three drawings, only the reporting lines of R6, R7 and governance move.

Team Size by Stage: One Use Case, a Handful, a Portfolio

An AI team structure for one use case looks nothing like one for twenty, because parallel work forces roles apart.

One use case. Kim, Zimmermann, DeLine and Begel described five working styles of Microsoft data scientists in 2016, including "polymaths," who "do all data science activities themselves." A first team is one polymath covering R3, R5 and R7, a product owner and a borrowed data engineer, with the program lead acting as R2. It lasts until the second use case.

A handful in parallel. The same authors' 2017 survey of 793 professional data scientists at Microsoft found "9 distinct clusters." We read that as specialization growing with the number of systems. The first dedicated hire is the MLOps engineer, since registry and pipelines are shared. Google's MLOps guidance adds that model builders "might not be experienced software engineers who can build production-class services," so R5 becomes mandatory here.

A portfolio. R6 and R7 form a platform team, and governance gets its own owner. The Google SRE book calls SRE "what happens when you ask a software engineer to design an operations team," and Google caps operational work at 50% of an SRE's time. We apply the same cap to AI platform teams; without it they turn into ticket desks.

Hiring order follows from this: R3 or R5 first, by use case type, then R7, then R6 once the platform has a second customer team. R1 is never hired; the business provides it.

A RACI for the AI Lifecycle: Who Is Accountable at Each Stage

A RACI (Responsible, Accountable, Consulted, Informed) is where an AI team structure stops being a slide. The NIST framework is voluntary, yet GOVERN 2.1 asks for documented roles, and GOVERN 2.3 states that "Executive leadership of the organization takes responsibility for decisions about risks associated with AI system development and deployment." Our six rows:

1. Intake and business goal: A = product owner (R1), C = AI lead (R2).
2. Data and features: A = data engineer (R4), R = data scientist (R3), C = governance owner.
3. Model and evaluation: A = data scientist, R = application engineer (R5), C = product owner.
4. Release to production: A = MLOps engineer (R7), R = platform engineer (R6), I = product owner.
5. Run and monitoring: A = MLOps engineer, R = platform engineer, I = governance owner.
6. Oversight and compliance: A = governance owner, C = AI lead, I = executive sponsor.

The release and run rows follow Databricks' guidance: "ML engineers own the production environment where ML pipelines are deployed and executed," and "Data scientists typically do not have write or compute access in the production environment."

Rule #11 in Zinkevich's Rules of Machine Learning reads "Give feature columns owners and documentation." If a column needs an owner, so does the model built on it. Every production system gets one named owner before release.

Six Structural Failures We Are Asked to Repair

Each of these six AI team structure failures has come up in a program we were brought in to fix.

1. The handoff to IT. Data scientists hand a notebook to operations. Fix: R7 in the same team as R3.
2. One hire for seven roles. The paper says R7 combines skills of "data scientists, data engineers, software engineers, DevOps engineers, and backend engineers," which describes a team. Fix: two people and a platform.
3. Skills on one side only. Amershi and colleagues at Microsoft (2019) found that customizing models takes skills most software teams lack; Zinkevich adds that "Most of the problems you will face are, in fact, engineering problems." Fix: R3 and R5 together from sprint one.
4. Governance as a committee. GOVERN 2.1 asks for roles clear throughout the organization, which a monthly meeting does not supply. Fix: a named owner in the RACI.
5. A platform team with no product owner. Without its own R1 and the 50% cap, it becomes a help desk.
6. Nobody accountable after go-live. Fix: name the owner before release.

How the Team Plugs Into an AI Factory, and How We Staff It

The AI team structure above is the fourth building block of an AI factory. Without a platform, R6 and R7 have nothing to run; without product teams the platform has no users.

We usually take the R2, R6 and R7 roles at the start, and each of our people has a named successor in the client's team from day one. The handover ends when our names leave the RACI; that is how the organization design and CoE building block of our enterprise AI factory service works.

A RACI with an untrained product owner accountable for intake is fiction, so we add role-based AI literacy for the product owners and domain experts around the team. To start, bring us your current org chart and the list of systems nobody owns after go-live.

FAQ

How many people do you need to start an AI team?

The right AI team structure for a first use case is three roles held by two or three people: a product owner from the business, an engineer who builds the system, and an MLOps engineer who runs it in production. One person can hold both technical roles for one use case, though never for three systems.

What are the core roles in an AI team?

Seven roles recur in research and vendor guidance: product owner, solution architect, data scientist, data engineer, software engineer, platform engineer and MLOps engineer. Governance is an eighth accountability that one of them holds until the program needs a dedicated owner. One person can hold several roles.

Where should ML engineers sit: in a central platform team or in product teams?

Engineers who run the shared platform and pipelines sit centrally, because every use case depends on them. Engineers building a specific use case sit with its product owner and domain experts. Which overall organizational shape fits is a separate decision, covered in our comparison of an AI center of excellence and an AI factory.

Who owns an AI model once it is in production?

The team that runs the production environment owns it, usually led by the MLOps engineer. Databricks' MLOps guidance gives ML engineers the production environment and data scientists no write access there. The product owner stays accountable for whether the system still meets its business goal.