Microsoft’s Synthetic Tenants: How Copilot Gets Evaluated Without Touching Real Enterprise Data
Microsoft Research has published an account of how it evaluates Microsoft Copilot without touching customer data: it builds synthetic organizations, complete with users, email threads, and org charts that change over time. The work describes an Industry Evaluation Data Generation Platform that Microsoft says establishes “the first standardized evaluation baseline for Microsoft Copilot.”
Every enterprise AI team eventually hits the same wall. You want to know whether your assistant actually works, but the only data realistic enough to prove it is the data you are not allowed to look at. Customer emails, HR threads, meeting transcripts, internal strategy documents: this is exactly the material that would make an evaluation meaningful, and exactly the material that cannot leave the building. Synthetic data for LLM evaluation is one answer to that problem, and Microsoft’s platform is a detailed case study in how to generate it at enterprise scale.
The framing rests on two questions the platform is designed to answer: is Copilot doing what users need, and can Microsoft prove it without touching user data? Everything else in the design follows from the tension between those two goals.
The evaluation dilemma
The problem with real enterprise data is not that it is hard to obtain. It is that using it for evaluation means exposing sensitive information to people and systems that should not see it. Hand-built test fixtures offer an escape hatch, but they run into a different problem: scale. A dozen manually written queries against a toy dataset will not surface the failure modes that appear when an assistant is asked to reason across months of project history in a workspace with hundreds of users.
Microsoft’s argument is that small curated fixtures are too thin to represent enterprise reality, while real data is too sensitive to use. Synthetic environments sit between the two: large enough to be realistic, controlled enough to be safe.
Inside the Industry Evaluation Data Generation Platform
Microsoft describes each simulated workspace as a “tenant”: an isolated organizational environment with its own users, data, and configurations. The term matters because it mirrors how enterprise software is actually deployed. A tenant is not a flat document dump. It has structure, permissions, and a population of simulated people who produce content.
To improve realism, the environments are grounded in publicly available real-world data sources. That grounding is the bridge between “fake” and “plausible”: the synthetic organization borrows the texture of real industries without borrowing anyone’s actual confidential material. Microsoft frames these tenants as scalable baselines that teams and organizations can build on, rather than one-off test sets.
Time as a first-class dimension
Perhaps the most interesting design decision is that the synthetic tenants are not static snapshots. They simulate organizations that evolve: meeting histories accumulate, project timelines advance, email threads develop over weeks, and organizational structures change.
This is not decoration. Many of the queries Copilot receives are time-aware. “What did the team decide last Thursday?” and “What’s changed since the last quarterly review?” cannot be answered correctly by a frozen dataset, because the answer depends on what existed at a particular moment and what changed afterward. A static fixture can test whether an assistant retrieves a fact. An evolving environment tests whether it understands sequence, recency, and revision, which is much closer to the work an enterprise assistant is actually hired to do. Anyone who has worked on retrieval pipelines knows that temporal reasoning is where plausible-sounding answers most easily drift from the truth.
Building an enterprise AI evaluation benchmark across industries
Microsoft generated three synthetic environments across multiple industries, each with distinct characteristics intended to mirror real-world usage. The rationale for industry specificity is straightforward: generic test cases miss domain-specific failure modes. A question that is easy in a software company’s workspace may be genuinely hard in a healthcare or manufacturing context, where terminology, document types, and the stakes of a wrong answer all differ.
Three environments is a modest number against the breadth of enterprise use, and Microsoft does not claim otherwise. The pitch is that the platform is a baseline others can extend, not a finished catalog.
The automated query pipeline, step by step
Generating realistic environments is only half the job. You also need questions worth asking of them, and Microsoft argues that manually writing thousands of grounded, industry-specific queries would be prohibitively expensive and slow. Its alternative is an automated pipeline that turns simple, generic query ideas into realistic, industry-specific questions:
- Start with a library of common workplace tasks, the kind of request that appears across nearly every organization.
- Narrow to what is relevant for a given industry, filtering out tasks that would not plausibly arise in that domain.
- Adapt using domain-specific language and context, so the phrasing matches how practitioners actually talk.
- Generate queries grounded in the synthetic environment’s data, using AI to tie each question to material that genuinely exists in the tenant.
- Rewrite for naturalness, so the queries sound like something a human would type rather than something a template produced.
- Validate each query for accuracy and uniqueness, catching duplicates and questions the environment cannot actually answer.
The ordering is sensible engineering. Grounding before rewriting means the natural-sounding version still points at real content; validating last means the checks apply to the final artifact rather than an intermediate draft.
A Copilot evaluation walkthrough in healthcare
Microsoft’s example comes from healthcare. The scenario is a healthcare tenant in which a team meets to review safety data, and Copilot is asked to summarize the meeting and extract action items. Copilot returns a structured summary that cites specific discussion points from the transcript.
The evaluation question is whether that response is accurate, complete, and properly grounded in the meeting content. Each criterion does distinct work. Accuracy asks whether the summary misstates anything. Completeness asks whether material decisions or assigned actions were dropped. Grounding asks whether the cited discussion points actually appear in the transcript, which is the check that separates a genuinely useful citation from a confident-sounding fabrication. For engineers building retrieval-augmented systems, that triad is a familiar and demanding bar.
One caveat about the source
The published material describes the method rather than reporting results, so it does not include benchmark scores, publication dates, or named customers, and it does not spell out every component of the evaluation strategy. For now the platform is best read as infrastructure and methodology: an argument about how enterprise assistants should be evaluated, with the numbers still to come. The broader governance question of how evaluation fits into a regulated ML life cycle is one the industry is still working through, as our coverage of compliance across the enterprise ML life cycle explores.
The bottom line
Microsoft’s core claim is that you cannot prove an enterprise assistant works by testing it on data you are not allowed to touch, and that small hand-built fixtures are too shallow to substitute. The proposed alternative is a synthetic organization realistic enough to exercise the assistant the way a real workplace would: populated with users, seeded from public sources, and moving through time. That answers both of the questions the platform set out to address. The first, whether Copilot does what users need, becomes measurable against environments that behave like real work. The second, whether Microsoft can prove it without touching user data, is satisfied by construction, since no customer material enters the loop. Whether the approach delivers a genuine standardized baseline depends on details the published material does not yet supply, including scores and dates. But the design instinct, that evaluation environments should be temporal, industry-specific, and generated rather than handwritten, is one more teams should probably borrow.
Related: four silent data errors.
