The WhatIfWe Open Benchmark

Can AI reason from inside a tradition, and do more than borrow its vocabulary?

The first test in WhatIfWe's programme, designed to be open, reproducible and judged by the traditions themselves. Its answer matters to everyone who builds AI, and to every tradition that the world will increasingly read through it.

Why this question

A 2026 study tested language models on ethical dilemmas rooted in Indigenous contexts and described what it calls a persona illusion, or cultural masquerade: the models adopt an Indigenous persona on the surface while defaulting to individualistic conclusions. It names a gap between knowing about a tradition’s values and reasoning from them (Pegah Merrikhi, Research Square, May 2026, a preprint not yet peer reviewed). Other work points the same way. Agarwal and colleagues at Cornell and Microsoft Research found that an average American was a closer match to Indian cultural values than any of the Indian-trained models they tested, and that prompting strategies did not meaningfully improve alignment (2025 preprint). Whether instructions can close the gap is exactly what the second condition below tests.

The usual response is to survey more people in more countries. That changes whose preferences enter a system. The persona illusion is about something else: whether the system can hold a framework’s own logic well enough to reason inside it. As more of the world thinks through AI, that capacity decides whether AI widens how humanity thinks or narrows it. A 2025 preprint finds that diversity across models slows this kind of knowledge collapse, up to an optimal level (Hodel and West, University of Washington).

The design

The benchmark puts real problems to three kinds of system, choosing problems on which the traditions are known to reason differently.

  1. Tradition-grounded agents. Each agent draws only on its own tradition’s material, held in its own store, and has to cite the passages it uses. The separation is built into the system itself: an agent cannot reach another tradition’s material at all.
  2. One model, prompted to speak as each tradition in turn.
  3. One model, asked to answer for all the traditions at once.

Practitioners of each tradition then read the answers without knowing which system produced them. They judge whether they recognise their own tradition’s reasoning in each answer, and whether it reasons from inside the tradition or only uses its words.

The question the benchmark answers is whether the grounded agents are recognised as their traditions more often than a single model asked to play the part.

Run in the open

  • The problems, the way judges are recruited and the analysis are registered before any result is seen.
  • Results are reported whatever they show: how often each system is recognised, how many judges each tradition had, and the size and confidence of every effect.
  • The full set of problems is published so that anyone can run the benchmark again.
  • It needs only published texts, used with permission. No new material is asked of any community.
  • It is also the project’s first gate. If the grounded agents are recognised no more often than a single prompted model, the programme stops there, and we publish that result.

What it means for the traditions

The traditions judge. What counts as faithful to a tradition is decided by people formed in it, never by a model and never by us. Every passage an agent uses is cited, so each answer carries its sources. Whatever the result, each tradition learns something it has a right to know: how faithfully AI is reading it.

What it leads to

If each tradition stays recognisable when its agent is grounded in its own material, that becomes the foundation for the rest of the system: an open global forum where people take part from their own tradition, long-running groups working at depth, and one record that grows. What we’ll build sets out the whole design. How we’ll prove it describes the second test, which asks whether people working together across traditions reach further than the same people working apart.

Help us design it

These are the questions we want to settle with partners before the benchmark runs:

  • Which traditions should the first round include, and which problems make good tests?
  • How many judges does each tradition need, and how should they be invited so that the judging is fair and recognised by the tradition?
  • Under what licence should the problems and the code be published?
  • Which models should be tested? Should each tradition’s agent run on a different base model, and should those models be open, so that they can be retrained and rerun by anyone?

We are looking for researchers in evaluation and multi-agent systems, AI labs able to offer model access and compute, practitioners willing to choose problems and to judge, and partners to host the benchmark openly. The design is set out in full in a working paper, Can AI reason from inside a tradition? A preregistered benchmark for tradition fidelity in language models (Zenodo, doi:10.5281/zenodo.23979452), and on GitHub, where anyone can open an issue to propose a problem, question the design or review it from inside a tradition. For AI labs and funders sets out how this fits wider priorities.