The Human Coordination Problem
Coordination Under Disagreement
Even a technically reliable AI can serve people whose interests conflict. A company seeks an advantage, a researcher seeks a discovery, and a government seeks security. Each may see a reason to proceed while expecting someone else to absorb part of the risk. The coordination problem begins when those individually understandable choices produce a collectively dangerous result.
Civilization works partly by making complete agreement unnecessary.
People can hold different values while establishing procedures for evidence, decisions, appeals, and limits on authority. That is the constitutional idea behind this inquiry. Human coordination under disagreement means developing enough common ground to address catastrophic risk while preserving disagreement about culture, politics, research, and the future.
People exposed to a risk also have interests that extend beyond the preferences of those creating it. An agreement among powerful developers needs a separate account of public representation, minority rights, and meaningful challenge. Participation by the organizations with technical expertise is one requirement; authorization by the people whose interests are at stake is another.
This is the human problem inside AI governance. The machine layer concerns artificial behavior. The human layer concerns the purposes people authorize and the procedures through which they contest decisions. SIRDA is examining whether an institution can address that second problem while leaving room for different visions of the future.