Aligning AI to values
What models should be trained toward, who supplies that target, and how values are elicited from real people.
- Curated by
- Taylor Sorensen
- Relevant for
- 10 grid cellsAgent negotiation & bargaining and 9 others
- Write to
- values@agi-institutions.orgAn operator routes it to the right experts.
Selected papers
5
Must-reads for understanding the field and why it matters for AGI institutions.
A foundational philosophical treatment of what aligning AI to instructions, preferences, interests, or values would each look like, and how to choose fairly among them amid disagreement.
Spells out the "helpful, honest, and harmless" framing many AI systems are now trained toward and how those objectives conflict, laying groundwork for Constitutional AI and Claude's constitution.
Proposes "thick models of value," a structured and substantive way of representing values, and argues for aligning both AI systems and the institutions around them to them.
Elicits values from people through an LLM interview, has people judge which values apply in which context, and offers the result as a concrete alignment target, with experiments.
Asks how AI systems can allow for legitimate variation in values across and within people and cultures, and proposes three ways to define and measure such pluralism.
Work in the field
9
Recent work in the field worth knowing.
Foundations
3
Older work that gives background on the field.
This list isn't exhaustive. About the lists.