LAMEN

Evaluating language models through negotiation tasks

The project proposes using structured negotiations as a dynamic benchmark for evaluating language model (LM) agents. The negotiation framework consists of a game setting, issues to negotiate, and optional preference weights, allowing for the design of complex games by increasing the number of issues, mixing issue types, and adding non-uniform preferences. The benchmark setup jointly evaluates performance metrics (utility and completion rate) and alignment metrics (faithfulness and instruction-following) in self-play and cross-play settings.