A new and flexible approach to the analysis of paired comparison data

Ivo F.D. Oliveira, Nir Ailon, Ori Davidov

Research output: Contribution to journalArticlepeer-review


We consider the situation where I items are ranked by paired comparisons. It is usually assumed that the probability that item i is preferred over item j is pij = F(µi−µj) where F is a symmetric distribution function, which we refer to as the comparison function, and µi and µj are the merits or scores of the compared items. This modelling framework, which is ubiquitous in the paired comparison literature, strongly depends on the assumption that the comparison function F is known. In practice, however, this assumption is often unrealistic and may result in poor fit and erroneous inferences. This limitation has motivated us to relax the assumption that F is fully known and simultaneously estimate the merits of the objects and the underlying comparison function. Our formulation yields a flexible semi-definite programming problem that we use as a refinement step for estimating the paired comparison probability matrix. We provide a detailed sensitivity analysis and, as a result, we establish the consistency of the resulting estimators and provide bounds on the estimation and approximation errors. Some statistical properties of the resulting estimators as well as model selection criteria are investigated. Finally, using a large data-set of computer chess matches, we estimate the comparison function and find that the model used by the International Chess Federation does not seem to apply to computer chess.

Original languageEnglish
JournalJournal of Machine Learning Research
StatePublished - 1 Nov 2018

Bibliographical note

Publisher Copyright:
© 2018 Ivo Fagundes David de Oliveira, Nir Ailon and Ori Davidov.


  • Chess
  • Linear stochastic transitivity
  • Model selection
  • Semi-definite programming
  • Sensitivity analysis
  • Statistical ranking

ASJC Scopus subject areas

  • Software
  • Artificial Intelligence
  • Control and Systems Engineering
  • Statistics and Probability


Dive into the research topics of 'A new and flexible approach to the analysis of paired comparison data'. Together they form a unique fingerprint.

Cite this