A new and flexible approach to the analysis of paired comparison data

Ivo F.D. Oliveira, Nir Ailon, Ori Davidov

Research output: Contribution to journalArticlepeer-review


We consider the situation where I items are ranked by paired comparisons. It is usually assumed that the probability that item i is preferred over item j is pij = F(µi−µj) where F is a symmetric distribution function, which we refer to as the comparison function, and µi and µj are the merits or scores of the compared items. This modelling framework, which is ubiquitous in the paired comparison literature, strongly depends on the assumption that the comparison function F is known. In practice, however, this assumption is often unrealistic and may result in poor fit and erroneous inferences. This limitation has motivated us to relax the assumption that F is fully known and simultaneously estimate the merits of the objects and the underlying comparison function. Our formulation yields a flexible semi-definite programming problem that we use as a refinement step for estimating the paired comparison probability matrix. We provide a detailed sensitivity analysis and, as a result, we establish the consistency of the resulting estimators and provide bounds on the estimation and approximation errors. Some statistical properties of the resulting estimators as well as model selection criteria are investigated. Finally, using a large data-set of computer chess matches, we estimate the comparison function and find that the model used by the International Chess Federation does not seem to apply to computer chess.

Original languageEnglish
JournalJournal of Machine Learning Research
StatePublished - 1 Nov 2018

Bibliographical note

Funding Information:
This paper was written when the first author was a graduate student at the Israel Institute of Technology. The research leading to these results has received funding from the European Research Council under European Union’s Horizon 2020 Program, ERC Grant agreement no. 682203 “SpeedInfTradeoff” and the research of Ori Davidov was partially supported by the Israeli Science Foundation Grants No. 1256/13 and 457/17.

Publisher Copyright:
© 2018 Ivo Fagundes David de Oliveira, Nir Ailon and Ori Davidov.


  • Chess
  • Linear stochastic transitivity
  • Model selection
  • Semi-definite programming
  • Sensitivity analysis
  • Statistical ranking

ASJC Scopus subject areas

  • Software
  • Control and Systems Engineering
  • Statistics and Probability
  • Artificial Intelligence


Dive into the research topics of 'A new and flexible approach to the analysis of paired comparison data'. Together they form a unique fingerprint.

Cite this