Evaluating User Model Effectiveness by Simulation

Accurate initialization of a User Model (UM) is important for every
system that provides personalized services. However, there are systems where
this initialization is critical yet no user data is available from prior interactions.
For example, one in which both the repeat usage by a user is rare, and the total
interaction on a single use is relatively limited.
Evaluating the effectiveness of a User Model and a particular instance of such
a model is never an easy task, particularly through user studies. Moreover,
evaluation typically focuses on the usability of an entire system rather than the
performance of a specific UM instantiation. In this paper we propose
evaluating the quality of UMs via simulation and comparison to a "gold
standard". This standard is an approximation of the user's ideal model. We will
demonstrate this through a case study of a museum's visitor guide system
implemented in the Hecht Museum at the University of Haifa, Israel