The Evaluation Process

Participants in the challenge will receive performance reports and will be able to submit improvements to their solvers during the entire competition period. A leader board will be continuously updated. The criteria for the evaluation of the different tasks are as follows:

• PR: The score for the partition function, evaluated only for those models for which we were able to obtain exact answers (given extensive time and memory resources), is as follows: Denote the exact partition function by Z^* and the approximated one by Z^s. The score will be |log(Z^*/Z^s)|.

• MPE: The performance of the most probable explanation estimate will be computed relative to the performance of the other competitors a simple asynchronous belief propagation baseline and a default result. Thus we will also evaluate this task on models where MPE cannot be computed exactly. The default result is the assignment that maximize only the one variable factors (and the first value if no such factor exist for some variable). The score will be calculated as follows. Denote the energy of a solution x by E(x). The energy is E(x) = - ∑ log f_a(X_a = x_a). We denote the standard BP result as x^bp and the default result by x^def. Solvers scores will be relative to the BP or the default solution. The score will be:
  

• MAR: The score for the marginals, evaluated only for those models for which we were able to obtain exact answers, will be calculated as follows. Denote the exact marginal for the i variable taking the x value as:P^*(X_i = x_i). In the same way the solver marginal will be denoted by: P^s(X_i = x_i). The score will be: