Criteria Of Efficiency

In the case of classification, an early criterion of efficiency was the percentage of multiple prediction regions (i.e., prediction regions containing more than one label) output on a test set. Vovk et al. (2017) argue that this criterion has serious disadvantages and propose a class of criteria of efficiency that they call "probabilistic". An example of a probabilistic criterion is the average size of the prediction region.

Bibliography

• Mauricio Sadinle, Jing Lei, and Larry Wasserman (2016). Least ambiguous set-valued classifiers with bounded error levels. Technical Report arXiv:1609.00451 [stat.ME].
• Vladimir Vovk, Ilia Nouretdinov, Valentina Fedorova, Ivan Petej, and Alex Gammerman (2014). Criteria of efficiency for conformal prediction. On-line Compression Modelling Project (New Series), Working Paper 11. Last revised 2017.