Nonparametric Predictive Inference for Selection and Ranking

Abstract

This thesis introduces Nonparametric Predictive Inference (NPI) for selection and ranking, based on a single future observation from each group, and focuses on two main contributions. First, the development of NPI procedures for various selection and ranking events. Secondly, the application of different loss functions to quantify the loss incurred from non-optimal selection and ranking decisions. Initially, NPI is applied to rank the best groups within subsets. A selected subset refers to one or more independent groups that are better than the rest, where better means that all future observations from the groups in the selected subset exceed all the future observations from the non-selected groups. The ’independent group’ means that information about the random quantities for one group does not provide any information about the random quantities for another group. Two special cases are considered: selecting a ranked subset of the two best groups and the three best groups. For the subset consisting of two groups, the exact NPI lower and upper probabilities are derived for the event that these are the two best groups, while for the subset consisting of three groups, approximate NPI lower and upper probabilities are derived for the event that these are the three best groups. The thesis further explores a more general case of selection and ranking, addressing how to rank buckets containing one or more independent groups. Here, ’bucket’ refers to a cluster or grouping of independent groups. This approach tackles two key questions: how to allocate groups to buckets, and how to determine the optimal number of buckets—defined as the allocation that maximises or minimises the NPI lower and upper probabilities for a given event. Various allocation methods are evaluated, including those based on measures such as the median. Additionally, the NPI-Bootstrap method is used to estimate probabilities, to approximate the probability of the event of interest itself, rather than its lower and upper probabilities. Throughout the thesis, data from the literature illustrate and support the methods. In this thesis, the NPI method is applied across various selection and ranking events, using different loss functions to quantify the loss incurred from non-optimal selection and ranking decisions. Uncertainty is quantified by calculating the NPI lower and upper expected losses for the events corresponding to these scenarios. In the selection scenario, zero-one, linear, and quadratic loss functions are used in both pairwise and multiple comparisons. Several selection events are considered, including selecting the best group, selecting the subset of best groups, and selecting the subset that includes the best group. In ranking scenarios, zero-one and general multi-level loss functions are applied to ranked subsets of best groups. The zero-one loss function provides a binary measure of whether the ranking is correct, while the general multi-level loss function allows for a more nuanced evaluation by assigning penalties based on the specific ranking of groups according to the next future observation per group. For the general event of selection and ranking, linear and quadratic loss functions are used to evaluate the ranking of groups assigned to different buckets. The effect of the use of different loss functions on the selection and ranking decisions is illustrated by examples.