Learning from Positive and Unlabeled Data with a Selection Bias

Masahiro Kato, Takeshi Teshima, and Junya Honda. 2019. International Conference on Learning Representations (ICLR 2019). Proceedings page (OpenReview). 日本語版

PU learning trains a classifier from positive and unlabeled data alone. The standard theory assumes the labeled positives were drawn at random from all positives, and real data often violates that assumption. This study formalizes the case where the selection of positives is biased and gives a method that still learns a classifier under it.

The problem

When prospects are estimated from purchase histories, only the people who bought appear as positives. In medical data, only the patients already diagnosed count as positives. The labels therefore arrive in an order that depends on the features: the more visible customers are found first, and the patients with severe symptoms are diagnosed first. Conventional PU learning assumes random labeling, so a classifier trained this way absorbs the bias. This work asks how much of the classification task can still be learned once a realistic assumption is placed on how the bias arises.

The proposed method

The method starts by granting that the chance of being labeled depends on the features. It then states, as an explicit assumption, the relation between that chance and the probability the classifier is meant to recover, and derives a learning procedure that is consistent under the assumption. In practice this rewrites the existing PU learning estimators into a form that has the bias built into it.

Main results and conditions

We show theoretically that the procedure learns a classifier under the stated assumption, and experiments on benchmark data confirm that it classifies biased data better than methods that assume random labeling. Where the bias is strong enough that the assumption fails, the guarantee does not carry over as it stands. The exact statement of the method and of the assumption is in the published version.

Relation to earlier work

The formulation of PU learning starts with Elkan and Noto (2008, Learning classifiers from only positive and unlabeled data). du Plessis and coauthors put the learning theory on a statistical risk estimation footing (2015, Convex Formulation for Learning from Positive and Unlabeled Data), and Kiryo and coauthors added a non-negative risk estimator to control overfitting in deep models (2017, Positive-Unlabeled Learning with Non-Negative Risk Estimator). All of these assume that the labeled positives were selected completely at random, and our work extends this line by taking on a setting where that assumption is relaxed. The survey by Bekker and Davis gives the wider map of PU learning (2020, Learning from positive and unlabeled data: a survey).

Where it applies

It suits classification on data where positive labels accumulate in the order things are found, so that the known cases are existing customers, diagnosed patients, or fraud that has already been caught. Later work building on this study carries the idea into identifying latent customers and into treatment effect estimation, and the Weakly Supervised Learning page traces that line.

Paper and materials

BibTeX

@inproceedings{pu-learning-selection-bias,
  author       = {Masahiro Kato and Takeshi Teshima and Junya Honda},
  title        = {Learning from Positive and Unlabeled Data with a Selection Bias},
  year         = {2019},
  booktitle    = {International Conference on Learning Representations (ICLR)},
  url          = {https://openreview.net/forum?id=rJzLciCqKm},
}