Variance Based Samples Weighting for Supervised Deep Learning

Paul Novello (1, 2, 3) , Gaël Poëtte (2) , David Lugato (4) , Pietro Marco Congedo (1, 3)

1 CMAP - Centre de Mathématiques Appliquées - Ecole Polytechnique
2 CEA - Commissariat à l'énergie atomique et aux énergies alternatives
3 DeFI - Shape reconstruction and identification
4 CESTA - Centre d'études scientifiques et techniques d'Aquitaine

Machine Learning (ML) aims at approximating functions defined on a measured space with a model. A relevant choice of distribution for the training data set can improve the performances of a given ML model. We claim and empirically justify that an ML model yields better results when the data set focuses on regions where the function to learn is steeper. We first traduce this assumption in a mathematically workable way. Then, theoretical derivations allow to construct a methodology that we call Variance Based Samples Weighting (VBSW). VBSW uses local variance of the labels to weight the training points. This methodology is general, scalable and cost effective. It is validated on Deep Learning models like Bert [10] or ResNet [14] by significantly increasing their performances for various Natural Language Processing and image classification tasks.

Domaines: Mathématiques [math]; Statistiques [math.ST]; Statistiques [stat]; Machine Learning [stat.ML]; Théorie [stat.TH]

Paul Novello, Gaël Poëtte, David Lugato, Pietro Marco Congedo. Variance Based Samples Weighting for Supervised Deep Learning. 2020. ⟨hal-02885827v1⟩