Learning Compressible Models (2010)

Yi Zhang, Jeff Schneider, Artur Dubrawski

Abstract

In this paper, we study the combination of compression and $\ell1$-norm regularization in a machine learning context: learning compressible models. By including a compression operation into the $\ell1$ regularization, the assumption on model sparsity is relaxed to compressibility: model coefficients are compressed before being penalized, and sparsity is achieved in a compressed domain rather than the original space. We focus on the design of different compression operations, by which we can encode various compressibility assumptions and inductive biases, e.g., piecewise local smoothness, compacted energy in the frequency domain, and semantic correlation. We show that use of a compression operation provides an opportunity to leverage auxiliary information from various sources, e.g., domain knowledge, coding theories, unlabeled data. We conduct extensive experiments on brain-computer interfacing, handwritten character recognition and text classification. Empirical results show clear improvements in prediction performance by including compression in $\ell1$ regularization. We also analyze the learned model coefficients under appropriate compressibility assumptions, which further demonstrate the advantages of learning compressible models instead of sparse models.

Full text

Download (application/pdf, 180.1 kB)

Approximate BibTeX Entry

@inproceedings{Compressible_SDM10,
    Year = {2010},
    Booktitle = {In Proceedings of SIAM Data Mining (SDM) Conference},
    Author = { Yi Zhang, Jeff Schneider, Artur Dubrawski },
    Title = {Learning Compressible Models}
}