Fast and Accurate Least-Mean-Squares Solvers for High Dimensional Data

Alaa Maalouf, Ibrahim Jubran, Dan Feldman

Research output: Contribution to journalArticlepeer-review

Abstract

Least-mean-squares (LMS) solvers such as Linear / Ridge-Regression and SVD not only solve fundamental machine learning problems, but are also the building blocks in a variety of other methods, such as matrix factorizations. We suggest an algorithm that gets a finite set of nn dd-dimensional real vectors and returns a subset of d+1d+1 vectors with positive weights whose weighted sum is exactly the same. The constructive proof in Caratheodory's Theorem computes such a subset in O(n2d2)O(n2d2) time and thus not used in practice. Our algorithm computes this subset in O(nd+d4\log {n})O(nd+d4logn) time, using O(log n) O(logn) calls to Caratheodory's construction on small but 'smart' subsets. This is based on a novel paradigm of fusion between different data summarization techniques, known as sketches and coresets. For large values of dd, we suggest a faster construction that takes O(nd) O(nd) time and returns a weighted subset of O(d) O(d) sparsified input points. Here, a sparsified point means that some of its entries were set to zero. As an application, we show how to boost the performance of existing LMS solvers, such as those in scikit-learn library, up to x100. Generalization for streaming and distributed data is trivial. Extensive experimental results and open source code are provided.

Original languageEnglish
Pages (from-to)9977-9994
Number of pages18
JournalIEEE Transactions on Pattern Analysis and Machine Intelligence
Volume44
Issue number12
DOIs
StatePublished - 1 Dec 2022

Bibliographical note

Publisher Copyright:
© 1979-2012 IEEE.

Keywords

  • Big Data
  • Caratheodory's Theorem
  • Coresets
  • Least Mean Squares Solvers
  • Regression
  • Sketches

ASJC Scopus subject areas

  • Software
  • Artificial Intelligence
  • Applied Mathematics
  • Computer Vision and Pattern Recognition
  • Computational Theory and Mathematics

Fingerprint

Dive into the research topics of 'Fast and Accurate Least-Mean-Squares Solvers for High Dimensional Data'. Together they form a unique fingerprint.

Cite this