TY - JOUR
T1 - A geometric approach to monitoring threshold functions over distributed data streams
AU - Sharfman, Izchak
AU - Schuster, Assaf
AU - Keren, Daniel
PY - 2007/11/1
Y1 - 2007/11/1
N2 - Monitoring data streams in a distributed system is the focus of much research in recent years. Most of the proposed schemes, however, deal with monitoring simple aggregated values, such as the frequency of appearance of items in the streams. More involved challenges, such as the important task of feature selection (e.g., by monitoring the information gain of various features), still require very high communication overhead using naive, centralized algorithms. We present a novel geometric approach which reduces monitoring the value of a function (vis - vis a threshold) to a set of constraints applied locally on each of the streams. The constraints are used to locally filter out data increments that do not affect the monitoring outcome, thus avoiding unnecessary communication. As a result, our approach enables monitoring of arbitrary threshold functions over distributed data streams in an efficient manner. We present experimental results on real-world data which demonstrate that our algorithms are highly scalable, and considerably reduce communication load in comparison to centralized algorithms.
AB - Monitoring data streams in a distributed system is the focus of much research in recent years. Most of the proposed schemes, however, deal with monitoring simple aggregated values, such as the frequency of appearance of items in the streams. More involved challenges, such as the important task of feature selection (e.g., by monitoring the information gain of various features), still require very high communication overhead using naive, centralized algorithms. We present a novel geometric approach which reduces monitoring the value of a function (vis - vis a threshold) to a set of constraints applied locally on each of the streams. The constraints are used to locally filter out data increments that do not affect the monitoring outcome, thus avoiding unnecessary communication. As a result, our approach enables monitoring of arbitrary threshold functions over distributed data streams in an efficient manner. We present experimental results on real-world data which demonstrate that our algorithms are highly scalable, and considerably reduce communication load in comparison to centralized algorithms.
KW - Distributed monitoring
UR - http://www.scopus.com/inward/record.url?scp=36949013219&partnerID=8YFLogxK
U2 - 10.1145/1292609.1292613
DO - 10.1145/1292609.1292613
M3 - Article
AN - SCOPUS:36949013219
SN - 0362-5915
VL - 32
JO - ACM Transactions on Database Systems
JF - ACM Transactions on Database Systems
IS - 4
M1 - 23
ER -