Discovering Reliable Dependencies from Data: Hardness and Improved Algorithms

Panagiotis Mandros; Mario Boley; Jilles Vreeken

doi:10.1109/ICDM.2018.00047

Discovering Reliable Dependencies from Data: Hardness and Improved Algorithms

Panagiotis Mandros, Mario Boley, Jilles Vreeken

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

The reliable fraction of information is an attractive score for quantifying (functional) dependencies in high-dimensional data. In this paper, we systematically explore the algorithmic implications of using this measure for optimization. We show that the problem is NP-hard, which justifies the usage of worst-case exponential-time as well as heuristic search methods. We then substantially improve the practical performance for both optimization styles by deriving a novel admissible bounding function that has an unbounded potential for additional pruning over the previously proposed one. Finally, we empirically investigate the approximation ratio of the greedy algorithm and show that it produces highly competitive results in a fraction of time needed for complete branch-and-bound style search.

Original language	English
Title of host publication	2018 IEEE International Conference on Data Mining, ICDM 2018
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	317-326
Number of pages	10
ISBN (Electronic)	9781538691588
DOIs	https://doi.org/10.1109/ICDM.2018.00047
State	Published - 27 Dec 2018
Externally published	Yes
Event	18th IEEE International Conference on Data Mining, ICDM 2018 - Singapore, Singapore Duration: 17 Nov 2018 → 20 Nov 2018

Publication series

Name	Proceedings - IEEE International Conference on Data Mining, ICDM
Volume	2018-November
ISSN (Print)	1550-4786

Conference

Conference	18th IEEE International Conference on Data Mining, ICDM 2018
Country/Territory	Singapore
City	Singapore
Period	17/11/18 → 20/11/18

Bibliographical note

Publisher Copyright:
© 2018 IEEE.

Keywords

Approximate functional dependency
Branch-and-bound
Information theory
Knowledge discovery
Optimization

ASJC Scopus subject areas

General Engineering

Access to Document

10.1109/ICDM.2018.00047

Cite this

Mandros, P., Boley, M., & Vreeken, J. (2018). Discovering Reliable Dependencies from Data: Hardness and Improved Algorithms. In 2018 IEEE International Conference on Data Mining, ICDM 2018 (pp. 317-326). Article 8594856 (Proceedings - IEEE International Conference on Data Mining, ICDM; Vol. 2018-November). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICDM.2018.00047

@inproceedings{e12e10a7b52f484ebadfa88d55a74a25,

title = "Discovering Reliable Dependencies from Data: Hardness and Improved Algorithms",

abstract = "The reliable fraction of information is an attractive score for quantifying (functional) dependencies in high-dimensional data. In this paper, we systematically explore the algorithmic implications of using this measure for optimization. We show that the problem is NP-hard, which justifies the usage of worst-case exponential-time as well as heuristic search methods. We then substantially improve the practical performance for both optimization styles by deriving a novel admissible bounding function that has an unbounded potential for additional pruning over the previously proposed one. Finally, we empirically investigate the approximation ratio of the greedy algorithm and show that it produces highly competitive results in a fraction of time needed for complete branch-and-bound style search.",

keywords = "Approximate functional dependency, Branch-and-bound, Information theory, Knowledge discovery, Optimization",

author = "Panagiotis Mandros and Mario Boley and Jilles Vreeken",

note = "Publisher Copyright: {\textcopyright} 2018 IEEE.; 18th IEEE International Conference on Data Mining, ICDM 2018 ; Conference date: 17-11-2018 Through 20-11-2018",

year = "2018",

month = dec,

day = "27",

doi = "10.1109/ICDM.2018.00047",

language = "English",

series = "Proceedings - IEEE International Conference on Data Mining, ICDM",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "317--326",

booktitle = "2018 IEEE International Conference on Data Mining, ICDM 2018",

address = "United States",

}

Mandros, P, Boley, M & Vreeken, J 2018, Discovering Reliable Dependencies from Data: Hardness and Improved Algorithms. in 2018 IEEE International Conference on Data Mining, ICDM 2018., 8594856, Proceedings - IEEE International Conference on Data Mining, ICDM, vol. 2018-November, Institute of Electrical and Electronics Engineers Inc., pp. 317-326, 18th IEEE International Conference on Data Mining, ICDM 2018, Singapore, Singapore, 17/11/18. https://doi.org/10.1109/ICDM.2018.00047

Discovering Reliable Dependencies from Data: Hardness and Improved Algorithms. / Mandros, Panagiotis; Boley, Mario; Vreeken, Jilles.
2018 IEEE International Conference on Data Mining, ICDM 2018. Institute of Electrical and Electronics Engineers Inc., 2018. p. 317-326 8594856 (Proceedings - IEEE International Conference on Data Mining, ICDM; Vol. 2018-November).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Discovering Reliable Dependencies from Data

T2 - 18th IEEE International Conference on Data Mining, ICDM 2018

AU - Mandros, Panagiotis

AU - Boley, Mario

AU - Vreeken, Jilles

PY - 2018/12/27

Y1 - 2018/12/27

N2 - The reliable fraction of information is an attractive score for quantifying (functional) dependencies in high-dimensional data. In this paper, we systematically explore the algorithmic implications of using this measure for optimization. We show that the problem is NP-hard, which justifies the usage of worst-case exponential-time as well as heuristic search methods. We then substantially improve the practical performance for both optimization styles by deriving a novel admissible bounding function that has an unbounded potential for additional pruning over the previously proposed one. Finally, we empirically investigate the approximation ratio of the greedy algorithm and show that it produces highly competitive results in a fraction of time needed for complete branch-and-bound style search.

AB - The reliable fraction of information is an attractive score for quantifying (functional) dependencies in high-dimensional data. In this paper, we systematically explore the algorithmic implications of using this measure for optimization. We show that the problem is NP-hard, which justifies the usage of worst-case exponential-time as well as heuristic search methods. We then substantially improve the practical performance for both optimization styles by deriving a novel admissible bounding function that has an unbounded potential for additional pruning over the previously proposed one. Finally, we empirically investigate the approximation ratio of the greedy algorithm and show that it produces highly competitive results in a fraction of time needed for complete branch-and-bound style search.

KW - Approximate functional dependency

KW - Branch-and-bound

KW - Information theory

KW - Knowledge discovery

KW - Optimization

UR - http://www.scopus.com/inward/record.url?scp=85061397133&partnerID=8YFLogxK

U2 - 10.1109/ICDM.2018.00047

DO - 10.1109/ICDM.2018.00047

M3 - Conference contribution

AN - SCOPUS:85061397133

T3 - Proceedings - IEEE International Conference on Data Mining, ICDM

SP - 317

EP - 326

BT - 2018 IEEE International Conference on Data Mining, ICDM 2018

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 17 November 2018 through 20 November 2018

ER -

Discovering Reliable Dependencies from Data: Hardness and Improved Algorithms

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this