On Symmetry and Initialization for Neural Networks

Ido Nachum; Amir Yehudayoff

doi:10.1007/978-3-030-61792-9_32

On Symmetry and Initialization for Neural Networks

Ido Nachum, Amir Yehudayoff

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

This work provides an additional step in the theoretical understanding of neural networks. We consider neural networks with one hidden layer and show that when learning symmetric functions, one can choose initial conditions so that standard SGD training efficiently produces generalization guarantees. We empirically verify this and show that this does not hold when the initial conditions are chosen at random. The proof of convergence investigates the interaction between the two layers of the network. Our results highlight the importance of using symmetry in the design of neural networks.

Original language	English
Title of host publication	LATIN 2020
Subtitle of host publication	Theoretical Informatics - 14th Latin American Symposium 2021, Proceedings
Editors	Yoshiharu Kohayakawa, Flávio Keidi Miyazawa
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	401-412
Number of pages	12
ISBN (Print)	9783030617912
DOIs	https://doi.org/10.1007/978-3-030-61792-9_32
State	Published - 2020
Externally published	Yes
Event	14th Latin American Symposium on Theoretical Informatics, LATIN 2020 - Sao Paulo, Brazil Duration: 5 Jan 2021 → 8 Jan 2021

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	12118 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	14th Latin American Symposium on Theoretical Informatics, LATIN 2020
Country/Territory	Brazil
City	Sao Paulo
Period	5/01/21 → 8/01/21

Bibliographical note

Publisher Copyright:
© 2020, Springer Nature Switzerland AG.

Keywords

Neural networks
Symmetry

ASJC Scopus subject areas

Theoretical Computer Science
General Computer Science

Access to Document

10.1007/978-3-030-61792-9_32

Cite this

Nachum, I., & Yehudayoff, A. (2020). On Symmetry and Initialization for Neural Networks. In Y. Kohayakawa, & F. K. Miyazawa (Eds.), LATIN 2020: Theoretical Informatics - 14th Latin American Symposium 2021, Proceedings (pp. 401-412). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12118 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-030-61792-9_32

Nachum, Ido ; Yehudayoff, Amir. / On Symmetry and Initialization for Neural Networks. LATIN 2020: Theoretical Informatics - 14th Latin American Symposium 2021, Proceedings. editor / Yoshiharu Kohayakawa ; Flávio Keidi Miyazawa. Springer Science and Business Media Deutschland GmbH, 2020. pp. 401-412 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{65f41d5485a1474d9181542754b287dc,

title = "On Symmetry and Initialization for Neural Networks",

abstract = "This work provides an additional step in the theoretical understanding of neural networks. We consider neural networks with one hidden layer and show that when learning symmetric functions, one can choose initial conditions so that standard SGD training efficiently produces generalization guarantees. We empirically verify this and show that this does not hold when the initial conditions are chosen at random. The proof of convergence investigates the interaction between the two layers of the network. Our results highlight the importance of using symmetry in the design of neural networks.",

keywords = "Neural networks, Symmetry",

author = "Ido Nachum and Amir Yehudayoff",

note = "Publisher Copyright: {\textcopyright} 2020, Springer Nature Switzerland AG.; 14th Latin American Symposium on Theoretical Informatics, LATIN 2020 ; Conference date: 05-01-2021 Through 08-01-2021",

year = "2020",

doi = "10.1007/978-3-030-61792-9_32",

language = "English",

isbn = "9783030617912",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "401--412",

editor = "Yoshiharu Kohayakawa and Miyazawa, {Fl{\'a}vio Keidi}",

booktitle = "LATIN 2020",

address = "Germany",

}

Nachum, I & Yehudayoff, A 2020, On Symmetry and Initialization for Neural Networks. in Y Kohayakawa & FK Miyazawa (eds), LATIN 2020: Theoretical Informatics - 14th Latin American Symposium 2021, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 12118 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 401-412, 14th Latin American Symposium on Theoretical Informatics, LATIN 2020, Sao Paulo, Brazil, 5/01/21. https://doi.org/10.1007/978-3-030-61792-9_32

On Symmetry and Initialization for Neural Networks. / Nachum, Ido; Yehudayoff, Amir.
LATIN 2020: Theoretical Informatics - 14th Latin American Symposium 2021, Proceedings. ed. / Yoshiharu Kohayakawa; Flávio Keidi Miyazawa. Springer Science and Business Media Deutschland GmbH, 2020. p. 401-412 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12118 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - On Symmetry and Initialization for Neural Networks

AU - Nachum, Ido

AU - Yehudayoff, Amir

PY - 2020

Y1 - 2020

N2 - This work provides an additional step in the theoretical understanding of neural networks. We consider neural networks with one hidden layer and show that when learning symmetric functions, one can choose initial conditions so that standard SGD training efficiently produces generalization guarantees. We empirically verify this and show that this does not hold when the initial conditions are chosen at random. The proof of convergence investigates the interaction between the two layers of the network. Our results highlight the importance of using symmetry in the design of neural networks.

AB - This work provides an additional step in the theoretical understanding of neural networks. We consider neural networks with one hidden layer and show that when learning symmetric functions, one can choose initial conditions so that standard SGD training efficiently produces generalization guarantees. We empirically verify this and show that this does not hold when the initial conditions are chosen at random. The proof of convergence investigates the interaction between the two layers of the network. Our results highlight the importance of using symmetry in the design of neural networks.

KW - Neural networks

KW - Symmetry

UR - http://www.scopus.com/inward/record.url?scp=85097715013&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-61792-9_32

DO - 10.1007/978-3-030-61792-9_32

M3 - Conference contribution

AN - SCOPUS:85097715013

SN - 9783030617912

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 401

EP - 412

BT - LATIN 2020

A2 - Kohayakawa, Yoshiharu

A2 - Miyazawa, Flávio Keidi

PB - Springer Science and Business Media Deutschland GmbH

T2 - 14th Latin American Symposium on Theoretical Informatics, LATIN 2020

Y2 - 5 January 2021 through 8 January 2021

ER -

Nachum I, Yehudayoff A. On Symmetry and Initialization for Neural Networks. In Kohayakawa Y, Miyazawa FK, editors, LATIN 2020: Theoretical Informatics - 14th Latin American Symposium 2021, Proceedings. Springer Science and Business Media Deutschland GmbH. 2020. p. 401-412. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-61792-9_32

On Symmetry and Initialization for Neural Networks

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this