Learning And Generalization In Feed Forward Neural Networks

Author : Adrian Tovar Lopez
Publisher :
ISBN 13 :
Total Pages : 0 pages
Book Rating : 4.:/5 (134 download)

DOWNLOAD NOW!

Download or read book Information-theoretic Perspectives on Generalization and Robustness of Neural Networks written by Adrian Tovar Lopez and published by . This book was released on 2022 with total page 0 pages. Available in PDF, EPUB and Kindle. Book excerpt: Neural networks as efficient as they are in practice, remain in several aspects still a mystery. Some of the most studied questions are: where does their generalization capabilities come from? What are the reason behind the existence of adversarial examples? In this thesis I use a formal mathematical representation of neural networks to investigate this questions. I also develop new algorithms based on the theory developed. The first par of the thesis is concerned with the generalization error which characterizes the gap between an algorithm's performance on test data versus performance on training data. I derive upper bounds on the generalization error in terms of a certain Wasserstein distance involving the distributions of input and the output under the assumption of a Lipschitz continuous loss function. Unlike mutual information-based bounds, these new bounds are useful for algorithms such as stochastic gradient descent. Moreover, I show that in some natural cases these bounds are tighter than mutual information-based bounds. In the second part of the thesis I study manifold learning. The goal is to learn a manifold that captures the inherent low-dimensionality of high-dimensional data. I present a novel training procedure to learn manifolds using neural networks. Parametrizing the manifold via a neural network with a low-dimensional input and a high-dimensional output. During training, I calculate the distance between the training data points and the manifold via a geometric projection and update the network weights so that this distance diminishes. The learned manifold is seen to interpolate the training data, analogous to autoencoders. Experiments show that the procedure leads to lower reconstruction errors for noisy inputs, and higher adversarial accuracy when used in manifold defense methods than those of autoencoders. In the final part of the thesis I propose an information bottleneck principle for causal time-series prediction. I develop variational bounds on the information bottleneck objective function that can be efficiently optimized using recurrent neural networks. Then implement an algorithm on simulated data as well as real-world weather-prediction and stock market-prediction datasets and show that these problems can be successfully solved using the new information bottleneck principle.