Low-Rank Methods in Julia

Abstract

The huge memory requirement and high computational cost of cutting-edge models limit the advancement in deep neural networks, especially when such models are deployed in resource-constrained environments. Using low-rank methods can be a good solution to this problem as they reduce the memory and computational requirements by restricting the parameter space to the manifold of low-rank matrices. Low-rank methods have been implemented in Python. In this thesis, the low-rank methods are implemented in Julia and evaluated on the MNIST dataset. These low-rank methods are implemented as vanilla low-rank training and dynamical low-rank training where dynamical low-rank training is further implemented as a fixed-rank and rank-adaptive approach. Then, the results are compared with those from the Python implementation. At last, the low-rank methods are applied to the stopped-flow biochemistry dataset to predict the rate constant for the chemical reaction.

This thesis highlights the advantages and costs of adding momentum and gradient clipping to the training algorithm. Further, the study compares the fixed-rank and the rank-adaptive approaches of dynamical low-rank training in terms of accuracy, timing, and memory requirement. The findings from the comparison between Python and Julia in terms of timing and accuracy show that Julia can be used as a tool for optimizing neural networks, however, Julia cannot save much time as it takes more epochs to converge compared to Python. This thesis also demonstrates numerically how low-rank methods are saving memory resources for both standard datasets like the MNIST dataset and real-world datasets like the stopped-flow data.

In summary, this thesis provides detailed results and a comprehensive discussion on the performance of various low-rank methods in Julia for a standard dataset like the MNIST dataset and a real-world dataset like the stoped-flow biochemistry data.

The huge memory requirement and high computational cost of cutting-edge models limit the advancement in deep neural networks, especially when such models are deployed in resource-constrained environments. Using low-rank methods can be a good solution to this problem as they reduce the memory and computational requirements by restricting the parameter space to the manifold of low-rank matrices. Low-rank methods have been implemented in Python. In this thesis, the low-rank methods are implemented in Julia and evaluated on the MNIST dataset. These low-rank methods are implemented as vanilla low-rank training and dynamical low-rank training where dynamical low-rank training is further implemented as a fixed-rank and rank-adaptive approach. Then, the results are compared with those from the Python implementation. At last, the low-rank methods are applied to the stopped-flow biochemistry dataset to predict the rate constant for the chemical reaction.

This thesis highlights the advantages and costs of adding momentum and gradient clipping to the training algorithm. Further, the study compares the fixed-rank and the rank-adaptive approaches of dynamical low-rank training in terms of accuracy, timing, and memory requirement. The findings from the comparison between Python and Julia in terms of timing and accuracy show that Julia can be used as a tool for optimizing neural networks, however, Julia cannot save much time as it takes more epochs to converge compared to Python. This thesis also demonstrates numerically how low-rank methods are saving memory resources for both standard datasets like the MNIST dataset and real-world datasets like the stopped-flow data.

In summary, this thesis provides detailed results and a comprehensive discussion on the performance of various low-rank methods in Julia for a standard dataset like the MNIST dataset and a real-world dataset like the stoped-flow biochemistry data.