Heterogeneous Decomposition of Convolutional Neural Networks Using Tucker Decomposition

Abstract

Convolutional Neural Network (CNN) remain the architecture of choice for computer vision tasks on compute-constrained platforms such as edge and personal devices, delivering both close to state-of-the-art performance metrics and linear inference complexity with respect to input resolution and number of channels. However, the deployment of larger and more complex CNN architectures is limited by the restrained memory offered by such platforms. This brings about a need to compress pretrained CNN into smaller models in number of parameters while controlling for degradation in performance. This thesis tackles CNN compression using low rank approximation of convolution layers using Tucker Decomposition (TD). We introduce a new heuristics-based Neural Architectural Search procedure to select low rank configurations for the convolution tensors, which we call Heterogeneous Tucker Decomposition (HTD).

Standard low rank approximation using TD factorizes and approximates convolution layers using uniform ranks for all convolution tensors, then applies a few fine–tuning epochs to recover degradation in performance. An approach we show to be suboptimal against a heterogeneous selection of ranks for each convolution layer, followed by same number of fine-tuning epochs.

Our primary contribution is the development and evaluation of TD, which applies layers-pecific compression rate (low rank divided by full rank) inferred from a Neural Architectural Search (NAS) process. Furthermore, we introduce a sampling heuristic to efficiently explore the search space of layer-specific compression rates, thus preserving performance while significantly reducing search time.

We present a mathematical formulation for the HTD optimization problem and an NAS algorithm to find admissible solutions. We test our approach on multiple varieties of CNN architectures: AlexNet, VGG16, and ResNet18, adapted for the MNIST classification task. Our findings confirm that HTD performs better than TD on all models tested. For the same compression rate, HTD enables to recover a higher precision after fine-tuning, with gains ranging from 1.2% to 5.8%. For equivalent accuracy targets, HTD delivers 15-30% higher compression rates than TD.

This thesis advances Neural Architectural Search by highlighting the efficacy of heterogeneous tensor decomposition approaches. It provides a robust framework for their implementation and evaluation, with significant implications for deploying convolutional deep learning models in resource-limited settings. Future work will explore incorporating low-rank constraints as a regularization objective during training, potentially enabling end-to-end compression-aware optimization.