CMNN: Coupled Modular Neural Network

CMNN: Coupled Modular Neural Network

Blog Article

In this paper, we propose a multi-branch neural network architecture named Coupled Modular Neural Network (CMNN).A CMNN is a network consisting of $eta $ closely coupled sub-networks, where $eta $ is termed as the branching factor in this paper.We call the whole network a super-graph and each sub-network a sub-graph.Each sub-graph is a stand-alone neural network and shares a common block grip right with other sub-graphs.

To effectively leverage the super-graph we propose a simple but easy-to-implement Round-Robin-based learning algorithm.Each training iteration contains two phases.In the first phase, we choose a sub-graph in a Round-Robin fashion and train it using knowledge of the super-graph (distillation).In the second phase, we fine-tune the super-graph based on the updated sub-graphs.

This algorithm produces a different copy of Suspension the super-graph at each iteration which acts as an improved teacher network for the sub-graph; and a different copy of one of the sub-graphs which functions as a new building block for the super-graph.To validate and test CMNN and the proposed algorithm, we conduct experiments on CIFAR-10, CIFAR-100, Tiny ImageNet and a private On-Road-Risk (ORR) datasets.Empirical results on all these four datasets indicate that we not only obtain a strong sub-graph network, the learning framework can also produce strong ensemble performance which substantiates the diversity introduced throughout the learning framework.