Renjie Li, Xinyi Wang, Guan Huang, Wen‐Li Yang, Kaining Zhang, Xiaotong Gu, Son N. Tran, Saurabh Garg, Jane Alty, Quan Bai
Deep supervision, also known as ‘intermediate supervision’ or ‘auxiliary supervision’, involves adding supervision at hidden layers of a neural network. This technique has gained increasing attention in deep neural network learning systems for various computer vision applications in recent years. There is a consensus that deep supervision can improve neural network performance by alleviating the gradient vanishing problem, among other strengths. However, how to effectively utilize deep supervision to improve network performance in different computer vision applications has not been thoroughly investigated. In this paper, we provide a comprehensive and in-depth review of deep supervision in both theory and applications. We propose a new classification of different deep supervision networks and discuss the advantages and limitations of current deep supervision networks in computer vision applications.