Reconsideration of Information-Theoretic Principles—Perspective from the Dual Probability Distribution

We reconsider information-theoretic principles, such as the maximum entropy principle/minimum Massieu potential principle, from the perspective of the dual probability distribution. This is introduced through Sanov's Lemma for the multinomial distribution. The dual correspondence becomes asymptotically manifest. The Massieu potential is rewritten as the Kullback-Leibler divergence between the dual probability distribution and the dual reference distribution. Similarly, the dual potential is rewritten as the cumulant generating function with respect to the dual reference distribution. This perspective gives us new insight into information-theoretic principles. As the dual probability distribution naturally arises in data sampling, we anticipate that this new perspective will play a significant role in data analysis.