This tutorial steps through the ideas from Information Theory that eventually lead to Information Gain...one of the most popular measures of association currently used in data mining. We visit the ideas of Entropy and Conditional Entropy along the way. Look at the lecture on Gaussians for discussion of Entropy in the case of continuous probability density functions.

Download Slides PDF