+---------------------------------------------------------------------------+
|                          Information Theory                               |
+---------------------------------------------------------------------------+

began with Claude Shannon's "The Mathematical Theory of Communication"
concerned with capacity of communications channels in terms of bits

Information Entropy

	entropy: 
		* a quantitative measure of the amount of thermal energy not
		  available to do work
		* the measure of the disorder or randomness in a closed system
	
	randomness in a signal; how much information is carred by a signal?

	H = - (sigma(i)) p(i) log2(p(i))

	log base 2 is used for bits; it is the size of tha alphabet

	this represents the odds of receiving a correct bit or set of bits over
	a transmission

	Maxwell's Demon
		
	Mutual Information
		Statistical dependence between two random variables
		H(x,Y) = - (sigma(x,y)) p(x,y) log(p(x,y))

	Conditional Entropy
		H(X|Y) = amount of uncertainty in X that is eliminated by
		observations of Y and vice versa
	
	Probability
		
		


----------------------------------------------------------------------------
# $Id$
# ex: set tabstop=4 noet textwidth=78:
----------------------------------------------------------------------------