Machine Learning: Naive Bayes

 

In the previous chapter, we studied Decision Tree.

 

In this chapter, we will study naive bayes.

 

 

Key Terms

 

Conditional Probability

Let A and B be two events associated with a random experiment, then, the probability of occurrence of event A under the condition that B has already occurred and P(B) ≠ 0, is called the conditional probability. 

i.e., P(A/B) = P(A ∩ B) / P(B)

 

If A has already occurred and P (A) ≠ 0, then

P(B/A) = P(A ∩ B) / P(A)

 

Also, P(A / B) + P (A / B) = 1

 

Multiplication Theorem on Probability

(i) If A and B are two events associated with a random experiment, then

P(A ∩ B) = P(A)P(B /A), IF P(A) ≠ 0

OR

P(A ∩ B) = P(B)P(A /B), IF P(B) ≠ 0

 

(ii) If A1, A2,…., An are n events associated with a random experiment, then 

P(A1 ∩ A2 ∩…. ∩ An) = P(A1) P(A2 / A1) P(A3 / (A1 ∩ A2)) …P(An / (A1 ∩ A2 ∩ A3 ∩…∩A n – 1))

 

Total Probability

Let S be the sample space and let E1, E2,…., En be n mutually exclusive and exhaustive events associated with a random experiment. If A is any event which occurs with E1 or E2 or … or En then

 

P(A) = P(E1)P(A / E1) + P(E2)P(A / E2) + … + P(En) P(A / En)

 

Baye’s Theorem

Let S be the sample space and let E1, E2,…, En, be n mutually exclusive and exhaustive events associated With a random experiment. If A is any event which occurs with E1 or E2 or … or En then the probability of occurrence of Ei, when A occurred,

 

where,

1. P ( A ∣ B ) is a conditional probability: the likelihood of event A occurring given that B is true.
2. P ( B ∣ A ) is also a conditional probability: the likelihood of event B occurring given that A is true.
3. P ( A ) and P ( B ) are the probabilities of observing A and Bindependently of each other; this is known as the marginal probability.

Random Variable

Let U or S be a sample space associated with a given random experiment. A real-valued function X defined on U or S, i:e.,

 

X: U → R is called a random variable.

 

There are two types of random variables.

 

If the range of the real function X: U → R is a finite set or an infinite set of real numbers, it is called a discrete random variable.

 

If the range of X is an interval (a, b) of R, then X is called a continuous random variable. e.g., In tossing of two coins S = {HH, HT, TH, TT}, let X denotes the number of heads in the tossing of two coins, then X(HH) = 2, X(TH) = 1, X(TT) = 0

 

Probability Distribution

If a random variable X takes values X1, X2,…., Xn with respective probabilities P1, P2,…., Pn then is known as the probability distribution of X, or Probability distribution gives the values of the random variable along with the corresponding probabilities.

 

Mathematical Expectation/Mean

If X is a discrete random variable which assume values X1, X2,…., Xn with respective probabilities P1, P2,…., Pn then the mean x of X is defined as

E(X) = X = P1X1 + P2X2 + … + PnXn = Σni = 1 PiXi

 

 

(i) Variance V(X) = σ2x = E(X2) – (E(X))2

where, E(X2) = Σni = 1 x2iP(xi)

 

(ii) Standard Deviation √V(X) = σx = √E(X2) – (E(X))2

 

(iii) If Y = a X + b, then

(a) E(Y) = E(aX + b) = aE(X) + b

(b) σ2y = a2V(Y) = a2σ2x

(c) σy = √V(Y) = |a|σx

 

(iv) If Z = aX2 + bX + c, then

E(Z) = E(aX2 + bX + c) = aE(X2) + bE(X) + c 

 

Baye's Theorem Explanation using an example

 

The naive Bayes classifier is an approximation to the Bayes classifier, in which we assume that the features are conditionally independent given the class instead of modeling their full conditional distribution given the class. A Bayes classifier is best interpreted as a decision rule. Suppose we seek to estimate the class of ("classify") an observation is given a vector of features. Denote the class C and the vector of features (F1, F2,…, Fk). Given a probability model underlying the data (that is, given the joint distribution of (C, F1, F2,…, Fk), the Bayes classification function chooses a class by maximizing the probability of the class given the observed features: argmaxc P(C=c∣F1=f1,…,Fk=fk)  

 

1. All the features are independent, that is there are no dependencies between any of the features.
2. Each of the features is given equal or the same importance or equal weight

 

 

 

Type	Long	Not Long	Sweet	Not Sweet	Yellow	Not Yellow	Total
Banana	400	100	350	150	450	50	500
Orange	0	300	150	150	300	0	300
Other	100	100	150	50	50	150	200
Total	500	500	650	350	800	200	1000

 

So the objective of the classifier is to predict if a given fruit is a ‘Banana’ or ‘Orange’ or ‘Other’ when only the 3 features (long, sweet, and yellow) are known.

 

So to predict this we need to find 3 probabilities. Let's start

 

1. We first calculate the "Prior" probabilities for each of the class of fruits

P[Y=Banana] = 500/1000 = 0.5

P[Y=Orange] = 300/1000 = 0.3

P[Y=Other] = 200/1000 = 0.2

 

2. We then compute the probability of evidence that goes in the denominator

P[x1=Long] =  500/100 = 0.5

P[x2=Sweet] = 650/100 = 0.65

P[x3=Yellow] = 800/100 = 0.8

 

3. Now we calculate the probability of likelihood of evidences that goes in the numerator

P[x1=Long | Y=Banana] = 400/500 = 0.8

P[x2=Sweet | Y=Banana] = 350/500 = 0.7

P[x3=Yellow | Y=Banana] = 450/500 = 0.9

 

4. At the end we substitute all the values in the Naive Bayes Formula,

 

a. P(Banana | Long, Sweet and Yellow)

= ((P(Long | Banana) * P(Sweet | Banana) * P(Yellow | Banana))*P(Banana))/ (P(Long) * P(Sweet) * P(Yellow))

= ((0.8 * 0.7 * 0.9) * 0.5)/(0.5 * 0.65 * 0.8)

=  0.97

 

b. P(Orange | Long, Sweet and Yellow)

=  ((P(Long | Orange) * P(Sweet | Orange) * P(Yellow | Orange))*P(Orange))/ (P(Long) * P(Sweet) * P(Yellow))

= 0

 

c. P(Others | Long, Sweet and Yellow)

= ((P(Long | Others) * P(Sweet | Others) * P(Yellow | Others))*P(Others))/ (P(Long) * P(Sweet) * P(Yellow))

= 0.07

 

So, from the Naive Bayes Classifier, we predict the fruit is a Banana.

 

Python Implementation of Decision Tree

1. Using functions

The output that I am getting is

 

[Iris-versicolor] => 0

[Iris-setosa] => 1

[Iris-virginica] => 2

 

Data=[5.7, 2.9, 4.2, 1.3], Predicted: 0

 

2. Using Sklearn

Conclusion