1, Naive Bayes overview
Naive Bayesian method is a classification method based on Bayesian theorem and the assumption of feature conditional independence. For a given training set, firstly, the joint probability distribution of input and output is learned based on the independent assumption of characteristic conditions (the naive Bayes method, which obtains the model through learning, obviously belongs to the generative model); Then, based on this model, for a given input x, the output y with the maximum a posteriori probability is obtained by using Bayesian theorem
2, Basic formula of naive Bayes
1. Joint distribution rate
The joint probability is expressed as the probability that multiple conditions are included and all conditions are true at the same time, which is recorded as P (x = a, y = b) P (x = a, y = b) P (x = a, y = b) or P (a, b) P (a, b) or P (a, b) P (AB) P (AB)
2. Conditional probability
There is a jar containing 7 stones, of which 3 are white and 4 are black. If a stone is taken randomly from the jar, what is the probability that it is a white stone
Obviously, the probability of taking out white stone is 3 / 7 and the probability of taking out black stone is 4 / 7. We use P(white) The probability value can be obtained by dividing the number of white stones by the total number of stones.
Seven stones are placed in two barrels as shown in the figure. How should the conditional probability be calculated ？
To calculate P(white) or P(black), obviously, the information of the bucket where the stone is located will change the result, which is the conditional probability. Assuming that the calculation is the probability of taking the white stone from bucket B, this probability can be recorded as P (white|bucket b), which we call "the probability of taking the white stone when the stone is known to come from bucket B".
It is easy to get that the value of P (white|bucket a) is 2 / 4 and the value of P (white|bucket b) is 1 / 3.
The calculation formula of conditional probability is as follows:
Put it in our example: P (white|bucket b) = P (white and bucket B) / P (bucket B)
P (white|bucket b): the probability of taking out white stone under the condition that the stone is known to come from bucket B
P (white and bucket B): probability of taking out the white stone in bucket B = 1 / 7
P (bucket B): the probability of taking out the stone in bucket B is 3 / 7
3. Bayesian theorem
Another effective method to calculate conditional probability is called Bayesian theorem. Bayesian theorem tells us how to exchange the conditions and results in conditional probability, that is, if known P(X|Y), requirements P(Y|X):
P(Y): a priori probability. A priori probability means that something has not happened yet. It is a priori probability to find the possibility of this thing happening. It often appears as "cause" in the problem of "seeking result from cause".
P (Y ∣ X) P (Y | X) P (Y ∣ X): a posteriori probability. A posteriori probability refers to the probability that something has happened and the reason for it is caused by a certain factor. The calculation of a posteriori probability should be based on a priori probability
P (x ∣ Y) P (x | Y) P (x ∣ Y): conditional probability, also known as likelihood probability, is generally obtained through historical data statistics. Generally, it is not called a priori probability, but it also conforms to the a priori definition.
4. Naive Bayesian classifier
The naive Bayesian method learns the joint probability distribution P (x, y) P (x, y) P (x, y) through the training data set, which is actually learning the prior probability distribution and conditional probability distribution:
A priori probability distribution:
Conditional probability distribution:
Therefore, the joint probability distribution can be obtained from the conditional probability formula
In naive Bayesian classification, for a given input x, the posterior probability distribution P(Y = c_k|X = x) is calculated through the above learned model, and the class with the largest posterior probability is taken as the output of the class of X. The posterior probability is calculated according to Bayesian theorem:
Apply the full probability formula to the denominator:
Bring conditional independence hypothesis 4.3 into the above formula to obtain the basic formula of naive Bayesian classification:
The denominator in the above formula is the same for all categories and will not affect the calculation results. Therefore, the naive Bayesian classifier can be simplified to
3, Spam filtering
1. Steps of realizing spam classification by naive Bayes
(1) Collect data: provide text files.
(2) Prepare data: parse the text file into an entry vector.
(3) Analyze data: check entries to ensure the correctness of parsing.
(4) Training algorithm: calculate the conditional probability of different independent features.
(5) Test algorithm: calculate the error rate.
(6) Using algorithms: build a complete program to classify a group of documents
2. Data set download
There are two folders ham and spam under the email folder: Click to download the dataset
txt file in ham folder is normal mail;
txt file under spam file is spam;
3. Code implementation
# -*- coding: UTF-8 -*- import numpy as np import re import random """ Function description:The segmented experimental sample entries are sorted into a list of non repeated entries, that is, a vocabulary Parameters: dataSet - Collated sample data set Returns: vocabSet - Returns a list of non repeating entries, that is, a glossary """ def createVocabList(dataSet): vocabSet = set() # Create an empty non repeating list for document in dataSet: vocabSet = vocabSet | set(document) # Union set return list(vocabSet) """ Function description:according to vocabList Glossary, adding inputSet Vectorization, each element of the vector is 1 or 0 Parameters: vocabList - createVocabList List of returned inputSet - Segmented entry list Returns: returnVec - Document vector,Word set model """ def setOfWords2Vec(vocabList, inputSet): returnVec =  * len(vocabList) #Create a vector with 0 elements for word in inputSet: #Traverse each entry if word in vocabList: #If the entry exists in the vocabulary, set 1 returnVec[vocabList.index(word)] = 1 else: print("the word: %s is not in my Vocabulary!" % word) return returnVec #Return document vector """ Function description:according to vocabList Vocabulary, constructing word bag model Parameters: vocabList - createVocabList List of returned inputSet - Segmented entry list Returns: returnVec - Document vector,Word bag model """ def bagOfWords2VecMN(vocabList, inputSet): returnVec =  * len(vocabList) # Create a vector with 0 elements for word in inputSet: # Traverse each entry if word in vocabList: # If the entry exists in the vocabulary, the count is incremented by one returnVec[vocabList.index(word)] += 1 return returnVec # Return word bag model """ Function description:Naive Bayesian classifier training function Parameters: trainMatrix - Training document matrix, i.e setOfWords2Vec Returned returnVec Composition matrix trainCategory - Training category label vector, i.e loadDataSet Returned classVec Returns: p0Vect - Conditional probability array of normal mail class p1Vect - Conditional probability array of spam class pAbusive - Probability that the document belongs to spam """ def trainNB0(trainMatrix, trainCategory): numTrainDocs = len(trainMatrix) # Calculate the number of training documents numWords = len(trainMatrix) # Calculate the number of entries per document pAbusive = sum(trainCategory) / float(numTrainDocs) # Probability that the document belongs to spam p0Num = np.ones(numWords) p1Num = np.ones(numWords) # Create numpy.ones array, initialize the number of entries to 1, and Laplace smoothing p0Denom = 2.0 p1Denom = 2.0 # Denominator initialized to 2, Laplace smoothing for i in range(numTrainDocs): if trainCategory[i] == 1: # The data required for statistics of conditional probability belonging to insult class, i.e. P(w0|1),P(w1|1),P(w2|1)··· p1Num += trainMatrix[i] p1Denom += sum(trainMatrix[i]) else: # The data required to count the conditional probability belonging to the non insult class, i.e. P(w0|0),P(w1|0),P(w2|0)··· p0Num += trainMatrix[i] p0Denom += sum(trainMatrix[i]) p1Vect = np.log(p1Num / p1Denom) p0Vect = np.log(p0Num / p0Denom) #Take logarithm to prevent lower overflow return p0Vect, p1Vect, pAbusive # Return the conditional probability array belonging to normal mail class, the conditional probability array belonging to insulting spam class, and the probability that the document belongs to spam class """ Function description:Naive Bayesian classifier classification function Parameters: vec2Classify - Array of entries to be classified p0Vec - Conditional probability array of normal mail class p1Vec - Conditional probability array of spam class pClass1 - Probability that a document is spam Returns: 0 - It belongs to normal mail class 1 - Belong to spam """ def classifyNB(vec2Classify, p0Vec, p1Vec, pClass1): #p1 = reduce(lambda x, y: x * y, vec2Classify * p1Vec) * pClass1 # Multiply corresponding elements #p0 = reduce(lambda x, y: x * y, vec2Classify * p0Vec) * (1.0 - pClass1) p1=sum(vec2Classify*p1Vec)+np.log(pClass1) p0=sum(vec2Classify*p0Vec)+np.log(1.0-pClass1) if p1 > p0: return 1 else: return 0 """ Function description:Receives a large string and parses it into a list of strings """ def textParse(bigString): # Convert string to character list listOfTokens = re.split(r'\W*', bigString) # Use special symbols as segmentation marks to segment strings, that is, non alphabetic and non numeric return [tok.lower() for tok in listOfTokens if len(tok) > 2] # Except for a single letter, such as a capital I, other words become lowercase """ Function description:Test naive Bayes classifier and use naive Bayes for cross validation """ def spamTest(): docList =  classList =  fullText =  for i in range(1, 26): # Traverse 25 txt files wordList = textParse(open('email/spam/%d.txt' % i, 'r').read()) # Read each spam and convert the string into a string list docList.append(wordList) fullText.append(wordList) classList.append(1) # Mark spam, 1 indicates junk file wordList = textParse(open('email/ham/%d.txt' % i, 'r').read()) # Read each non spam message and convert the string into a string list docList.append(wordList) fullText.append(wordList) classList.append(0) # Mark normal mail, 0 indicates normal file vocabList = createVocabList(docList) # Create vocabulary without repetition trainingSet = list(range(50)) testSet =  # Create a list that stores the index values of the training set and the test set for i in range(10): # From the 50 emails, 40 were randomly selected as the training set and 10 as the test set randIndex = int(random.uniform(0, len(trainingSet))) # Random selection of index value testSet.append(trainingSet[randIndex]) # Add index value of test set del (trainingSet[randIndex]) # Delete the index value added to the test set in the training set list trainMat =  trainClasses =  # Create training set matrix and training set category label coefficient vector for docIndex in trainingSet: # Ergodic training set trainMat.append(setOfWords2Vec(vocabList, docList[docIndex])) # Add the generated word set model to the training matrix trainClasses.append(classList[docIndex]) # Add a category to the training set category label coefficient vector p0V, p1V, pSpam = trainNB0(np.array(trainMat), np.array(trainClasses)) # Training naive Bayesian model errorCount = 0 # Error classification count for docIndex in testSet: # Traversal test set wordVector = setOfWords2Vec(vocabList, docList[docIndex]) # Word set model of test set if classifyNB(np.array(wordVector), p0V, p1V, pSpam) != classList[docIndex]: # If the classification is wrong errorCount += 1 # Error count plus 1 print("Misclassified test sets:", docList[docIndex]) print('Error rate:%.2f%%' % (float(errorCount) / len(testSet) * 100)) if __name__ == '__main__': spamTest()
4. Operation results
5. Naive Bayesian evaluation
Advantages: it is still effective in the case of less data, and can deal with multi category problems
Disadvantages: it is sensitive to the preparation method of input data; Because of simplicity Bayes Because of its "simplicity" characteristics, it will bring some loss of accuracy