Introduction
This chapter demonstrates a tweet classifier based on the naive bayes algorithm.
Tweet Classifier
Let us take Twitter's tweets and build a classifier based on the given tweets. This classifier will tell whether a tweet is under the category of "Politics or Sports" or not.
The basic example of the tweet data will be classified based on the containing texts (Table1).
Tweet Id | Text | Category |
294051752079159296 | 99 days to go until the start of #ct13. Did you know Chris Gayle is the only player in the event\u2019s history to be dismissed for 99? | Sports |
291019672701255681 | On Jan 10, PM #Abe received a courtesy call from Mr. Yoshihiro Murai, Governor of Miyagi Prefecture. \nhttp://t.co/EsyP40Gl | Politics |
305581742104932352 | Video of last week's hot topics: #2pack, #Draghi, pensions & #drug tests. @Europarltv video http://t.co/9GVBa315vM | Politics |
291520568396759041 | 10 off the over, 10 required! Captain Faulkner to bowl the last over, in close discussion with veteran Warne. The final spot on the line #BBL02 | Sports |
I have the below "training" data from Twitter's feed (Table2).
Tweet Id | Category | Text |
306624404287275009 | Sports | 99 days to go until the start of #ct13. Did you know Chris Gayle is the only player in the event\u2019s history to be dismissed for 99? |
306481199130505216 | Sports | Tonight's Scottish First Division match between Dumbarton and Raith Rovers has been postponed due to a frozen pitch |
304353716117590016 | Politics | @GSANetwork raises awareness & stands up to stop #LGBT #bullying in school & online. http://t.co/FWIG5vvVmi @glaad |
304844614517547008 | Politics | Blasts Deja Vu. How many times have we been in this *exact* moment? Failed or ignored intel/no CCTV/blame game and innocents dead. |
Below is the tweet test data that is unclassified (Table3).
TweetId | Text |
301733794770190336 | RT @aliwilgus: @tweetsoutloud How serious is NASA's commitment to the SLS and Orion programs, and the future of human space flight beyond ... |
301576909517619200 | RT @FardigJudith: This line in the President's State of the Union Address spoke to me. Check it out & share your #SOTU #CitizenRespo |
256056214880919553 | What is your favorite place to play badminton? Do you have a specific club in mind? Give them a shoutout! #badminton #clubs |
300248062209691648 | Sam wins the first game v Safarova #FedCup #AusvCze http://t.co/yjyZLnjr |
I will classify or categorize the data of table2 by using the Naive Bayes Algorithm with the help of Python 3 code. Let us extract the important words from the tweeted sentences below.
- def extract_tweet_words(tweet_words):
- words = []
- alpha_lower = string.ascii_lowercase
- alpha_upper = string.ascii_uppercase
- numbers = [str(n) for n in range(10)]
- for word in tweet_words:
- cur_word = ''
- for c in word:
- if (c not in alpha_lower) and (c not in alpha_upper) and (c not in numbers):
- if len(cur_word) >= 2:
- words.append(cur_word.lower())
- cur_word = ''
- continue
- cur_word += c
- if len(cur_word) >= 2:
- words.append(cur_word.lower())
- return words
Get Training data from a tweet.
- def get_tweet_training_data():
- f = open('training.txt', 'r')
- training_data = []
- for l in f.readlines():
- l = l.strip()
- tweet_details = l.split()
- tweet_id = tweet_details[0]
- tweet_label = tweet_details[1]
- tweet_words = extract_words(tweet_details[2:])
- training_data.append([tweet_id, tweet_label, tweet_words])
- f.close()
- return training_data
Get test data from tweet that will be classified.
- def get_tweet_test_data():
- f = open('test.txt', 'r')
- validation_data = []
- for l in f.readlines():
- l = l.strip()
- tweet_details = l.split(' ')
- tweet_id = tweet_details[0]
- tweet_words = extract_words(tweet_details[1:])
- validation_data.append([tweet_id, '', tweet_words])
-
- f.close()
-
- return validation_data
Get a list of words in the training data.
- def get_words(training_data):
- words = []
- for data in training_data:
- words.extend(data[2])
- return list(set(words))
Get the probability of each word in the training data of tweet.
- def get_tweet_word_prob(training_data, label = None):
- words = get_words(training_data)
- freq = {}
-
- for word in words:
- freq[word] = 1
-
- total_count = 0
- for data in training_data:
- if data[1] == label or label == None:
- total_count += len(data[2])
- for word in data[2]:
- freq[word] += 1
-
- prob = {}
- for word in freq.keys():
- prob[word] = freq[word]*1.0/total_count
-
- return prob
Get the probability of a given label.
- def get_tweet_label_count(training_data, label):
- count = 0
- total_count = 0
- for data in training_data:
- total_count += 1
- if data[1] == label:
- count += 1
- return count*1.0/total_count
Apply the Naive Bayes Model like below.
- def label_tweet_data(test_data, sports_word_prob, politics_word_prob, sports_prob, politics_prob):
- labels = []
- for data in test_data:
- data_prob_sports = sports_prob
- data_prob_politics = politics_prob
-
- for word in data[2]:
- if word in sports_word_prob:
- data_prob_sports *= sports_word_prob[word]
- data_prob_politics *= politics_word_prob[word]
- else:
- continue
-
- if data_prob_sports >= data_prob_politics:
- labels.append([data[0], 'Sports', data_prob_sports, data_prob_politics])
- else:
- labels.append([data[0], 'Politics', data_prob_sports, data_prob_politics])
-
- return labels
Print the labeled or categorize the test data like below.
- def print_labelled_data(labels):
- f_out = open('test_labelled_output.txt', 'w')
- for [tweet_id, label, prob_sports, prob_politics] in labels:
- f_out.write('%s %s\n' % (tweet_id, label))
-
- f_out.close()
Read the training and test data like below.
- training_data = get_tweet_training_data()
- test_data = get__tweet_test_data()
Get the probability of each word.
- word_prob = get_tweet_word_prob(training_data)
- sports_word_prob = get_tweet_word_prob(training_data, 'Sports')
- politics_word_prob = get_tweet_word_prob(training_data, 'Politics')
Get the probability of each label.
- sports_prob = get_tweet_label_count(training_data, 'Sports')
- politics_prob = get_tweet_label_count(training_data, 'Politics')
Normalize for stop words.
- for (word, prob) in word_prob.items():
- sports_word_prob[word] /= prob
- politics_word_prob[word] /= prob
Label the test data and print it.
- test_labels = label_tweet_data(test_data, sports_word_prob, politics_word_prob, sports_prob, politics_prob)
- print_labelled_data(test_labels)
The output example of this algorithm will be something like below.
TweetId | Category |
301733794770190336 | Politics |
301576909517619200 | Politics |
305057161682227200 | Sports |
286543227178328066 | Politics |
Conclusion
So in this chapter, you learned how to build a tweet classifier.