{"id":1140,"date":"2015-04-29T09:15:44","date_gmt":"2015-04-29T16:15:44","guid":{"rendered":"https:\/\/blogs.ubc.ca\/coetoolbox\/?p=1140"},"modified":"2015-05-22T16:28:15","modified_gmt":"2015-05-22T23:28:15","slug":"naive-bayes-classifier-for-document-classification","status":"publish","type":"post","link":"https:\/\/blogs.ubc.ca\/coetoolbox\/2015\/04\/29\/naive-bayes-classifier-for-document-classification\/","title":{"rendered":"Na\u00efve Bayes Classifier for Document Classification"},"content":{"rendered":"

\"\"Na\u00efve Bayes Classifiers are a family of simple probabilistic classifiers which apply Bayes\u2019 theorem with a strong (na\u00efve) assumption about the independence between observations. In the context of text analytics, the assumption is that the prevalence of words in a \u2018document\u2019 (article, email, post, tweet, etc.) are independent of each other. Although this is clearly a very na\u00efve assumption, Na\u00efve Bayes has been shown to produce very strong results. One way of compensating for its shortcoming is to study biagrams or triagrams (sets of two or three words at a time) which helps to capture some of the dependence in the text.<\/p>\n

Na\u00efve Bayes essentially works as follows: <\/p>\n

    \n
  1. Pre-process the data to count how many times each word in a document appears in that document<\/li>\n
  2. Use this training set to calculate the independent likelihood of each unique word appearing as many times as it did given that the document is of a certain class (such as \u201cspam\u201d or \u201cnot spam\u201d)<\/li>\n
  3. Similarly, calculate the prior probability of a document being a certain class, as well as the probability of each word appearing as many times as it did in a document, regardless of class<\/li>\n
  4. Finally, calculate the conditional probability that a document is of a certain class based on the independent likelihood of each unique word appearing as many times as it did, multiplied by the prior probability of a document being a certain class, and divided by the probability of the words appearing as many times as they did<\/li>\n<\/ol>\n

    In summary, the posterior probability of a document being spam = (prior probability of spam) x (likelihood of having each word so many times given that its spam) \/ (the probability of having each word so many times). This way, high incident words like \u201cthe\u201d and \u201ca\u201d receive \u201cdiscounted\u201d weights in the probability calculation, and rarer words like \u201cfantastic\u201d or \u201cterrible\u201d will be taken as being better predictors of the document\u2019s class.<\/span><\/p>\n

    Na\u00efve Bayes Classification Theory<\/strong><\/p>\n

    The i-th word of a given document occurs in a document from class C with a probability:
    \np(w<\/span>i<\/sub>|C)
    \nAnd a given document D contains all the works w<\/span>i<\/sub>, given a class C. has the probability:
    \np(D|C)=\u00d5<\/span>i<\/sub>p(w<\/span>i<\/sub>|C)
    \nWe want to know what is the probability that a given document D belongs to a given class C, which is p(C|D).
    \nSo we have:p(C|D)=p(C)\/p(D)*p(D|C).
    \nFor example, we assume that there are 2 mutually exclusive classes, S and \u00acS. Then we have:
    \np(D|S)=\u00d5<\/span>i<\/sub>p(w<\/span>i<\/sub>|S)
    \nand
    \np(D|\u00acS)=\u00d5<\/span>i<\/sub>p(w<\/span>i<\/sub>|\u00acS)
    \nThen we can derive p(S|D) and p(\u00acS|D) as:
    \np(S|D)=p(S)\/p(D)*\u00d5<\/span>i<\/sub>p(w<\/span>i<\/sub>|S)
    \np(\u00acS|D)=p(\u00acS)\/p(D)*\u00d5<\/span>i<\/sub>p(w<\/span>i<\/sub>|\u00acS)
    \nDividing one by the other:
    \np(S|D)\/p(\u00acS|D)=p(S)\/p(\u00acS)*\u00d5<\/span>i<\/sub>[p(w<\/span>i<\/sub>|S)\/p(w<\/span>i<\/sub>|\u00acS)]
    \nThen we take logarithm on this formular to derive the formula for ln[p(S|D)\/p(\u00acS|D)].
    \nln[p(S|D)\/p(\u00acS|D)]=ln[p(S)\/p(\u00acS)]+\u00e5<\/span>i<\/sub>ln[p(w<\/span>i<\/sub>|S)\/p(w<\/span>i<\/sub>|\u00acS)]
    \nWhen ln[p(S|D)\/p(\u00acS|D)]>0, it means that p(S|D)>p(\u00acS|D) and this document can be classified into class S.<\/p>\n

    Applications of Na\u00efve Bayes Classification<\/strong><\/p>\n

    Digit Recognition: The Na\u00efve Bayes classification is used to identify the digits by analyzing the pixels in an image and assigning a posterior probability to the image being identified as a certain digit. An image consist of black and white pixels. The rules for assigning probability are first developed on a training sample and later used on actual data to identify the image.
    \nSpam Classification: Represents the messages as vectors that are described and evaluated as spam or non-spam based on the presence of the attributes or words in the email.
    \nMedical Diagnosis: Using the patient data to estimate the likelihood of a patient getting affected by a certain disease.<\/p>\n

    References<\/strong>
    \nhttp:\/\/en.wikipedia.org\/wiki\/Naive_Bayes_classifier#Document_classification
    \nhttp:\/\/burakkanber.com\/blog\/machine-learning-naive-bayes-1\/
    \nhttp:\/\/users.utcluj.ro\/~igiosan\/Resources\/PRS\/L8\/lab_08e.pdf
    \nhttp:\/\/arxiv.org\/pdf\/cs\/0006013.pdf
    \nhttp:\/\/www.ijarcce.com\/upload\/2014\/may\/IJARCCE9E%20%20a%20rupali%20%20Heart%20Disease%20Prediction.pdf<\/p>\n

     <\/p>\n","protected":false},"excerpt":{"rendered":"

    Na\u00efve Bayes Classifiers are a family of simple probabilistic classifiers which apply Bayes\u2019 theorem with a strong (na\u00efve) assumption about the independence between observations. In the context of text analytics, the assumption is that the prevalence of words in a \u2018document\u2019 (article, email, post, tweet, etc.) are independent of each other. Although this is clearly […]<\/p>\n","protected":false},"author":27103,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1054137],"tags":[585,1054147],"class_list":["post-1140","post","type-post","status-publish","format-standard","hentry","category-text-analytics","tag-python","tag-text-analytics"],"_links":{"self":[{"href":"https:\/\/blogs.ubc.ca\/coetoolbox\/wp-json\/wp\/v2\/posts\/1140","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blogs.ubc.ca\/coetoolbox\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.ubc.ca\/coetoolbox\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.ubc.ca\/coetoolbox\/wp-json\/wp\/v2\/users\/27103"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.ubc.ca\/coetoolbox\/wp-json\/wp\/v2\/comments?post=1140"}],"version-history":[{"count":3,"href":"https:\/\/blogs.ubc.ca\/coetoolbox\/wp-json\/wp\/v2\/posts\/1140\/revisions"}],"predecessor-version":[{"id":1176,"href":"https:\/\/blogs.ubc.ca\/coetoolbox\/wp-json\/wp\/v2\/posts\/1140\/revisions\/1176"}],"wp:attachment":[{"href":"https:\/\/blogs.ubc.ca\/coetoolbox\/wp-json\/wp\/v2\/media?parent=1140"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.ubc.ca\/coetoolbox\/wp-json\/wp\/v2\/categories?post=1140"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.ubc.ca\/coetoolbox\/wp-json\/wp\/v2\/tags?post=1140"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}