Semi-supervised Naive Bayes with NLTK [closed]

问题

This question is unlikely to help any future visitors; it is only relevant to a small geographic area, a specific moment in time, or an extraordinarily narrow situation that is not generally applicable to the worldwide audience of the internet. For help making this question more broadly applicable, visit the help center.

Closed 7 years ago.

I have built a semi-supervised version of NLTK's Naive Bayes in Python based on the EM (expectation-maximization algorithm). However, in some iterations of EM I am getting negative log-likelihoods (the log-likelihoods of EM must be positive in every iteration), therefore I believe that there must be some mistakes in my code. After carefully reviewing my code, I have no idea why is this happenning. It would be really appreciated if someone could spot any mistakes in my code below:

(Reference material of semi-supervised Naive Bayes)

EM-algorithm main loop

#initial assumptions:
#Bernoulli NB: only feature presence (value 1) or absence (value None) is computed 

#initial data:
#C: classifier trained with labeled data
#labeled_data: an array of tuples (feature dic, label)
#features: dictionary that outputs feature dictionary for a given document id  

for iteration in range(1, self.maxiter):    

  #Expectation: compute probabilities for each class for each unlabeled document
  #An array of tuples (feature dictionary, probability dist) is built       
  unlabeled_data = [(features[id],C.prob_classify(features[id])) for id in U]

  #Maximization: given the probability distributions of previous step,
  #update label, feature-label counts and update classifier C
  #gen_freqdists is a custom function, see below
  #gen_probdists is the original NLTK function
  l_freqdist_act,ft_freqdist_act, ft_values_act = self.gen_freqdists(labeled_data,unlabeled_data)
  l_probdist_act, ft_probdist_act = self.gen_probdists(l_freqdist_act, ft_freqdist_act, ft_values_act, ELEProbDist)
  C = nltk.NaiveBayesClassifier(l_probdist_act, ft_probdist_act)                    

  #Compute log-likelihood 
  #NLTK Naive bayes classifier prob_classify func gives logprob(class) + logprob(doc|class))
  #for labeled data, sum logprobs output by the classifier for the label
  #for unlabeled data, sum logprobs output by the classifier for each label
  log_lh = sum([C.prob_classify(ftdic).prob(label) for (ftdic,label) in labeled_data])      
  log_lh += sum([C.prob_classify(ftdic).prob(label) for (ftdic,ignore) in unlabeled_data for label in l_freqdist_act.samples()])

  #Continue until convergence               
  if log_lh_old == "first": 
    if self.debug: print "\tM: #iteration 1",log_lh,"(FIRST)"
    log_lh_old =  log_lh            
  else:
    log_lh_diff = log_lh - log_lh_old
    if self.debug: print "\tM: #iteration",iteration,log_lh_old,"->",log_lh,"(",log_lh_diff,")"
    if log_lh_diff < self.log_lh_diff_min: break        
    log_lh_old =  log_lh

Custom function gen-freqdists, used to create needed frequency distributions

def gen_freqdists(self, instances_l, instances_ul):     
    l_freqdist = FreqDist() #frequency distrib. of labels
    ft_freqdist= defaultdict(FreqDist) #dictionary of freq. distrib. for ft-label pairs
    ft_values = defaultdict(set) #dictionary of possible values for each ft (only 1/None)
    fts = set() #set of all fts

    #counts for labeled data
    for (ftdic,label) in instances_l:
      l_freqdist.inc(label,1)
      for f in ftdic.keys():
        fts.add(f) 
        ft_freqdist[label,f].inc(1,1)
        ft_values[f].add(1)

    #counts for unlabeled data
    #we must compute maximum a posteriori label estimate
    #and update label/ft occurrences accordingly
    for (ftdic,probs) in instances_ul:
      map_l = probs.max() #label with highest probability
      map_p = probs.prob(map_l) #probability of map_l
      l_freqdist.inc(map_l,count=map_p)
      for f in ftdic.keys():
        fts.add(f)
        ft_freqdist[map_l,f].inc(1,count=map_p)             
        ft_values[f].add(1)

    #features not appearing in documents get implicit None values
    for l in l_freqdist.samples():
    num_samples = l_freqdist[l] 
    for f in fts:
      count = ft_freqdist[l,f].N()              
      ft_freqdist[l,f].inc(None, num_samples-count)
      ft_values[f].add(None)    

    #return computed frequency distributions
    return l_freqdist, ft_freqdist, ft_values   

回答1:

I think you're summing the wrong values.

This is your code that is supposed to compute the sum of the log probs:

  #Compute log-likelihood 
  #NLTK Naive bayes classifier prob_classify func gives logprob(class) + logprob(doc|class))
  #for labeled data, sum logprobs output by the classifier for the label
  #for unlabeled data, sum logprobs output by the classifier for each label

  log_lh = sum([C.prob_classify(ftdic).prob(label) for (ftdic,label) in labeled_data])
  log_lh += sum([C.prob_classify(ftdic).prob(label) for (ftdic,ignore) in unlabeled_data for label in l_freqdist_act.samples()])

According to the NLTK documentation for prob_classify (on NaiveBayesClassifier) a ProbDistI object is returned (not logprob(class) + logprob(doc|class)). When you get this object, You're calling the prob method on it for a given label. You probably want to call logprob, and negate that return as well.

来源：https://stackoverflow.com/questions/13032114/semi-supervised-naive-bayes-with-nltk