Finding conditional probability of trigram in python nltk

问题

I have started learning NLTK and I am following a tutorial from here, where they find conditional probability using bigrams like this.

import nltk
from nltk.corpus import brown
cfreq_brown_2gram = nltk.ConditionalFreqDist(nltk.bigrams(brown.words()))

However I want to find conditional probability using trigrams. When I try to change nltk.bigrams to nltk.trigrams I get the following error.

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "home/env/local/lib/python2.7/site-packages/nltk/probability.py", line 1705, in __init__
    for (cond, sample) in cond_samples:
ValueError: too many values to unpack (expected 2)

How can I calculate the conditional probability using trigrams?

回答1:

nltk.ConditionalFreqDist expects its data as a sequence of (condition, item) tuples. nltk.trigrams returns tuples of length 3, which causes the exact error you posted.

From your post it's not exactly clear what you want to use as conditions, but the convention when doing language modeling is to condition the last word on its predecessors. The following code demonstrates how you'd implement that.

brown_trigrams = nltk.trigrams(brown.words())
condition_pairs = (((w0, w1), w2) for w0, w1, w2 in brown_trigrams)
cfd_brown = nltk.ConditionalFreqDist(condition_pairs)

回答2:

You can use the n-gram model described here.

An example for usage:

from nltk.util import ngrams

input= '...'
N = 3
trigrams = ngrams(input.split(), N)
for grams in trigrams:
  print grams

I strongly encourage you to read the above documentation, and I hope it would help.

来源：https://stackoverflow.com/questions/38068539/finding-conditional-probability-of-trigram-in-python-nltk

标签

python

nlp

nltk

n-gram