collocation | 易学教程

How to get PMI scores for trigrams with NLTK Collocations? python

阅读更多关于 How to get PMI scores for trigrams with NLTK Collocations? python

问题 I know how to get bigram and trigram collocations using NLTK and I apply them to my own corpora. The code is below. My only problem is how to print out the birgram with the PMI value? I search NLTK documentation multiple times. It's either I'm missing something or it's not there. import nltk from nltk.collocations import * myFile = open("large.txt", 'r').read() myList = myFile.split() myCorpus = nltk.Text(myList) trigram_measures = nltk.collocations.TrigramAssocMeasures() finder =

How to get n-gram collocations and association in python nltk?

阅读更多关于 How to get n-gram collocations and association in python nltk?

问题 In this documentation, there is example using nltk.collocations.BigramAssocMeasures() , BigramCollocationFinder , nltk.collocations.TrigramAssocMeasures() , and TrigramCollocationFinder . There is example method find nbest based on pmi for bigram and trigram. example: finder = BigramCollocationFinder.from_words( ... nltk.corpus.genesis.words('english-web.txt')) >>> finder.nbest(bigram_measures.pmi, 10) I know that BigramCollocationFinder and TrigramCollocationFinder inherit from

NLTK: Find contexts of size 2k for a word

阅读更多关于 NLTK: Find contexts of size 2k for a word

问题 I have a corpus and I have a word. For each occurrence of the word in the corpus I want to get a list containing the k words before and the k words after the word. I am doing this algorithmically OK (see below) but I wondered whether NLTK is providing some functionality for my needs that I missed? def sized_context(word_index, window_radius, corpus): """ Returns a list containing the window_size amount of words to the left and to the right of word_index, not including the word at word_index.

Forming Bigrams of words in list of sentences with Python

阅读更多关于 Forming Bigrams of words in list of sentences with Python

I have a list of sentences: text = ['cant railway station','citadel hotel',' police stn']. I need to form bigram pairs and store them in a variable. The problem is that when I do that, I get a pair of sentences instead of words. Here is what I did: text2 = [[word for word in line.split()] for line in text] bigrams = nltk.bigrams(text2) print(bigrams) which yields [(['cant', 'railway', 'station'], ['citadel', 'hotel']), (['citadel', 'hotel'], ['police', 'stn']) Can't railway station and citadel hotel form one bigram. What I want is [([cant],[railway]),([railway],[station]),([citadel,hotel]),

Forming Bigrams of words in list of sentences with Python

阅读更多关于 Forming Bigrams of words in list of sentences with Python

问题 I have a list of sentences: text = ['cant railway station','citadel hotel',' police stn']. I need to form bigram pairs and store them in a variable. The problem is that when I do that, I get a pair of sentences instead of words. Here is what I did: text2 = [[word for word in line.split()] for line in text] bigrams = nltk.bigrams(text2) print(bigrams) which yields [(['cant', 'railway', 'station'], ['citadel', 'hotel']), (['citadel', 'hotel'], ['police', 'stn']) Can't railway station and

NLTK collocations for specific words

阅读更多关于 NLTK collocations for specific words

I know how to get bigram and trigram collocations using NLTK and I apply them to my own corpora. The code is below. I'm not sure however about (1) how to get the collocations for a particular word? (2) does NLTK have a collocation metric based on Log-Likelihood Ratio? import nltk from nltk.collocations import * from nltk.tokenize import word_tokenize text = "this is a foo bar bar black sheep foo bar bar black sheep foo bar bar black sheep shep bar bar black sentence" trigram_measures = nltk.collocations.TrigramAssocMeasures() finder = TrigramCollocationFinder.from_words(word_tokenize(text))

NLTK collocations for specific words

阅读更多关于 NLTK collocations for specific words

问题 I know how to get bigram and trigram collocations using NLTK and I apply them to my own corpora. The code is below. I'm not sure however about (1) how to get the collocations for a particular word? (2) does NLTK have a collocation metric based on Log-Likelihood Ratio? import nltk from nltk.collocations import * from nltk.tokenize import word_tokenize text = "this is a foo bar bar black sheep foo bar bar black sheep foo bar bar black sheep shep bar bar black sentence" trigram_measures = nltk