Creating N-Grams with tm & RWeka - works with VCorpus but not Corpus

问题

Following the many guides to creating biGrams using the 'tm' and 'RWeka' packages, I was getting frustrated that only 1-Grams were being returned in the tdm. Through much trial and error I discovered that proper function was achieved using 'VCorpus' but not using 'Corpus'. BTW, I'm pretty sure this was working with 'Corpus' ~1 month ago but it is not now.

R (3.3.3), RTools (3.4), RStudio (1.0.136) and all packages (tm 0.7-1, RWeka 0.4-31) have been updated to the latest.

I would appreciate any insight on what this won't work with Corpus and if others have this same problem.

#A Reproducible example
#
#Weka bi-gram test
#

library(tm)
library(RWeka)

someCleanText <- c("Congress shall make no law respecting an establishment of",
                    "religion, or prohibiting the free exercise thereof or",
                    "abridging the freedom of speech or of the press or the",
                    "right of the people peaceably to assemble and to petition",
                    "the Government for a redress of grievances")

aCorpus <- Corpus(VectorSource(someCleanText))   #With this, only 1-Grams are created
#aCorpus <- VCorpus(VectorSource(someCleanText)) #With this, biGrams are created as desired

BigramTokenizer <- function(x) NGramTokenizer(x, Weka_control(min=2, max=2))

aTDM <- TermDocumentMatrix(aCorpus, control=list(tokenize=BigramTokenizer))

print(aTDM$dimnames$Terms)

Result with 'Corpus'

 [1] "congress"      "establishment" "law"           "make"         
 [5] "respecting"    "shall"         "exercise"      "free"         
 [9] "prohibiting"   "religion"      "the"           "thereof"      
[13] "abridging"     "freedom"       "press"         "speech"       
[17] "and"           "assemble"      "peaceably"     "people"       
[21] "petition"      "right"         "for"           "government"   
[25] "grievances"    "redress"

Result with 'VCorpus'

 [1] "a redress"        "abridging the"    "an establishment" "and to"          
 [5] "assemble and"     "congress shall"   "establishment of" "exercise thereof"
 [9] "for a"            "free exercise"    "freedom of"       "government for"  
[13] "law respecting"   "make no"          "no law"           "of grievances"   
[17] "of speech"        "of the"           "or of"            "or prohibiting"  
[21] "or the"           "peaceably to"     "people peaceably" "press or"        
[25] "prohibiting the"  "redress of"       "religion or"      "respecting an"   
[29] "right of"         "shall make"       "speech or"        "the free"        
[33] "the freedom"      "the government"   "the people"       "the press"       
[37] "thereof or"       "to assemble"      "to petition"

回答1:

I was working with R.3.4.1 and changed to R3.3.3, now the VCorpus solution worked for me. Both TM and RWeka create the bigrams correctly.

sessionInfo()
R version 3.3.3 (2017-03-06)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)

回答2:

---

I was able to reproduce exactly the same results you got.

When I started reading about Corpus and VCorpus most references pointed out that the difference was basically that VCorpus was a volatile Corpus that stays in memory, but it is not the only difference. Corpus uses SimpleCorpus as default which does not have all the properties that VCorpus has, that is why you are able to get the 2-grams using VCorpus and not with regular Corpus. For more information on this one go to this posting in stackexchange: https://stats.stackexchange.com/questions/164372/what-is-vectorsource-and-vcorpus-in-tm-text-mining-package-in-r

来源：https://stackoverflow.com/questions/42757183/creating-n-grams-with-tm-rweka-works-with-vcorpus-but-not-corpus