2-gram and 3-gram instead of 1-gram using RWeka

Question

I am trying to extract 1-gram, 2-gram and 3-gram from the train corpus, using RWeka NGramTokenizer function. Unfortunately, getting only 1-grams. There is my code:

train_corpus
# clean-up
cleanset1<- tm_map(train_corpus, tolower)
cleanset2<- tm_map(cleanset1, removeNumbers)
cleanset3<- tm_map(cleanset2, removeWords, stopwords("english"))
cleanset4<- tm_map(cleanset3, removePunctuation)
cleanset5<- tm_map(cleanset4, stemDocument, language="english")
cleanset6<- tm_map(cleanset5, stripWhitespace)

# 1-gram
NgramTokenizer1 <- function(x) NGramTokenizer(x, Weka_control(min = 1, max = 1))
train_dtm_tf_1g <- DocumentTermMatrix(cleanset6, control=list(tokenize=NgramTokenizer1))
dim(train_dtm_tf_1g)
[1]  5905 15322

# 2-gram
NgramTokenizer2 <- function(x) NGramTokenizer(x, Weka_control(min = 2, max = 2))
train_dtm_tf_2g <- DocumentTermMatrix(cleanset6, control=list(tokenize=NgramTokenizer2))
dim(train_dtm_tf_2g)
[1]  5905 15322

# 3-gram
NgramTokenizer3 <- function(x) NGramTokenizer(x, Weka_control(min = 3, max = 3))
train_dtm_tf_3g <- DocumentTermMatrix(cleanset6, control=list(tokenize=NgramTokenizer3))
dim(train_dtm_tf_3g)
[1]  5905 15322

Every time getting the same result, which is obviously, wrong.

# combining together 1-gram, 2-gram and 3-gram from corpus 
    NgramTokenizer <- function(x) NGramTokenizer(x, Weka_control(min = 1, max = 3))
train_dtm_tf_ng <- DocumentTermMatrix(cleanset6, control=list(tokenize=NgramTokenizer))
dim(train_dtm_tf_ng)
[1]  5905 15322

# A numeric for the maximal allowed sparsity in the range from bigger zero to smaller one
train_rmspa_m_tf_ng_95<-removeSparseTerms(train_dtm_tf_ng, 0.95)
    [1] 5905  172

# creat bag of words (BOW) vector of these terms for use later
train_BOW_3g_95 <- findFreqTerms(train_rmspa_m_tf_3g_95)

# take a look at the terms that appear in the last 5% of the instances
train_BOW_3g_95

  [1] "avg"        "februari"   "januari"    "level"      "nation"     "per"        "price"     
  [8] "rate"       "report"     "reserv"     "reuter"     "also"       "board"      "export"    
  [15] "march"      "may"        "month"      "oil"        "product"    "total"      "annual"    
  [22] "approv"     "april"      "capit"      "common"     "compani"    "five"       "inc"       
  [29] "increas"    "meet"       "mln"        "record"     "said"       "share"      "sharehold" 
  [36] "stock"      "acquir"     "addit"      "buy"        "chang"      "complet"    "continu" 

     ...

Only 1-grams. I tried to rewrite my comand in the following way:

NgramTokenizer <- function(x) NGramTokenizer(x, Weka_control(min = 1, max = 3))

But did not work out. Also tried to add another line:

options(mc.cores=1)

before NgramTokenizer comand, but no changes. Any help?

Answer 1

I came across the same issue today. It seems "tm_map" is not working well with SimpleCorpus for some reasons.

I changed my code from

corpus = Corpus(VectorSource(pd_cmnt$QRating_Explaination))

to

corpus = VCorpus(VectorSource(pd_cmnt$QRating_Explaination))

And now it works and gives back 2-gram properly.