Keep document ID with R corpus

问题

I have searched stackoverflow and the web and can only find partial solutions OR some that don't work due to changes in TM or qdap. Problem below:

I have a dataframe: ID and Text (Simple document id/name and then some text)

I have two issues:

Part 1: How can I create a tdm or dtm and maintain the document name/id? It only shows "character(0)" on inspect(tdm).
Part 2: I want to keep only a specific list of terms, i.e. opposite of remove custom stopwords. I want this to happen in the corpus, not the tdm/dtm.

For Part 2, I used a solution I got here: How to implement proximity rules in tm dictionary for counting words?

This one happens on the tdm part! Is there a better solution for Part 2 where you use something like "tm_map(my.corpus, keepOnlyWords, customlist)"?

Any help will be greatly appreciated. Thanks much!

回答1:

In newer versions of tm this is a lot easier with the DataframeSource() function.

"A data frame source interprets each row of the data frame x as a document. The first column must be named "doc_id" and contain a unique string identifier for each document. The second column must be named "text" and contain a "UTF-8" encoded string representing the document's content. Optional additional columns are used as document level metadata."

So in this case:

dd <-data.frame(
    doc_id=10:13,
    text=c("No wonder, then, that ever gathering volume from the mere transit ",
      "So that in many cases such a panic did he finally strike, that few ",
      "But there were still other and more vital practical influences at work",
      "Not even at the present day has the original prestige of the Sperm Whale")
    ,stringsAsFactors=F
 )

Corpus = VCorpus(DataframeSource(dd))

回答2:

First, here's a sample data.frame

dd<-data.frame(
    id=10:13,
    text=c("No wonder, then, that ever gathering volume from the mere transit ",
      "So that in many cases such a panic did he finally strike, that few ",
      "But there were still other and more vital practical influences at work",
      "Not even at the present day has the original prestige of the Sperm Whale")
    ,stringsAsFactors=F
 )

Now, in order to read special attributes from a data.frame, we will use the readTabular function to make our own custom data.frame reader. This is all we need to do

library(tm)
myReader <- readTabular(mapping=list(content="text", id="id"))

We just specify the column to use for the contents and the id in the data.frame. Now we read it in with DataframeSource but use our custom reader.

tm <- VCorpus(DataframeSource(dd), readerControl=list(reader=myReader))

Now if we want to only keep a certain set of words, we can create our own content_transformer function. One way to do this is

keepOnlyWords<-content_transformer(function(x,words) {
    regmatches(x, 
        gregexpr(paste0("\\b(",  paste(words,collapse="|"),"\\b)"), x)
    , invert=T)<-" "
    x
})

This will replace everything that's not in the word list with a space. Note that you probably want to run stripWhitespace after this. Thus our transformations would look like

keep<-c("wonder","then","that","the")

tm<-tm_map(tm, content_transformer(tolower))
tm<-tm_map(tm, keepOnlyWords, keep)
tm<-tm_map(tm, stripWhitespace)

And then we can turn that into a document term matrix

dtm<-DocumentTermMatrix(tm)
inspect(dtm)

# <<DocumentTermMatrix (documents: 4, terms: 4)>>
# Non-/sparse entries: 7/9
# Sparsity           : 56%
# Maximal term length: 6
# Weighting          : term frequency (tf)

#     Terms
# Docs that the then wonder
#   10    1   1    1      1
#   11    2   0    0      0
#   12    0   1    0      0
#   13    0   3    0      0

and you can it it has our list of words and the proper document IDs from the data.frame

来源：https://stackoverflow.com/questions/24501514/keep-document-id-with-r-corpus

标签

text

text-mining

corpus