text mining with tm package in R ,remove words starting from [http] or any other specifc word

Question

I am new to R and text mining. I had made a word cloud out of twitter feed related to some term. The problem that I'm facing is that in the wordcloud it shows http:... or htt... How do I deal about this issue I tried using metacharacter * but I still doubt if I'm applying it right

tw.text = removeWords(tw.text,c(stopwords("en"),"rt","http\\*"))

somebody into text-minning please help me with this.

Answer 1

If you are looking to remove URLs from your string, you may use:

gsub("(f|ht)tp(s?)://(.*)[.][a-z]+", "", x)

Where x would be:

x <- c("some text http://idontwantthis.com", 
         "same problem again http://pleaseremoveme.com")

---

It would be easier to provide you with a specific answer if you could post sample of your data but the following example would give you a clean text with no URLs:

> clean_x <- gsub("(f|ht)tp(s?)://(.*)[.][a-z]+", "", x)
> clean_x
[1] "some text "          "same problem again "

As a side point, I would suggest that it may be worth searching for the existing methods to clean text before mining. For example the clean function discussed here would enable you to do this automatically. On similar lines, there are function to clean your text from tweets (#,@), punctuation and other undesirable entries.

Answer 2

Apply the below code to corpus to replace a string pattern with space. String pattern can be urls or terms you want to remove from the wordcloud. For example to remove terms starting with https:

replace with space

toSpace = content_transformer( function(x, pattern) gsub(pattern," ",x) )

tweet_corpus_clean = tm_map( tweet_corpus, toSpace, "https*")

Or pass a pattern as below to remove urls

tweet_corpus_clean = tm_map( tweet_corpus, toSpace, "(f|ht)tp(s?)://(.*)[.][a-z]+")