corpus

The dataset is one big file with three columns: An ID of a section, something irrelevant and a line of text. An example could look like the following: A01 001 This is a simple test. A01 002 Just for exemplary purpose. A01 003 A02 001 This is another text I want to use the first column (in this example A01 and A02, which represent different texts) to be the file name, whichs content is everything in that line after the second column. The example above should result two files, one with name A01 and content: This is a simple test. Just for exemplary purpose. and another one A02 with content: This

I'm using the Brown Corpus. I want some way to print out all the possible tags and their names (not just tag abbreviations). There are also quite a few tags, is there a way to 'simplify' the tags? By simplify I mean combine two extremely similar tags into one and re-tag the merged words with the other tag? alvas It's somehow discussed previously in: Java Stanford NLP: Part of Speech labels? Simplifying the French POS Tag Set with NLTK https://linguistics.stackexchange.com/questions/2249/turn-penn-treebank-into-simpler-pos-tags The POS tag output from nltk.pos_tag are PennTreeBank tagset, https

I am working with a CountVectorizer from scikit learn, and I'm possibly attempting to do some things that the object was not made for...but I'm not sure. In terms of getting counts for occurrence: vocabulary = ['hi', 'bye', 'run away!'] corpus = ['run away!'] cv = CountVectorizer(vocabulary=vocabulary) X = cv.fit_transform(corpus) print X.toarray() gives: [[0 0 0 0]] What I'm realizing is that the CountVectorizer will break the corpus into what I believe is unigrams: vocabulary = ['hi', 'bye', 'run'] corpus = ['run away!'] cv = CountVectorizer(vocabulary=vocabulary) X = cv.fit_transform(corpus

Context I have been trying to implement the algorithm recently proposed in this paper . Given a large amount of text (corpus), the algorithm is supposed to return characteristic n -grams (i.e., sequence of n words) of the corpus. The user can decide the appropriate n , and at the moment I am trying with n = 2-6 as in the original paper. In other words, using the algorithm, I want to extract 2- to 6-grams that characterize the corpus. I was able to implement the part that calculates the score based on which characteristic n -grams are identified, but have been struggling to eliminate non

Is any place I can download Treebank of English phrases for free or less than $100? I need training data containing bunch of syntactic parsed sentences (>1000) in English in any format. Basically all I need is just words in this sentences being recognized by part of speech. cyborg NLTK (for Python) offers several treebanks for free . Here are a couple (English) treebanks available for free: American National Corpus: MASC Questions: QuestionBank and Stanford's corrections British news: BNC TED talks: NAIST-NTT TED Treebank Georgetown University Multilayer Corpus: GUM Biomedical: NaCTeM GENIA

I have a list of html files, I have taken some texts from the web and make them read with the read_html . My files names are like: a1 <- read_html(link of the text) a2 <- read_html(link of the text) . . . ## until: a100 <- read_html(link of the text) I am trying to create a corpus with these. Any ideas how can I do it? Thanks. You could allocate the vector beforehand: text <- rep(NA, 100) text[1] <- read_html(link1) ... text[100] <- read_html(link100) Even better, if you organize your links as vector. Then you can use, as suggested in the comments, lapply : text <- lapply(links, read_html)