language-detection

I'm using langdetect to determine the language of a set of strings which I know are either in English or French. Sometimes, langdetect tells me the language is Romanian for a string I know is in French. How can I make langdetect choose between English or French only, and not all other languages? Thanks! Option 1 One option would be using the package langid instead. Then you can simply restrict the languages with a method call: import langid langid.set_languages(['fr', 'en']) # ISO 639-1 codes lang, score = langid.classify('This is a french or english text') print(lang) # en Option 2 If you

Given a block of text that's known to be Chinese and encoded in UTF-8, is there a way to determine if it's Simplified or Traditional? I don't know if this will work, but I'd try using iconv to see if it will translate between the charsets correctly, comparing the results from the same conversion with //TRANSLIT and //IGNORE. If the two results match, then the charset conversion hasn't encountered any characters that fail to translate, so you should have a match. $test1 = iconv("UTF-8", "big5//TRANSLIT", $text); $test2 = iconv("UTF-8", "big5//IGNORE", $text); if ($test1 == $test2) { echo

I'm creating an application for detecting the language of short texts, with an average of < 100 characters and contains slang (e.g tweets, user queries, sms). All the libraries I tested work well for normal web pages but not for very short text. The library that's giving the best results so far is Chrome's Language Detection (CLD) library which I had to build as a shared library. CLD fails when the text is made of very short words. After looking at the source code of CLD, I see that it uses 4-grams so that could be the reason. The approach I'm thinking of right now to improve the accuracy is:

I'm looking for a library or technique to detect the input language of blocks of text provided by users. Online lookups (like Google translate) won't work for this task as I'm writing an app which must run offline. Thanks. Here are two more n-gram -based gems you might want to try. They work offline. https://github.com/echen/unsupervised-language-identification , optimized for separating english and other languages (has a live demo) https://github.com/feedbackmine/language_detector , less specialized, will detect more languages. Some languages may need some extra training — I found it to be

I have a table in PostgreSQL where a column is a text. I need a library or tool that can identify the language of each text for a test purpose. There is no need for a PostgreSQL code because I'm having problems to install languages, but any language that can connect to the database, retrieve the texts and identify it arewelcome. I used Lingua::Identify suggested in the answers right in the Perl script, it worked, but the results are not precise. The texts I want to identify comes from the web and most are in portuguese, but Lingua::Identify is classifying much as french, italian and spanish

Are there any good, open source engines out there for detecting what language a text is in, perhaps with a probability metric? One that I can run locally and doesn't query Google or Bing? I'd like to detect language for each page in about 15 million pages of OCR'ed text. Not all documents will contain languages which use the Latin alphabet. Depending on what you're doing, you might want to check out the python Natural Language Processing Toolkit (NLTK), which has some support for Bayesian Learning Algorithms. In general, the letter and word frequencies would probably be the fastest evaluation,