language-detection

问题 Closed . This question needs details or clarity. It is not currently accepting answers. Want to improve this question? Add details and clarify the problem by editing this post. Closed 3 months ago . Improve this question I am using fasttext (v=0.9.1) to detect the language of a text (see this). Norwegian text is being detected as Danish when using this model. !curl "https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.176.bin" > lid.bin import fastText language_detector=fastText.load

问题 I am having a dataframe of which one column has a list of strings at each row. On average, each list has 150 words of about 6 characters each. Each of the 700 rows of the dataframe is about a document and each string is a word of this document; so basically I have tokenised the words of the document. I want to detect the language of each of these documents and to do this I firstly try to detect the language of each word of the document. For this reason I do the following: from textblob import

来源： https://stackoverflow.com/questions/40783383/error-using-langdetect-in-python-no-features-in-text

问题 Are there any good, open source engines out there for detecting what language a text is in, perhaps with a probability metric? One that I can run locally and doesn't query Google or Bing? I'd like to detect language for each page in about 15 million pages of OCR'ed text. Not all documents will contain languages which use the Latin alphabet. 回答1: Depending on what you're doing, you might want to check out the python Natural Language Processing Toolkit (NLTK), which has some support for

问题 I have a program that reads a bunch of text and analyzes it. The text may be in any language, but I need to test for japanese and chinese specifically to analyze them a different way. I have read that I can test each character on it's unicode number to find out if it is in the range of CJK characters. This is helpful, however I would like to separate them if possible to process the text against different dictionaries. Is there a way to test if a character is Japanese OR Chinese? 回答1: You won

问题 Closed. This question is off-topic. It is not currently accepting answers. Want to improve this question? Update the question so it's on-topic for Stack Overflow. Closed 3 years ago . First of all, i have lot of text available. Let's say, i have 10000 characters for each try. The script is php based, but i can use whatever i want. C++, java, no problem. The google language api can't be used: their usage limits are to low. I'ts 6 hours that i try to come out with anything great, but none for

问题 Closed. This question is off-topic. It is not currently accepting answers. Want to improve this question? Update the question so it's on-topic for Stack Overflow. Closed 3 years ago . I am dealing with an application that is accepting user input in different languages (currently 3 languages fixed). The requirement is that users can enter text and dont bother to select the language via a provided checkbox in the UI. Is there an existing Java library to detect the language of a text? I want

问题 Closed. This question is off-topic. It is not currently accepting answers. Want to improve this question? Update the question so it's on-topic for Stack Overflow. Closed 3 years ago . I am dealing with an application that is accepting user input in different languages (currently 3 languages fixed). The requirement is that users can enter text and dont bother to select the language via a provided checkbox in the UI. Is there an existing Java library to detect the language of a text? I want

问题 What's the best way to detect the language of a string? 回答1: If the context of your code have internet access, you can try to use the Google API for language detection. http://code.google.com/apis/ajaxlanguage/documentation/ var text = "¿Dónde está el baño?"; google.language.detect(text, function(result) { if (!result.error) { var language = 'unknown'; for (l in google.language.Languages) { if (google.language.Languages[l] == result.language) { language = l; break; } } var container =

问题 I'm looking for a library or technique to detect the input language of blocks of text provided by users. Online lookups (like Google translate) won't work for this task as I'm writing an app which must run offline. Thanks. 回答1: Here are two more n-gram -based gems you might want to try. They work offline. https://github.com/echen/unsupervised-language-identification, optimized for separating english and other languages (has a live demo) https://github.com/feedbackmine/language_detector, less