linguistics

I wondered how you would go about tokenizing strings in English (or other western languages) if whitespaces were removed? The inspiration for the question is the Sheep Man character in the Murakami novel ' Dance Dance Dance ' In the novel, the Sheep Man is translated as saying things like: "likewesaid, we'lldowhatwecan. Trytoreconnectyou, towhatyouwant," said the Sheep Man. "Butwecan'tdoit-alone. Yougottaworktoo." So, some punctuation is kept, but not all. Enough for a human to read, but somewhat arbitrary. What would be your strategy for building a parser for this? Common combinations of

TL;DR Summary: I need a single command-line application which I can use to get synonyms and other related words. It needs to be multi-lingual and works cross platform. Can anyone suggest a suitable program for me, or help me with the ones I've already found? Thanks. Longer version: I've been tasked with writing a system in PHP that can come up with alternative suggestions for words entered by the user. I need to find a thesaurus application / API or similar which I can use to generate these suggestions. Importantly, it needs to be multilingual (English, Danish, French and German). This rules

I have a site which is searchable using Lucene. I've noticed from logs that users sometimes don't find what they're looking for because they enter a singular term, but only the plural version of that term is used on the site. I would like the search to find uses of other forms of a word as well. This is a problem that I'm sure has been solved many times over, so what are the best practices for this? Please note: this site only has English content . Some approaches I've thought of: Look up the word in some kind of thesaurus file to determine alternate forms of a given word. Some examples: