问题
Wiktionary is a wiki dictionary that covers many languages. It even has translations. I would be interested in parsing it and playing with the data, has anyone does anything like this before? Is there any library I can use? (Preferably Python.)
回答1:
Wiktionary runs on MediaWiki, which has an API.
One of the subpages for the API documentation is Client code, which lists some Python libraries.
回答2:
I had at one time downloaded a wiktionary dump, trying to gather together words and definitions for slavic languages. I approached it using elementtree to go thru the xml file that is the dump. I would avoid trying to scrape or crawl the site, and just download the xml dump that wikimedia provides for wiktionary. Go to the wikimedia downloads, look for the english wiktionary dumps (enwiktionary) and go to the most recent dump. You'll probably want the pages-articles.xml.bz2 file, which is just the article content, no history or comments. Parse this with whatever xml processing libraries you prefer in python. I personally prefer elementtree. Good luck.
回答3:
wordnik has done a good job parsing-out definitions, etc and they have a great api
like the others have mentioned, wiktionary is a formatting-disaster, and was not built to be computer-readable
回答4:
Yes, many people parsed Wiktionary. You can usually find past experiences in the Wiktionary-l mailing list archives.
A project not mentioned by other answers is DBPedia's Wiktionary RDF extraction.
Dozens other research projects parsed Wiktionary: you can find some examples in a recent Wiktionary special and in other issues of the Wikimedia research newsletter.
Recently someone also made an English Wiktionary REST API which includes an unspecified subset of the Wiktionary data; future plans for the thing are not known yet.
回答5:
I had a crack at parsing the german wiktionary. I ended up writing it off as too difficult, but I put my (not at all tidied up) code up at https://github.com/benreynwar/wiktionary-parser before I gave up. Although there are conventions used by the editors they are not enforced by anything other than peer oversight. The diversity of templates used along with all the typos in the pages makes the parsing quite challenging.
I think the problem is that they've used the same system as for wiktionary which is great for ease of use by the editors, but is not appropriate for the much more structured content of wiktionary. It's a shame because if wiktionary could be easily parsed it would be a hugely useful resource.
回答6:
I just made a word list from the German dump like that:
bzcat pages-articles.xml.bz2 | grep '<title>[^[:space:][:punct:]]*</title>' | sed 's:.*<title>\(.*\)</title>.*:\1:' > words
回答7:
You are welcome to play with the MySQL parsed Wiktionary database. There are two databases (English Wiktionary and Russian Wiktionary) created by the parser written in Java: http://wikokit.googlecode.com
If you like PHP, then you are welcome to play with piwidict - PHP API to this machine-readable Wiktionary 2
回答8:
You may be interested in dbnary project, not python but interesting. Claims support parsing for 21 languages and it powers wikdict.
回答9:
There is also JWKTL which does a good job at parsing and extracting structured data from wiktionary. It is written in Java and has support for the English, German and Russian editions.
回答10:
It depends on how thoroughly you need to parse it. If you just need to get all contents of a word in a language (definition, etymology, pronunciation, conjugation, etc.) then it's pretty easy. I had done this before, although in Java using jsoup
However, if you need to parse it down to different components of the content (e.g. just getting the definitions of a word), then it will be much more challenging. A Wiktionary entry for a word in a language has no pre-defined template, so a header can be anything from <h3> to <h6>, the order of the sections may be jumbled, they can be repetitive, etc.
回答11:
I wrote a primitive parser for the German Wiktionary dump in Java that only extracts nouns and their articles, plus their Arabic translation, without any dependencies. Execution takes a long time, so be warned. If there’s interest/need to parse more or other data, please tell me, I might look into it as time permits.
来源：https://stackoverflow.com/questions/3364279/has-anyone-parsed-wiktionary