Writing a compiler for a DSL in python

Question

I am writing a game in python and have decided to create a DSL for the map data files. I know I could write my own parser with regex, but I am wondering if there are existing py

Answer 1

There are plenty of Python parsing tools: http://nedbatchelder.com/text/python-parsers.html

Answer 2

On the lines of declarative python, I wrote a helper module called 'bpyml' which lets you declare data in python in a more XML structured way without the verbose tags, it can be converted to/from XML too, but is valid python.

https://svn.blender.org/svnroot/bf-blender/trunk/blender/release/scripts/modules/bpyml.py

Example Use http://wiki.blender.org/index.php/User:Ideasman42#Declarative_UI_In_Blender

Answer 3

Yes, there are many -- too many -- parsing tools, but none in the standard library.

From what what I saw PLY and SPARK are popular. PLY is like yacc, but you do everything in Python because you write your grammar in docstrings.

Personally, I like the concept of parser combinators (taken from functional programming), and I quite like pyparsing: you write your grammar and actions directly in python and it is easy to start with. I ended up producing my own tree node types with actions though, instead of using their default ParserElement type.

Otherwise, you can also use existing declarative language like YAML.

Answer 4

Here's an approach that works really well.

abc= ONETHING( ... )
xyz= ANOTHERTHING( ... )
pqr= SOMETHING( this=abc, that=123, more=(xyz,123) )

Declarative. Easy-to-parse.

And...

It's actually Python. A few class declarations and the work is done. The DSL is actually class declarations.

What's important is that a DSL merely creates objects. When you define a DSL, first you have to start with an object model. Later, you put some syntax around that object model. You don't start with syntax, you start with the model.

Answer 5

For "small languages" as the one you are describing, I use a simple split, shlex (mind that the # defines a comment) or regular expressions.

>>> line = 'SOMETHING: !abc @123 #xyz/123'

>>> line.split()
['SOMETHING:', '!abc', '@123', '#xyz/123']

>>> import shlex
>>> list(shlex.shlex(line))
['SOMETHING', ':', '!', 'abc', '@', '123']

The following is an example, as I do not know exactly what you are looking for.

>>> import re
>>> result = re.match(r'([A-Z]*): !([a-z]*) @([0-9]*) #([a-z0-9/]*)', line)
>>> result.groups()
('SOMETHING', 'abc', '123', 'xyz/123')

Answer 6

I've always been impressed by pyparsing. The author, Paul McGuire, is active on the python list/comp.lang.python and has always been very helpful with any queries concerning it.