How to define a grammar for a programming language

Question

How to define a grammar (context-free) for a new programming language (imperative programming language) that you want to design from scratch.

In other words: How do you

Answer 1

You'll need to know quite a lot about programming languages before you start designing one. I recommend Programming Languages: Application and Interpretation by Shriram Krishnamurthi.

Answer 2

One step at a time.

No seriously, start with expressions and operators, work upwards to statements, then to functions/classes etc. Keep a list of what punctuation is used for what.

In parallel define syntax for referring to variables, arrays, hashes, number literals, string literals, other builtin literal. Also in parallel define your data naming model and scoping rules.

To check whether your grammar makes sense focus on a level (literal/variable, operator, expression, statement, function etc) and make sure that punctuation and tokens from other levels interspersed or appended/prepended is not gonna cause an ambiguity.

Finally write it all out in EBNF and run it through ANTLR or similar.

Also best not to reinvent the wheel. I normally start off by choosing sequences to start and end statement blocks and functions, and mathematical operators, that are usually fundamentally C-like, ECMAScript-like, Basic-like, command-list based or XML-based. This helps a lot cos this is what people are used to working with.

Of course you have to come up with a pretty compelling reason not to abandon writing a new language and just stick with C, ECMAScript, or Basic which are well tested and much used.

I've often started defining new language only to find someone else has already implemented a feature somewhere in some existing language.

If your goal is speed of development for some specific project, you might be better off prototyping in something like Python, Lua or SpiderMonkey if you're looking to get up and running quickly and want to reduce the amount of typing necessary in most compiled languages.

Answer 3

You'll want to have a look at EBNF (Extended Backus-Naur Form).

(Assuming you want to write a context free grammar, that is.)

Answer 4

Have a look at Bison, maybe that's what you are looking for?

Answer 5

If you mean defining a grammar, you would be best served by starting with an existing language and modifying its grammar to match what it is that you are after. Creating a grammar specification is a fairly mechanical exercise, using a set of patterns in your own head. For instance, what does an if statement look like? Does it look like C

if <- if(exp) block

if <- if(exp) block else block2

or like ML?

if <- if exp then block else block end

or maybe you want to use elseifs like Lua:

if <- if exp then exp end

if <- if exp then exp (elseif exp)* else exp end

The grammar and semantics codify these decisions. Note that none of these are quite suitable for implementation in a LALR or LL(*) compiler generator yet, and would have to be massaged for implementation because they are ambiguous.

Programming Language Pragmatics by Michael Scott is a good introduction to the design of programming languages. It's available on Amazon here