Writing a syntax highlighter

Question

I was hoping to write my own syntax highlighter for a summer project I am thinking of working on but I am not sure how to write my own syntax highlighter.

I know tha

Answer 1

Unfortunatelly, I never used Actionscript, so I cannot help with that part.

But apart from that, a good start to writing a syntax highlighter would be to look at existing ones. For example, vim has syntax files in form of ordinary text files, so you could look at those for a start. There is a bunch of regular expressions there (regular expressions come in several flavours, but they're not so different ...), so for that part you might take a glance at some book.

Personally, I've found Beginning regular expressions to be a nice one. Mastering regular expressions is also nice for more advanced subjects. Regular expressions pocket reference is on the other hand nice for determining the differences in the above mentioned flavours, since it includes a chapter on vim's regex as well.

Answer 2

It might help if you explain what this syntax highlighter is for. If you are writing it in actionscript, is your idea to have a text box in a flash movie and highlight the syntax after a submit button is pushed? Or do you want to read the text from some webservice and then display the highlighted syntax? ...it's hard for me to help, because it is hard for me to imagine what you are doing

However, a syntax highlighter reads in text, then compares the lines of codes to some regex's which help the syntax highlighter figure out what the words mean. For example, it might read the word "function" or "int" as reserved words, and replace them with the html text:

<span class="reserved">function</span>, <span class="reserved"></span>

assuming you have the css and want reserved words in red,

.reserved{
  color: #ff0000;
}

This is the basic concept and you may want to take ideas from geshi since you can view the source.

Answer 3

A good start to one approach for this is the Udacity course CS262. The title is building a web browser, but really the class focuses on exactly the problems you are looking for - how to parse and lex a set of text. In your case, you'd use that info for highlighting. I just took it and it was very good. The course is "over" now, but the videos and practice problems/homeworks are still up and available for viewing.

Answer 4

I have posted an SQL code coloring tool on my blog a while ago: http://gruchalski.com/2009/04/26/flex-textrange-performance-issue-on-linux/

You can find a link to sqlcodecoloring.zip with the source. It is implemented using tokenizer and a TextRange class.

Another link, sql code coloring as part of the prototype app: http://github.com/radekg/mysqlinterface/tree/master

Answer 5

Syntax highlighters can work in two very general ways. The first implements a full lexer and parser for the language(s) being highlighted, exactly identifying each token's type (keyword, class name, instance name, variable type, preprocessor directive...). This provides all the information needed to exactly highlight the code according to some specification (keywords in red, class names in blue, what have you).

The second way is something like the one Google Code Prettify employs, where instead of implementing one lexer/parser per language, a couple of very general parsers are used that can do a decent job on most syntaxes. This highlighter, for example, will be able to parse and highlight reasonably well any C-like language, because its lexer/parser can identify the general components of those kinds of languages.

This also has the advantage that, as a result, you don't need to explicitely specify the language, as the engine will determine by itself which of its generic parsers can do the best job. The downside of course is that highlighting is less perfect than when a language-specific parser is used.

Answer 6

Building a syntax highlighter is all about finding specific keywords in the code and giving them a specific style (font, font style, colour etc.). In order to achieve this, you will need to define a list of keywords specific to the programming language in which the code is written, and then parse the text (e.g. using regular expressions), find the specific tokens and replace them with properly-styled HTML tags.

A very basic highligher written in JavaScript would look like this:

var keywords = [ "public", "class", "private", "static", "return", "void" ];
for (var i = 0; i < keywords.length; i++)
{
        var regex = new RegExp("([^A-z0-9])(" + keywords[i] + ")([^A-z0-9])(?![^<]*>|[^<>]*</)", "g");
        code = code.replace(regex, "$1<span class='rm-code-keyword'>$2</span>$3");
}