How would you go about parsing Markdown? [closed]

Answer 1

If I was to try to parse markdown (and its extension Markdown extra) I think I would try to use a state machine and parse it one char at a time, linking together some internal structures representing bits of text as I go along then, once all is parsed, generating the output from the objects all stringed together.

Basically, I'd build a mini-DOM-like tree as I read the input file.
To generate an output, I would just traverse the tree and output HTML or anything else (PS, LaTex, RTF,...)

Things that can increase complexity:

• The fact that you can mix HTML and markdown, although the rule could be easy to implement: just ignore anything that's between two balanced tags and output it verbatim.

• URLs and notes can have their reference at the bottom of the text. Using data structures for hyperlinks could simply record something like:

  [my text to a link][linkkey]
  results in a structure like: 
      URLStructure: 
      |  InnerText : "my text to a link"
      |  Key       : "linkkey"
      |  URL       : <null>
  

• Headers can be defined with an underline, that could force us to use a simple data structure for a generic paragraph and modify its properties as we read the file:

  ParagraphStructure:
  |  InnerText    : the current paragraph text 
  |                 (beginning of line until end of line).
  |  HeadingLevel : <null> or 1-4 when we can assess 
  |                 that paragraph heading level, if any.
  

Anyway, just some thoughts.

I'm sure that there are many small details to take care of and I'm pretty sure that Regexes could become handy during the process.
After all, they were meant to process text.

Answer 2

Markdown is a JAWL (just another wiki language)

There are plenty of open source wiki's out there that you can examine the code of the parser. Most use REGEX

Check out the screwturn wiki, is has an interesting multi pass formatter pipeline, a very nice technique - see /core/Formatter.cs and /core/FormatterPipeline.cs

Best is to use/join an existing project, these sorts of things are always much harder than they appear

Answer 3

MarkdownPapers is another Java implementation whose parser is defined in a JavaCC grammar.

Answer 4

I'd probably read the syntax specification enough times to know it, and get a feel for how to parse it.

Reading the existing parser code is of course brilliant, both to see what seems to be the main source of complexity, and if any special clever tricks are being used. The use of MD5 checksumming seems a bit weird, but I haven't studied the code enough to understand why it's being done. A comment in a routine called _EscapeSpecialChars() states:

We're replacing each such character with its corresponding MD5 checksum value; this is likely overkill, but it should prevent us from colliding with the escape values by accident.

Replacing a single character by a full MD5 does seem extravagant, but perhaps it really makes sense.

Of course, it'd be clever to consider creating a "true" syntax, for a tool such as Flex to get out of the regex bog.

Answer 5

If you are using a programming language that has more than three other users, you should be able to find a library to parse it for you. A quick Google-ing reveals libraries for CL, Haskell, Python, JavaScript, Ruby, and so on. It is highly unlikely that you will need to reinvent this wheel.

If you really have to write it from scratch, I recommend writing a proper parser. With this technique, you won't have to escape things with MD5 hashes. (I agree that if you have to do something like this, it's time to reconsider your design.)

Answer 6

There are libraries available in a number of languages, including php, ruby, java, c#, javascript. I'd suggest looking at some of these for ideas.

It depends on which language you wish to use, for the best way to implement it, there will be idiomatic and non idiomatic ways to do it.

Regexes work in perl, because perl and regex are best friends.