Looking for a clear definition of what a “tokenizer”, “parser” and “lexers” are and how they are related to each other and used?

Question

I am looking for a clear definition of what a \"tokenizer\", \"parser\" and \"lexer\" are and how they are related to each other (e.g., does a parser use a tokenizer or vice

Answer 1

A tokenizer breaks a stream of text into tokens, usually by looking for whitespace (tabs, spaces, new lines).

A lexer is basically a tokenizer, but it usually attaches extra context to the tokens -- this token is a number, that token is a string literal, this other token is an equality operator.

A parser takes the stream of tokens from the lexer and turns it into an abstract syntax tree representing the (usually) program represented by the original text.

Last I checked, the best book on the subject was "Compilers: Principles, Techniques, and Tools" usually just known as "The Dragon Book".

Answer 2

Example:

int x = 1;

A lexer or tokeniser will split that up into tokens 'int', 'x', '=', '1', ';'.

A parser will take those tokens and use them to understand in some way:

• we have a statement
• it's a definition of an integer
• the integer is called 'x'
• 'x' should be initialised with the value 1

Answer 3

(adding to the given answers)

• Tokenizer will also remove any comments, and only return tokens to the Lexer.
• Lexer will also define scopes for those tokens (variables/functions)
• Parser then will build the code/program structure

Answer 4

I would say that a lexer and a tokenizer are basically the same thing, and that they smash the text up into its component parts (the 'tokens'). The parser then interprets the tokens using a grammar.

I wouldn't get too hung up on precise terminological usage though - people often use 'parsing' to describe any action of interpreting a lump of text.