This file contains the scanner for Pico. A scanner is a 'pre-phase' for a parser. Because a parser has to work with the (context-free) structure of programs it should be freed from the 'character level'. E.g. a parser works with the concepts 'colon' and 'colon equals', and not with the individual charachters : and :=. It is the job of the scanner to take a string of characters and transform it to a stream of 'tokens'. It therefore works with 'a current character' and analyzes the string further and further each time the parser makes a request for the next token. The tokens are indivisible units. However, some tokens are associated with an extra value: suppose the scanner tells the parser that the next token is a number, then the parser should be able to ask the scanner the exact number it is talking about.

This means that: