This file contains the scanner for Pico. A scanner is a 'pre-phase' for a parser. Because a parser has to work with the (context-free) structure of programs it should be freed from the 'character level'. E.g. a parser works with the concepts 'colon' and 'colon equals', and not with the individual charachters : and :=. It is the job of the scanner to take a string of characters and transform it to a stream of 'tokens'. It therefore works with 'a current character' and analyzes the string further and further each time the parser makes a request for the next token. The tokens are indivisible units. However, some tokens are associated with an extra value: suppose the scanner tells the parser that the next token is a number, then the parser should be able to ask the scanner the exact number it is talking about.

This means that:

There is a global function init_scan that has to be called for each pico program (that is still in string format) one wants to evaluate.
There is a function scan that is called by the parser each time it needs a new token.
There is a globally accessible variable scan_data that contains any extra information that is associated to the 'current token' (such as the number value described above).