Scanning

We have split out the scanner into a separate github repository which has its own PyPI package.

A little bit about the scanner… There are two passes made in the scanner, a “pre-scan” in found in mathics_scanner.prescanner which converts some WL-specific character codes to character or long names and the mathics_scanner.tokeniser which runs after that. The tokenizer breaks up a string into tokens, classifications of a sequence of characters, which is then as the atoms on which the parser pattern matches on.