pyparsing v flex/bison
Actually, updated without some guff
Tokenising longest file I’m likely to need with pyparsing – about 8.5 seconds
Tokenising same file with Flex/Bison – about 0.07 seconds
ie. 2 orders of magnitude speed up.
Of course, Flex/Bison is an order of magnitude more tedious (and involves me understanding strdup and free()).