Refactoring our lexer, so all the awful things we did to make the tests pass are behind us.

More info: https://artificialworlds.net/blog/2024/11/22/mini-rust-in-rust-004-refactoring-our-lexer/