tree c1e1a9b48d7f8405e0fcb0a15e3285cf7b38e404
parent d04f362cd432ad643a69ca89f4aba57987b87170
author Sketch🕴️ <skallywag@sketch.dev> 1772291435 +0400
committer Sketch🕴️ <skallywag@sketch.dev> 1772291435 +0400

Add lexer: Tokenize function with full test coverage

Implements lexer.Tokenize(input string) ([]token.Token, error)
- Skips whitespace
- Parses integer and decimal numbers (including leading dot e.g. .5)
- Handles all operators: + - * /
- Handles parentheses: ( )
- Appends EOF token
- Returns error on invalid characters with position info
- 12 unit tests covering: empty, whitespace-only, integers, decimals,
  leading-dot numbers, operators, parens, full expressions, no-space
  expressions, invalid chars, multiple decimals (1.2.3)