adjust the entire tokens/parser impl - #57

Merged

Lokathor merged 63 commits into

mainfrom

Aug 21, 2026

Commits on May 11, 2026

add more punctuation tokens so we don't have to merge them later in the pipeline
Lokathor
committed
e7d57c0

Commits on May 13, 2026

move stuff to "old" folders
Lokathor
committed
f67ddb5
update versions
Lokathor
committed
1a9f5d4
lib to old, and a new lib file
Lokathor
committed
0621a86
make a target minimum program
Lokathor
committed
038d565
begin test update
Lokathor
committed
8177e97

Commits on May 14, 2026

update tests
Lokathor
committed
fdfb2f5
configure some warnings
Lokathor
committed
0003c9b
docs
Lokathor
committed
b2be626
more token examples.
Lokathor
committed
38e89a8
clear old langref stub to be even more of a stub
Lokathor
committed
075927e
try to depend on less, but we'll probably depend on all this again soon enough.
Lokathor
committed
db6f6bd

Commits on May 18, 2026

start making a resilient ll parser
Lokathor
committed
913e4a6

Commits on May 20, 2026

updates
Lokathor
committed
2e9b643

Commits on May 22, 2026

.
Lokathor
committed
0f4a4a4

Commits on May 23, 2026

lit strings work
Lokathor
committed
0493b9c

Commits on May 27, 2026

updates
Lokathor
committed
942e7c2

Commits on Aug 14, 2026

update tokenizer, i think it's done now
Lokathor
committed
cdc9693
Merge branch 'main' into more_tokens
Lokathor
authored
3accc5d
update CI
Lokathor
committed
1c6e997
delete
Lokathor
committed
a06f726
rename
Lokathor
committed
af16cf2

Commits on Aug 15, 2026

try a hand tokenizer
Lokathor
committed
2482eba

Commits on Aug 16, 2026

docs
Lokathor
committed
60465b1
parser outline
Lokathor
committed
96f453d
organize
Lokathor
committed
d9d2a00
more2
Lokathor
committed
076e84e
.
Lokathor
committed
d67ce72
todo so it builds
Lokathor
committed
50f0bef
freeze the type on the transmutation
Lokathor
committed
9e7e8d8
quiet down while in development
Lokathor
committed
ac3a730
comment tokens
Lokathor
committed
0a693a4
progress
Lokathor
committed
711616e
work on lit string
Lokathor
committed
804652c
fix and expand tests
Lokathor
committed
2941157
literal strings work
Lokathor
committed
a12c9cd
more
Lokathor
committed
dcba62a
literal numbers
Lokathor
committed
ddd17b6
tests and fix for lit num
Lokathor
committed
e9b39e4
keywords and idents
Lokathor
committed
84ac9e8
it 99% works now!
Lokathor
committed
ea830c8
it's fine to label things
Lokathor
committed
6380272
complete!
Lokathor
committed
3327220
convert fully to the new system
Lokathor
committed
24ae58f
.
Lokathor
committed
d98c689
notes
Lokathor
committed
2bcddc2

Commits on Aug 18, 2026

just print some tokens as a demo
Lokathor
committed
ad4ae05
type cleanup
Lokathor
committed
6237fbb

Commits on Aug 19, 2026

more
Lokathor
committed
4ffc63e
design
Lokathor
committed
8f5ab1e
tokena for whitespace
Lokathor
committed
33cfeee
fix code and adjust tests
Lokathor
committed
4a752a5
handle extra block comment closes
Lokathor
committed
e911aee
merge litstr and litrawstr token kinds
Lokathor
committed
6293acd

Commits on Aug 20, 2026

cleanup
Lokathor
committed
6a0c6ac
core parsing api cleanup
Lokathor
committed
c15650e
cleanup
Lokathor
committed
398456a
cleanup
Lokathor
committed
96eebf8
fix loop error in last commit
Lokathor
committed
b70cf56
expr progress
Lokathor
committed
ce89547

Commits on Aug 21, 2026

parser
Lokathor
committed
bf7f70a
more tokens, including multi-punctuation tokens
Lokathor
committed
e2c44ce
operators type and helpers to pick out an operator
Lokathor
committed
6e2ab0e