talezero/integration
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Better documentation about tokens

Browse Source
This commit is contained in:
jyelon
2026-02-19 00:32:29 -05:00
parent cf77eb8544
commit 9f286a1648
2 changed files with 66 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files

11
Docs/A-Summary-of-our-Lua-Patches.md
View File

@@ -251,15 +251,14 @@ Update 2: I don't remember using userdata objects at all. I am not sure that Upd
## Token Literal Syntax Patch
Tokens are lightuserdata values encoding short alphanumeric strings as base37 numbers (see `Tokens-A-New-Lua-Type.md`). Previously, tokens could only be created in C++ and inserted into the Lua environment via `LuaTokenConstant`. This patch adds a literal syntax to the Lua parser so that tokens can be written directly in Lua source code using the `@` prefix:
Tokens are lightuserdata values encoding short alphanumeric
strings as base37 numbers (see `Tokens-A-New-Lua-Type.md`).
This patch adds a literal syntax to the Lua parser so that
tokens can be written directly in Lua source code using the
`@` prefix:
```lua
local x = @null
local y = @found
```
The lexer (llex.c) recognizes `@` followed by one or more alphanumeric characters (a-z, 0-9, case insensitive, max 12 characters). It encodes the string as a base37 number using the same encoding as `LuaToken::parse()` in luastack.hpp and produces a `TK_TOKEN` token. The parser (lparser.c) handles `TK_TOKEN` in `simpleexp()` by storing it as a lightuserdata constant in the function's constant table via `luaK_lightuserdataK()` in lcode.c.
Underscores are not valid in token literals. Writing `@foo_bar` produces a lexer error rather than silently splitting into token `@foo` and identifier `_bar`.
This patch is live and functioning.