tokenize
@forge/monorepo / backend/src / tokenize
Function: tokenize()
tokenize(
text): readonlystring[]
Defined in: backend/src/adapters/memory/knowledge.ts:262
Tokens for BM25.
Lowercased, split on non-alphanumerics, and - is kept inside a token — ERR-4021 and Q3-2026 are
single terms, and splitting them is how an exact-code query stops matching the document containing that
exact code. That is the one case keyword retrieval exists for.
Parameters
text
string
Returns
readonly string[]