TOTEN Improves Technical Text Tokenization for Portuguese.

Traditional tokenization methods, such as Byte-Pair Encoding (BPE), are efficient for general vocabulary compression but often fail to semantically preserve structured technical entities. These methods tend to fragment physical quantities, numbers, units, and symbolic expressions into arbitrary subwords, losing critical meaning in scientific and engineering texts. To address this, researchers have introduced TOTEN (Knowledge-Based Ontological Tokenization), a framework specifically designed for Brazilian Portuguese technical notation. TOTEN replaces statistical derivation with a declarative classification approach, grounded in a formal ontology of engineering entities (OEE). Its robustness is enhanced by deterministic integration with external oracles like Pint for dimensional analysis, Unicode Character Database for typography, and RSLP for Portuguese morphology. Evaluations against eight state-of-the-art baselines on both internal and external Brazilian Portuguese corpora demonstrated TOTEN's superior performance. It achieved perfect unit ontological atomicity and significantly higher numerical reconstruction scores (0.775-0.904 on external corpora vs. 0.627-0.703 for the best baseline), confirming its ability to maintain the integrity and meaning of technical expressions.