OntoBook Creates Synthetic Medical Textbooks for Enhanced AI Pretraining

Researchers have introduced OntoBook, an innovative method designed to improve the pretraining of language models for medical applications. The system leverages existing medical ontologies, which are structured representations of medical knowledge, to generate synthetic textbook content. This process involves performing random walks through ontology graphs to capture hierarchical and causal relationships between medical codes, then using a large language model to rephrase these walks into fluent, textbook-like prose. The generated text is then used to train encoder models, such as ModernCamemBERT, with two objectives: masked language modeling and predicting relationships between code pairs. This dual-objective training, when aligned with the same data, has shown substantial performance gains on various French medical coding benchmarks. The project has released 1.3 million LLM-reformulated medical textbooks and pretrained model checkpoints, offering a valuable resource for the medical AI community.