How Not to Build Documentation: The Tale of an HTML Parser
· 3 min read
We sat down to write our own documentation engine purely out of desperation. Originally, no one planned to develop the massive Universal Documentation Engine (UDE). We had a simple, utilitarian task — to document a fresh Python wrapper. As usual, the most monstrous workarounds always start with a harmless script of a few lines.
The Sudden Python
Our core SDK is written in C++. To make it accessible from other languages, wrappers are created. It was finally Python’s turn. The developers delivered the finished API, which now needed to be published on the documentation portal.
We had been relying on Doc-o-Matic for years. It was an old, heavy, battle-tested generator. There was just one problem: it completely refused to understand Python code. The software hadn’t been updated for years, tech support was long dead, so hoping for a patch was out of the question. We have strict corporate requirements for the portal’s design. None of the standard generators met them, and documenting the code was an absolute necessity.
We looked into Doxygen. The utility can extract structure from virtually anything. The catch is that its default HTML output looks like a greeting from the nineties. That’s when a brilliant idea struck us.
”Let’s Just Tweak the Styles a Bit”
“Doxygen generates ready-made pages. Let’s just unleash a script on it, swap out a couple of CSS classes, and everything will be done in a few days.”
It turned out we had catastrophically underestimated the scale of the problem. The plan was simple: intercept the output, inject it into our structure, and go drink coffee. But the deeper we dug into the generated DOM, the worse it got. Doxygen absolutely ignored modern layout practices (instead of clean tags, we were shoveling through piles of tables and inline styles).
It was too late to retreat. The prototype was needed yesterday, so we had to grit our teeth and finish the script. It gradually became overgrown with dozens of regular expressions, becoming increasingly fragile.
The Sidebar and JS Necromancy
The wildest part began at the navigation stage. We have a strict UI rule in the company: absolutely all pages must sit in the left sidebar (so the developer sees the whole picture). Doxygen doesn’t agree with this approach. It builds the menu dynamically from deeply nested arrays smeared across several generated JavaScript files.
AI and I had to physically read these JS files as plain text. We parsed the arrays, extracted the class hierarchy from them, and manually rebuilt the HTML tree of the page. This was starting to feel like digital necromancy.
Measure Twice, Parse Once
The prototype eventually fired up. On the outside, everything looked decent and up to standards. But on the inside, our script was a house of cards held together by duct tape. A minor Doxygen update comes out — and the entire portal falls apart with errors.
We wasted a ton of time trying to cut a corner. Instead, we could have calmly designed and implemented a proper architecture. We should have immediately extracted the abstract syntax tree, or at least just parsed the clean XML from Doxygen.
It became obvious: we needed our own independent engine. We hooked up the AI again so it could quickly throw together a proper parser to read Doxygen’s XML. But everything went wrong there too. I’ll tell you what happens when you leave an AI unsupervised in the next part.