Show HN: Markdown is 40× more token-efficient with SQLite (withnative.ai)
My first focus has been making Markdown vaults powerful enough so that teams can self-host all of their context (messaging, docs and projects), without it being externally fragmented or rate-limited.
I'm sharing some early results: Markdown vaults can be 40× more efficient with SQLite!
https://www.withnative.ai/research-and-posts/markdown-contex...
Here's some of the main ideas:
- SQLite or Markdown folders, you pick. You can project the same authoritative event log into either form; more projection formats will follow (e.g. maybe neo4j; we do use a graph structure).
- Empirically: SQLite is better for agents! On a 120 trial evaluation for 11,000 records, agents use 40× fewer tokens with SQLite vs Markdown, per correct answer.
- Everything is a file or SQLite row. Messages, documents and projects: all can be modelled in your vault. You get local-first collaboration with teammates, synced to a cloud of your choice.
- You can render your vault with arbitrary UI. Your WhatsApp-like UI speaks to your teammate's Slack-like UI; both are just views over the same database.
A few things I've learned building this:
- Agents do pretty well even without SQL if you give them strongly typed Markdown frontmatter. This was one arm of the trial I ran. I didn't have any clear expectations, but it was cool to watch: agents ended up writing Python scripts to do SQL-like queries.
- Cumulatively, cached input tokens can get expensive. I feel silly in hindsight here. I'd gotten used to letting my sessions run long, "It's all cached so it's very cheap". But when I looked at my results here, over 2/3 of the token bill on long sessions was cached input...
- "Markdown vaults don't scale as well as databases" isn't as self-evident as I think. This is why I decided to benchmark it... I've had some very blank stares when I try to explain how Markdown files might break at true agent-scale volume.
Unfortunately, the dataset is private, because it's two months of my own work. I am considering open-sourcing it even so, but there is definitely some embarrassing stuff in there. TBD... I know that independent result verification matters!
The core engine that I evaluated is open-source, with more to be released soon: https://github.com/withnative/native
If you're interested in trying connected context on an open engine, you can get updates from our website or email me directly: richard@withnative.ai
Comments (0)