Carving 300+ invoices back from a dropped database
A production database was dropped by accident. Using the previous day’s backup plus raw disk forensics, I got back every transaction written after it.
- AED 200K+
- in 300+ invoices restored
- AED 130K+
- in 120+ payments restored
- 100%
- of lost transactions recovered
- 8 wks
- from raw disk image to done
The problem
During a post-migration check, a colleague dropped a live client database. The last backup was from the day before, so a full day of invoices and payments existed only as fragments on disk. No off-the-shelf tool could reconstruct them.
Approach
- 01
Learn the storage engine from the bytes up
Studied InnoDB’s on-disk layout (pages, B-tree indexes, record formats and undo log structure), all well below the level most application developers ever touch.
- 02
Write a Go carver for raw pages
Reverse-engineered the page format and wrote tooling that scanned disk sectors, identified InnoDB pages, and decoded records back into rows. That recovered about 90% in four weeks.
- 03
Switch mental models for the last 10%
The remaining data only existed in undo logs, which use a different format and a different logic. Four more weeks rebuilt those records and reconciled them with the carved rows.
Outcome
Every transaction from the gap was restored and reconciled against the backup, so the client lost no financial records.
What I took awayRecovery work rewards changing your mental model. The technique that got the first 90% was useless for the last 10%.