2019-12-29 - FreeNAS recovery adventure

From Digital Satori
Jump to: navigation, search

When setting up this new wiki on the same VM as my old blog, I saved my old blog entries, then nuked the vdisk and started fresh. As is tradition.

At least I thought I'd saved them all. I was missing two. Oops.

That's fine! I'm a good sysadmin, and I have backups.

The way my virtual environment is set up is like this:

  • Physical Ubuntu host running KVM
  • Virtual disks for KVM stored on a FreeNAS box
  • FreeNAS box connected to KVM host via iSCSI over dedicated 10 GbE Twinax

I'm doing both automatic periodic snapshots within FreeNAS and replicating the whole thing to a second FreeNAS box that lives out in the shed, in case the house burns down or something.

So I'm covered! I go clone a recent snapshot, try to mount the snapshot so I can dump a copy of the vdisk and fire it up in KVM or VirtualBox or something so I can grab the files I need.

But wait, everything is blank when I try to access the snapshot. What's wrong!?

Three hours of troubleshooting later, my brain switches on and I realize that, hey moron, you have this set up using iSCSI. Your snapshots are of a block device that's formatted with ext4.

So within ten minutes, I create a new iSCSI target that points to the cloned snapshot and mount it on my KVM host, then boot the VM directly from the snapshot.

Grabbed the files and reverted everything. Worked like a charm. Took longer than it should have.

But testing backups is good and I'm glad I did it.