How to recover Partition Layout from unexpected 4K Block Size Emulation

Abstract

Typical consumer-grade hard drives today use block sizes of 4 KiB internally and report a block size of 512 byte to the host for compatibility. This mode of operation is called 512e for 512 byte “emulated” block size.

Partition tables use the block size to address the locations of the different partitions on disk. Usually this is done with the 512 byte block size. Partitioning tools ensure that partitions are aligned to larger numbers up to 1 MiB to assure compatibility with whatever native block size the HDD (or SSD) really has.

There are external USB HDD enclosures which report the block size as 4 KiB to the host, irrespective of what the HDD indicates. This allows very old OSes like Windows XP to address larger drive sizes.

However, after moving the HDD to another enclosure or after moving it from the external enclosure to become an internal drive in the PC again, this causes issues: Newly, the OS uses the 512 byte block size reported by the HDD and cannot successfully process the (newly all wrong) partition table on the drive anymore.

The HDD then appears as “unformatted” to the OS.

This post is concerned with recovering from such a change i.e. to continue using the HDD which was previously reported with 4 KiB sector size and which is newly attached to a case that doesn’t unexpectedly translate the block size or even used as an internal HDD.

Danger

Please note that attempting to do any of the recovery mechanisms here can cause the loss of all data on any of the drives attached to your computer including those which are not actually targeted at the recovery.

This is due to the fact that after messing up the partition table, it may be in worst case impossible to recover from the mistake. Also, specifying a wrong device node in any of the commands may destroy data on a previously perfectly working and accessible drive.

I put the information here down mostly for my own use. If they help you, feel free to use them. Do not blindly follow any of the steps without knowing their implications. Never copy & paste commands from the Internet especially not those which are run as root!

Remember this part of the GPL (find the full text at web_gpl(31)) which applies to this page just as well:

Disclaimer of Warranty.

THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM “AS IS” WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.

Approaches to Recovery

There are multiple ways to approach the recovery from the mismatching block size between partition table and HDD:

Use testdisk. Set block size to 4096 (“old” value). Observe that testdisk identifies the partition table correctly. Recover individual files from inside testdisk. This approach is comparatively safe and well-suited if you want to recover some selected files from an unencrypted volume on the drive. In my case I would have been happy with this but unfortunately the data of interest was on a LUKS volume hence the content was not easily available from inside testdisk. On the other hand, with about 4.5 TiB the volume of interest was too large for “casual” copying around.
Use testdisk. Keep block size at 512 (correct “now” value). Scan for partitions. Have testdisk detect the effective partition layout from the data contents. I tried out this method with a 6T HDD of which one large part was encrypted but there were two issues: First, scanning the entire HDD for partitions takes a very long time - I quit after half an hour or such. Second, the encrypted part basically containing random bytes can cause testdisk to wrongly identify it as multiple partitions. It may be possible to override the wrongly detected layout, but I did not proceed with this approach far enough to see if this could be viable.
Use loopback devices. This allows performing the 4 KiB block size emulation in software. This helps identifying the partition layout and reading data including from encrypted LUKS volumes. The partition layout can be processed with tools like sfdisk and using some manual editing can then be restored on the non-emulated 512 bytes block size device to recover the partition table. I found parts of this approach documented in various places of the Internet when trying the recovery but not the complete steps. See Section Use Loopback Devices for an end-to-end treatise of this approach.

Use Testdisk

Testdisk is a fantastic tool which has helped me in many situations already. It can scan arbitrary data dumps (including device nodes) for partitions and extract files even in event of missing file system metadata.

If you don’t know about this tool already, I can only recommend learning the basics of it.

Please see here for a general introduction to testdisk: https://www.cgsecurity.org/wiki/TestDisk_Step_By_Step

Use Loopback Devices

I learned about this approach from a StackExchange answer which was lowest ranked among the ones given at the time: https://unix.stackexchange.com/a/722450/379262

The idea is to use a software-based emulation of the 4 KiB sector size. In my case, the HDD was sdd hence I applied this as follows:

losetup --sector-size 4096 -f /dev/sdd

losetup creates a virtual copy of the drive where 4 KiB sectors are emulated and represented by loop device (here loop0).

Reading From Encrypted Volume (Optional)

Having setup a loop device, kpartx makes the contained partitions of the loop device available as individual device nodes (like loop0p1, loop0p2 etc.):

kpartx -av /dev/loop0

Afterwards, they can be accessed with LUKS as needed:

cryptsetup open /dev/mapper/loop0p3 masysma-backup
mount -o ro /dev/mapper/masysma-backup /mnt

One can then access all the data. After completing the immediate need for access (i.e. backing up important files), close the LUKS container as follows:

umount /mnt
cryptsetup close masysma-backup

Having backed up the data of interest, I proceeded to recover the partition table using sfdisk

Recover the Partition Table with SFDISK

The correct partition table (except for wrong block size that is) can be retrieved from the loopback device:

sfdisk -d /dev/loop0

The output in my case was as follows:

label: gpt
label-id: 004D8B6B-9A91-4D55-8876-273D4F8BEA05
device: /dev/loop0
unit: sectors
first-lba: 6
last-lba: 1465130640
sector-size: 4096

/dev/loop0p1 : start=         256, size=      262144, type=[...]
/dev/loop0p2 : start=      262400, size=   196608000, type=[...]
/dev/loop0p3 : start=   196870400, size=  1268260096, type=[...]

The trick to recover the partition table correctly using a 512 byte block size is to update all of the offset and size values to account for that as follows:

Replace the old sector-size by the new size i.e. change from 4096 to 512
Multiply all other decimal figures (not the UUIDs of course) with * 4096 / 512 i.e. * 8 to account for the larger offsets that are needed when using 512 byte-based addressing.
Replace the names of the device nodes from the loopback device with the real device node i.e. in my case: loop0 -> sdd and loop0p1 -> sdd1 etc. I am not sure if this step is really necessary, but I wanted to be safe rather than sorry here.

In my case, the resulting updated layout looked as follows:

label: gpt
label-id: 004D8B6B-9A91-4D55-8876-273D4F8BEA05
device: /dev/sdd
unit: sectors
first-lba: 48
last-lba: 11721045120
sector-size: 512

/dev/sdd1 : start=        2048, size=     2097152, type=[...]
/dev/sdd2 : start=     2099200, size=  1572864000, type=[...]
/dev/sdd3 : start=  1574963200, size= 10146080768, type=[...]

Save the hand-edited partition table information to a text file (e.g. newtbl.txt) and double check that all figures are indeed computed correctly!

Remove the loopback devices as follows:

kpartx -d /dev/loop0
losetup -d /dev/loop0

Finally apply the correted partition table to the raw device based on 512 bytes block sizes:

sfdisk /dev/sdd < newtbl.txt # DANGER DANGER DANGER

Afterwards, all partitions become visible correctly again and the drive is fully read-writable again as it should be…

Future Directions

I think it is quite the bad behaviour of an external HDD enclosure to mess with the block size reported by the HDD even if this “fixes” the 512 bytes emulated block size to the actual 4 KiB that the HDD might be using and allows weird use cases like a 4T external HDD on Windows XP.

The correct way for an external HDD enclosure is to behave just the same way as if the HDD were internally installed and report the block-size as is reported by the HDD not by its own firmware.

For future buying of external HDD enclosures I like to avoid the ones with such a translation although in practice the only way to find out seems to be trial’n’error as the HDD enclosure manufacturers do not typically document this “feature” if present.