Severe damage to ext4 filesystem - advice needed

Altoid · 2022-02-11 00:34:51

Hello:

For whatever reasons (from carelesness to a USB drive with ID issues) I ended up dd'ing an *.iso image on to a 300Gb HDD containing relatively important stuff. I say relatively because I cannot exactly recall what was there.

[Please don't ask about the backup ...]

So I went to see what the testdisk 7.0 utility had to say with respect to the screwed up drive (/dev/sdf):

---
TestDisk 7.0, Data Recovery Utility, April 2015
Christophe GRENIER <grenier@cgsecurity.org>
http://www.cgsecurity.org

TestDisk is free software, and
comes with ABSOLUTELY NO WARRANTY.

Select a media (use Arrow keys, then press Enter):
--- snip ---
> Disk /dev/sdf - 300 GB / 279 GiB - HITACHI HUS153030VLS300
---

As expected TestDisk detected an Intel/PC partition which I selected:

Disk /dev/sdf - 300 GB / 279 GiB - HITACHI HUS153030VLS300

Please select the partition table type, press Enter when done.
>[Intel  ] Intel/PC partition
--- snip ---

Disk /dev/sdf - 300 GB / 279 GiB - HITACHI HUS153030VLS300
     CHS 36472 255 63 - sector size=512

I then selected Analyse but it found this:

Disk /dev/sdf - 300 GB / 279 GiB - CHS 36472 255 63
Current partition structure:
     Partition                  Start        End    Size in sectors

No ext2, JFS, Reiser, cramfs or XFS marker
 1 * Linux                    0  32 33 36472 225 41  585934848
 1 * Linux                    0  32 33 36472 225 41  585934848

I then selected [Quick Search] and waited as it found the only partition (primary/300Gb) the drive had:

Disk /dev/sdf - 300 GB / 279 GiB - CHS 36472 255 63
     Partition               Start        End    Size in sectors
>* Linux                    0  32 33 36472 225 41  585934848 [300Gb]   <- black fg on green bg ...   =^D

Structure: Ok.  Use Up/Down Arrow keys to select partition.   <-- Says it is OK ...   =^D
Use Left/Right Arrow keys to CHANGE partition characteristics:
*=Primary bootable  P=Primary  L=Logical  E=Extended  D=Deleted
Keys A: add partition, L: load backup, T: change type, P: list files,
     Enter: to continue
ext4 blocksize=4096 Large_file Sparse_SB Backup_SB, 299 GB / 279 GiB

I did not go for the [Deeper Search] option because I knew it was only one partition on the drive.

So I went ahead and selected [Write] to write the partition structure to the disk, confirming with (Y).

It (apparently) wrote the partition and informed me that I had to reboot:

You will have to reboot for the change to take effect.

>Ok

After rebooting I see that the drive is still an isoimage and that nothing was written to the drive.
Maybe because *.iso images are RO?

So I went to [Advanced] Filesystem Utils to see what it was about.

Disk /dev/sdf - 300 GB / 279 GiB - HITACHI HUS153030VLS300
     CHS 36472 255 63 - sector size=512

 [ Analyse  ] Analyse current partition structure and search for lost partitions
>[ Advanced ] Filesystem Utils
--- snip ---

I got this:

Disk /dev/sdf - 300 GB / 279 GiB - CHS 36472 255 63

     Partition                  Start        End    Size in sectors
> 1 * Linux                    0  32 33 36472 225 41  585934848

 [  Type  ] >[Superblock]  [*ist]  [Image Creation]  [  Quit  ]
                    Locate ext2/ext3/ext4 backup superblock

I first tried the [*ist] option but that went badly:

 [  Type  ]  [Superblock] >[ *ist ]  [Image Creation]  [  Quit  ]
                              List and copy files

Support for this filesystem hasn't been enable during compilation.

So I tried with the [Superblock] option and got this:

Disk /dev/sdf - 300 GB / 279 GiB - CHS 36472 255 63

     Partition                  Start        End    Size in sectors

  Linux                    0  32 33 36472 225 41  585934848 [300Gb]
superblock 819200, blocksize=4096 [300Gb]
superblock 884736, blocksize=4096 [300Gb]
superblock 1605632, blocksize=4096 [300Gb]
superblock 2654208, blocksize=4096 [300Gb]
superblock 4096000, blocksize=4096 [300Gb]
superblock 7962624, blocksize=4096 [300Gb]
superblock 11239424, blocksize=4096 [300Gb]
superblock 20480000, blocksize=4096 [300Gb]
superblock 23887872, blocksize=4096 [300Gb]
superblock 71663616, blocksize=4096 [300Gb]

To repair the filesystem using alternate superblock, run
fsck.ext4 -p -b superblock -B blocksize device

This is where I loose footing, way out of my depth.

Anyone know how to solve this problem?

I assume (?) that the partition and it's contents is recoverable by writing the partiton structure to the drive.

But TestDisk was not able to do that and I have no idea as to what to do to get it done.

Not to mention all that superblock/alternate superblock stuff.

I'd appreciate some input on this.

Thanks in advance,

A.

ralph.ronnquist · 2022-02-11 01:04:15

If it was me, I would run fdisk to set up a new partition table... I would make a dos partition table starting from scratch possibly deleting whatever entries it has, then using defaults, all on my habit of normally using the defaults for my disk partitioning.

Altoid · 2022-02-11 01:19:32

Hello:

ralph.ronnquist wrote:

... run fdisk to set up a new partition table...
... make a dos partition table starting from scratch possibly deleting whatever entries it has ...
... using defaults ...

Hmm ...
No idea as to how to do that.
Never done that.

This is what fdisk says about the botched drive:

groucho@devuan:~$ sudo fdisk -l
--- snip ---
Disk /dev/sdf: 279.4 GiB, 300000000000 bytes, 585937500 sectors
Disk model: HUS153030VLS300 
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x563437db

Device     Boot Start       End   Sectors   Size Id Type
/dev/sdf1  *     2048 585936895 585934848 279.4G 83 Linux
groucho@devuan:~$

Even if I knew how to do it, TestDisk was not able to write recovered new partition data to the drive.

Thanks for your input.

Best,

A.

andyprough · 2022-02-11 02:39:02

Altoid wrote:

Hmm ...
No idea as to how to do that.
Never done that.

I don't know what TestDisk is, but here's some simple instructions for fdisk: https://linuxize.com/post/fdisk-command-in-linux/

"o" should create a new DOS partition table for you.

Camtaf · 2022-02-11 10:31:24

Write a copy of the super block to the drive, even if you lose what was overwritten by the .iso file, you should be able to recover the rest to copy to another disk.

https://ngelinux.com/how-to-view-backup … ilesystem/

Last edited by Camtaf (2022-02-11 10:38:59)

Plentyn · 2022-02-11 21:29:21

Whatever you do, create a backup of the drive with dd first.

Then you might use debugfs, pointing it at one of the remaining superblocks. As the inode table is probably history, you will have to use pattern search for anything you find worthwhile to scrap from the disk. This is horrible hand work, I can tell you from experience. ;-S

EDIT: as I darkly remembered, FS layout has changed - which is a good news. Parts of the inode table might be salvageable. See https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout

Last edited by Plentyn (2022-02-11 21:46:18)

GlennW · 2022-02-11 22:52:16

Altoid wrote:

Hello:
For whatever reasons (from carelesness to a USB drive with ID issues) I ended up dd'ing an *.iso image on to a 300Gb HDD containing relatively important stuff. I say relatively because I cannot exactly recall what was there.
[Please don't ask about the backup ...]
So I went to see what the testdisk 7.0 utility had to say with respect to the screwed up drive (/dev/sdf):
---
TestDisk 7.0, Data Recovery Utility, April 2015
Christophe GRENIER <grenier@cgsecurity.org>
http://www.cgsecurity.org

TestDisk is free software, and
comes with ABSOLUTELY NO WARRANTY.

Select a media (use Arrow keys, then press Enter):
--- snip ---
> Disk /dev/sdf - 300 GB / 279 GiB - HITACHI HUS153030VLS300
---
As expected TestDisk detected an Intel/PC partition which I selected:
Disk /dev/sdf - 300 GB / 279 GiB - HITACHI HUS153030VLS300

Please select the partition table type, press Enter when done.
>[Intel  ] Intel/PC partition
--- snip ---

Disk /dev/sdf - 300 GB / 279 GiB - HITACHI HUS153030VLS300
     CHS 36472 255 63 - sector size=512
I then selected Analyse but it found this:
Disk /dev/sdf - 300 GB / 279 GiB - CHS 36472 255 63
Current partition structure:
     Partition                  Start        End    Size in sectors

No ext2, JFS, Reiser, cramfs or XFS marker
 1 * Linux                    0  32 33 36472 225 41  585934848
 1 * Linux                    0  32 33 36472 225 41  585934848
I then selected [Quick Search] and waited as it found the only partition (primary/300Gb) the drive had:
Disk /dev/sdf - 300 GB / 279 GiB - CHS 36472 255 63
     Partition               Start        End    Size in sectors
>* Linux                    0  32 33 36472 225 41  585934848 [300Gb]   <- black fg on green bg ...   =^D

Structure: Ok.  Use Up/Down Arrow keys to select partition.   <-- Says it is OK ...   =^D
Use Left/Right Arrow keys to CHANGE partition characteristics:
*=Primary bootable  P=Primary  L=Logical  E=Extended  D=Deleted
Keys A: add partition, L: load backup, T: change type, P: list files,
     Enter: to continue
ext4 blocksize=4096 Large_file Sparse_SB Backup_SB, 299 GB / 279 GiB
I did not go for the [Deeper Search] option because I knew it was only one partition on the drive.
So I went ahead and selected [Write] to write the partition structure to the disk, confirming with (Y).
It (apparently) wrote the partition and informed me that I had to reboot:
You will have to reboot for the change to take effect.

>Ok
After rebooting I see that the drive is still an isoimage and that nothing was written to the drive.
Maybe because *.iso images are RO?
So I went to [Advanced] Filesystem Utils to see what it was about.
Disk /dev/sdf - 300 GB / 279 GiB - HITACHI HUS153030VLS300
     CHS 36472 255 63 - sector size=512

 [ Analyse  ] Analyse current partition structure and search for lost partitions
>[ Advanced ] Filesystem Utils
--- snip ---
I got this:
Disk /dev/sdf - 300 GB / 279 GiB - CHS 36472 255 63

     Partition                  Start        End    Size in sectors
> 1 * Linux                    0  32 33 36472 225 41  585934848

 [  Type  ] >[Superblock]  [*ist]  [Image Creation]  [  Quit  ]
                    Locate ext2/ext3/ext4 backup superblock
I first tried the [*ist] option but that went badly:
 [  Type  ]  [Superblock] >[ *ist ]  [Image Creation]  [  Quit  ]
                              List and copy files

Support for this filesystem hasn't been enable during compilation.
So I tried with the [Superblock] option and got this:
Disk /dev/sdf - 300 GB / 279 GiB - CHS 36472 255 63

     Partition                  Start        End    Size in sectors

  Linux                    0  32 33 36472 225 41  585934848 [300Gb]
superblock 819200, blocksize=4096 [300Gb]
superblock 884736, blocksize=4096 [300Gb]
superblock 1605632, blocksize=4096 [300Gb]
superblock 2654208, blocksize=4096 [300Gb]
superblock 4096000, blocksize=4096 [300Gb]
superblock 7962624, blocksize=4096 [300Gb]
superblock 11239424, blocksize=4096 [300Gb]
superblock 20480000, blocksize=4096 [300Gb]
superblock 23887872, blocksize=4096 [300Gb]
superblock 71663616, blocksize=4096 [300Gb]

To repair the filesystem using alternate superblock, run
fsck.ext4 -p -b superblock -B blocksize device
This is where I loose footing, way out of my depth.
Anyone know how to solve this problem?
I assume (?) that the partition and it's contents is recoverable by writing the partiton structure to the drive.
But TestDisk was not able to do that and I have no idea as to what to do to get it done.
Not to mention all that superblock/alternate superblock stuff.
I'd appreciate some input on this.
Thanks in advance,
A.

Hi, I've done this once or twice,

Generally just chose the last superblock before the trouble began.

superblock 884736, blocksize=4096 [300Gb]

I have also used photorec to recover files... but the names are all machine-like.

regards, Glenn

Altoid · 2022-02-12 10:38:21

Hello:

Plentyn wrote:

... create a backup of the drive with dd ...

That would mean a 300Gb image.
I have no drive to hold that, all my drives are 300Gb.

Plentyn wrote:

Then you might ...
... horrible hand work, I can tell you from experience. ;-S

I tried again with TestDisk 7.1 from a Knoppix USB.
No dice.

It will not write to the drive. 8^|

I also tried my luck with photorec.

I was able to "recover" (a way of putting it) a huge mess of numbered folders filled with numbered files, some with the proper extension, a great many with the wrong extension but viewable, many without any extension and an absolutely huge amount of files with the *.dwg (AutoCAD) extension.

A real mess ...
So, yes.
Horrible hand work.

Plentyn wrote:

Parts of the inode table might be salvageable.
See https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout

I'll have a look.
Thanks for your input.

Best,

A.

Last edited by Altoid (2022-02-12 10:45:26)

Altoid · 2022-02-12 11:03:10

Hello:

GlennW wrote:

... done this once or twice,

Hmm ...
Nice club we belong to.

TestDisk has saved my hide once or twice.
But I had never before had to face such a screw up.

I dd'id the *iso file to a data HDD drive instead of dd'ing it to the USB stick. 8^/

~$ sudo dd if=/home/user/Downloads/chimaera_amd64_desktop-live.iso of=/dev/sdg1 bs=1M

I've realised that this happened because I used lsblk to re-check the USB drive's asigned letter.

This instead of checking it with gparted as I always do.
Big mistake ...

For some reason the USB drive did not mount.
As a result of that, /dev/sdg1 ended up being assigned to the data HDD drive drive instead of the USB drive.

I knew (from the last time I plugged it in) that the USB drive mounted at /dev/sdg1, so when I saw it in the lsblk printout, I assumed everything was all right.

Obviously, I did not notice that the size was 279.4G instead of 7.2G.
Had I checked with gparted, this would have not happened.

GlennW wrote:

Generally just chose the last superblock before the trouble began.
superblock 884736, blocksize=4096 [300Gb]

No idea how to do that.
Would you mind explaining how to do it?

GlennW wrote:

... used photorec to recover files...
... but the names are all machine-like.

Indeed ...
A real mess.
See my previous post.

Thanks in advance,

A.

Andre4freedom · 2022-02-12 11:05:26

Hello,
please know that dd is a disk-dumper, a block-copying program, that acts on physical devices (like disks, partitions etc). If you give a file-name as output device it will write the blocks to that file. If the output device is partition or a disk, then for heavens sake you will have the contents of that input file (the iso image) on your partition or device. Even if you manage to recover the superblock or some inodes from you original filesystem, the blocks pointed to will still contain copied blocks from your source-device (or file). The dd program starts to copy block by block, from the beginning. man dd helps....
Sorry, but reality can be hard.

Altoid · 2022-02-12 13:26:25

Hello:

Andre4freedom wrote:

... know that dd is a disk-dumper, a block-copying program ...

Yes, I'm aware of that.

But the drive is 300G and the image I dumped into it is 7.2G.

The problem at hand, at least with TestDisk 7.1, is that (even if it says so) it does not write the partition structure to the drive.
I think it has to do with the *.iso structure, no idea if it can be changed.

Andre4freedom wrote:

... dd program starts to copy block by block, from the beginning.

Yes, I'm also aware of that.
But at7.2G + 1 block, it would have stopped copying.

Andre4freedom wrote:

... reality can be hard.

And painful. 8^/

Thanks for your input.

Best,

A.

chris2be8 · 2022-02-12 17:19:39

Altoid wrote:

I've realised that this happened because I used lsblk to re-check the USB drive's asigned letter.

Next time run lsblk twice, once *before* you plug the USB drive in and once *afterwards*. Then look for a device (or devices) that was not there before you plugged the USB drive in. Eg:

chris@rigel:~$ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0 298.1G  0 disk 
├─sda1   8:1    0   100M  0 part 
├─sda2   8:2    0    40G  0 part /
├─sda3   8:3    0 254.2G  0 part /home
└─sda4   8:4    0   3.9G  0 part 
sr0     11:0    1  1024M  0 rom  

chris@rigel:~$ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0 298.1G  0 disk 
├─sda1   8:1    0   100M  0 part 
├─sda2   8:2    0    40G  0 part /
├─sda3   8:3    0 254.2G  0 part /home
└─sda4   8:4    0   3.9G  0 part 
sdb      8:16   1   250M  0 disk 
└─sdb1   8:17   1   250M  0 part 
sr0     11:0    1  1024M  0 rom

HTH

Chris

Dutch_Master · 2022-02-12 17:39:11

It's probably better to use dmesg:

{sudo} dmesg | tail

will give you the last 10 lines, usually enough to find which drive letter the kernel has assigned to the USB stick you've just inserted.

pcalvert · 2022-02-13 15:00:10

This is the approach that I use:

df -h

That shows me my mounted drives. If I see (for example) that /dev/sdb is my "highest" drive, then I'll guess that the USB stick will be /dev/sdc.

Although my guess has usually been correct, I always verify it like this:

# fdisk -l /dev/sdc

I then look at the output of fdisk to see if the size of /dev/sdc (and other drive info) matches what I expect. It usually does, but if were to ever see something I wasn't expecting, then I would know to be extra careful before proceeding.

ralph.ronnquist · 2022-02-13 21:03:23

@Altoid, to recover an DOS partition table, you clear the first 1 Mb of the device:

# dd if=/dev/zero of=/dev/sdf bs=1M count=1 conv=notrunc

NOTE THIS IS HARMFUL TO A DISK. Make sure doubly /dev/sdf is the disk at hand.

Then run fdisk to create a DOS partition table as per post #3, create a single primary partition of type 83 starting at 2048 and ending at end.

Thereafter run fsck as per superblock advice at post #7, trying some of the discovered superblocks.

Then mount sdf1.

Might work, though it doesn't recover overwritten files of course.

Head_on_a_Stick · 2022-02-13 21:12:04

ralph.ronnquist wrote:

to recover an DOS partition table, you clear the first 1 Mb of the device

An MS-DOS partition table only occupies the first 512 bytes. Reference: https://en.wikipedia.org/wiki/Master_bo … rtitioning

This will clear the table for both MS-DOS ("MBR" type) and GPT devices:

# sgdisk --zap-all /dev/sdX

That command is supplied by the gdisk package.

ralph.ronnquist · 2022-02-13 21:32:09

Yes, good, that's better, since GPT decorates both the head and the tail of the device.

Altoid · 2022-02-14 11:06:20

Hello:

ralph.ronnquist wrote:

... to recover an DOS partition table ...
... run fdisk to create a DOS partition table ...
... run fsck as per superblock advice at post #7 ...
Might work ...

It's well worth trying ...

First I zapped with sgdisk:

~$ sudo sgdisk --zap-all /dev/sdf

***************************************************************
Found invalid GPT and valid MBR; converting MBR to GPT format
in memory.
***************************************************************

GPT data structures destroyed! You may now partition the disk using fdisk or other utilities.
~$

Then I used fdisk to create a partition and delete the iso9660 signature:

~$ sudo fdisk /dev/sdf

Welcome to fdisk (util-linux 2.33.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0x0be77ae3.

Command (m for help): n
Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)

Select (default p): p
Partition number (1-4, default 1):
First sector (2048-585937499, default 2048):
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-585937499, default 585937499):

Created a new partition 1 of type 'Linux' and of size 279.4 GiB.
Partition #1 contains a iso9660 signature.

Do you want to remove the signature? [Y]es/[N]o: y

The signature will be removed by a write command.

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
~$

Unfortunately I was not able to recover anything from fsck and any of the listed superblocks.
ie: 819200, 884736, 1605632, 2654208, 4096000, 7962624, 11239424, 20480000, 23887872 and 71663616

They were all corrupted. 8^|

At this point I decided to cut my losses and cleared the #$%& drive, formatted as ext4 and put it back on-line.
Fortunately, I was able to scrape back some files using PhotoRec but, as I posted earlier, it is a huge mess I have to sift through by hand.

My sincere thank you to all who pitched in.

Best,

A.

Andre4freedom · 2022-02-14 17:54:35

I'm sorry to hear, Altoid.
As I wrote, dd writes from the beginning of the disk/partition. Even only a smaller iso-file dumped to it will overwrite the partition-boot-block and a lot of important filesystem information, like the superblock, some inode-tables, directories, datablocks etc.
Even if you are successful in creating a new partition table or taking the one from the end of the blockdevice (GPT), the overwritten information is gone.
When you create a new filesystem and install an OS or copy data to it, the data is mostly written to the beginning of the filesystem. So there are more likely data erased there when an accidental dd wreaks its havoc. You may be able to recover some files if they and the related inode-tables are in the non-overwritten area. But blocks my still be missing since the filesystem can reuse emptied blocks if their files were once deleted and the associated file was extended, before the accident happened.
The explanation may not please - life can be miserable.

The officially official Devuan Forum!

#1 2022-02-11 00:34:51

Severe damage to ext4 filesystem - advice needed

#2 2022-02-11 01:04:15

Re: Severe damage to ext4 filesystem - advice needed

#3 2022-02-11 01:19:32

Re: Severe damage to ext4 filesystem - advice needed

#4 2022-02-11 02:39:02

Re: Severe damage to ext4 filesystem - advice needed

#5 2022-02-11 10:31:24

Re: Severe damage to ext4 filesystem - advice needed

#6 2022-02-11 21:29:21

Re: Severe damage to ext4 filesystem - advice needed

#7 2022-02-11 22:52:16

Re: Severe damage to ext4 filesystem - advice needed

#8 2022-02-12 10:38:21

Re: Severe damage to ext4 filesystem - advice needed

#9 2022-02-12 11:03:10

Re: Severe damage to ext4 filesystem - advice needed

#10 2022-02-12 11:05:26

Re: Severe damage to ext4 filesystem - advice needed

#11 2022-02-12 13:26:25

Re: Severe damage to ext4 filesystem - advice needed

#12 2022-02-12 17:19:39

Re: Severe damage to ext4 filesystem - advice needed

#13 2022-02-12 17:39:11

Re: Severe damage to ext4 filesystem - advice needed

#14 2022-02-13 15:00:10

Re: Severe damage to ext4 filesystem - advice needed

#15 2022-02-13 21:03:23

Re: Severe damage to ext4 filesystem - advice needed

#16 2022-02-13 21:12:04

Re: Severe damage to ext4 filesystem - advice needed

#17 2022-02-13 21:32:09

Re: Severe damage to ext4 filesystem - advice needed

#18 2022-02-14 11:06:20

Re: Severe damage to ext4 filesystem - advice needed

#19 2022-02-14 17:54:35

Re: Severe damage to ext4 filesystem - advice needed

Board footer