Server lost changes and partially reverted

dcolburn · 2023-01-07 04:00:06

Server settings are not holding across a Shutdown - not even persisting across a Restart - the one that worked seems to be a fluke at the moment.

I'm checking some BIOS settings - Raid is Off, ACHI is On, I unchecked drives not present, Integrated NIC changed to Enabled from Enabled with PXE, UEFI Boot Path Security was set by default at Never, Advanced Boot Options are both checked - Enable Legacy Option ROMS & Enable Attempt Legacy Boot, Boot Sequence is Debian then Windows Boot Manager (Do both need to be checked?), Boot list option is UEFI (Legacy is unchecked).

Anything seem 'off', please?

EDIT: On reboot everything is working, again ...

Time to sleep ... will return to this tomorrow.

Last edited by dcolburn (2023-01-07 04:01:28)

Andre4freedom · 2023-01-07 10:24:18

dcolburn,
I'm trying to help you. I am building up a system that reflects your configuration, more or less.
System using EFI mode (but secure boot disabled!!!)
2 disks in a Software-RAID1 setup
Devuan Daedalus (not recommended, use Chimaera as it's current stable)
I will write the procedure for you to follow - if you wish so.

I have a production server running for years without any problems. It's an enterprise server (meaning BIOS mode).
It runs with a RAID1 setup (mdadm, no HW-RAID) and never fails me.
It runs with Devuan Beowulf.

I will not cover backup, this is another topic.
My procedure will take a little time.
Good day!

dcolburn · 2023-01-07 14:02:35

I look forward to reading your procedure for a correct setup.

I have backup covered now (Deja Dup). At least that piece of the puzzle appears to be stable.

Thanks!

EDIT: Just rebooted into my User account vs Root and the server is also working OK, so it's not a difference between accounts (permissions, etc).

I found just this, which may be helpful ...

Cold start refers to starting the CPU from power off. Current configuration is discarded and program processing begins again with the initial values. Warm Start. Warm start refers to restarting the CPU without turning the power off. Program processing starts once again where Retentive data is retained.

Source: https://forums.debian.net/viewtopic.php?t=137068

EDIT 2: Is this telling me anything relevant?

root@devuan1:~# df -T -h /etc/inittab
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/sda2      ext4   44G  4.0G   38G  10% /
root@devuan1:~#

Last edited by dcolburn (2023-01-08 03:39:17)

Andre4freedom · 2023-01-08 12:08:34

Colburn,
I have done some testing and research to your problems.
First of all it seems that your computer/server/mainboard has not been set up properly.
There are 2 ways to operate hardware and its OS to be installed: UEFI or classic MBR type. This affects the boot process and how the disks are used.

1. UEFI
=======
This is a very tricky new solution. EFI is a good solution to overcome limitations to the classic PC restrictions. It requires you to label and format disks in GPT manner.
Now U-EFI, provoked by big tech companies adds complexities to "increase" security against malware, but it's mainly used to lock out any software that is NOT "certified" by Microsoft.
To avoid that UEFI snag, you have to disable the "Secure Boot" option in the mainboard (or BIOS) setup.

To do that, I suggest to reset the mainboard to factory defaults and the selectively adjust the few settings you really need:
- disable secure boot
- keep the board in EFI mode
- check the boot-order (priority) for the system to boot from
- Install from a CD, DVD or USB-stick (select the UEFI-USB install media)
- Scrub your disks and fdisk them to GPT and so on (I suppose you let the Devuan installer do it, it does it well!)

2. MBR / BIOS mode
==================
Most enterprise level servers work in that mode. Ask an datacenter operator.
There are no worries about secure boot etc.

To do that, I suggest to reset the mainboard to factory defaults and the selectively adjust the few settings you really need:
- check the boot-order (priority) for the system to boot from
- Install from a CD, DVD or USB-stick (select the USB install media)
- Scrub your disks and fdisk them to MBR (DOS Label) and so on (I suppose you let the Devuan installer do it, it does it well!)

3. RAID
=======
If your motherboard or system includes a Hardware RAID controller, make sure you know it well and follow the manufacturers indications for the OS setup. I have worked quite a lot with such kind of hardware and it worked well (Dell PERC) Others are quite problematic, because usually they let you access RAID members from the OS, which is a very bad thing!
If in doubt, switch the controllers to simple SCSI or SATA controller through-mode and disable the RAID functions on that hardware.
Then install Devuan using the software RAID option. This works really well.

4. My experience with an UEFI board and SW-RAID
===============================================
My system worked well with one single disk, all in UEFI and GPT mode. That's why I added a disk and tried to set it up with the Devuan installer to assemble the RAID and partitions. I failed miserably.
I switched the system to BIOS/MBR mode, relabeled the disks to DOS Labels and set up the thing all using the Devuan Daedalus installer (Netinstall)
It worked really well, like a charm.

5. Example setup
================
- Reset the computer to factory defaults
- Set options sensibly as mentioned above
- Boot the Devuan Netinstall stick
- Choose the standard install method (just "Install")
- Give all answers accordingly to your needs
- Enter the Disk Partitioner:
-- Remove any residual config whatsoever There shall be no LVM volumes, no LV Groups, no RAID partitions, just nothing.
-- Create a 512MB primary partition, the first, on each disk and set the type to type fd (Linux RAID). (sda1 sdb1)
-- Create a very big primary or extended partition. the second, on each disk and set the type to fd (Linux RAID). (sda2 sdb2)
-- Go to the RAID submenu of the partitioner and create the the RAID volumes.
(Assemble sda1 and sdb1 to md0, and sda2 and sdb2 to md1)
-- Now back in the main partition menu, define md0 as the /boot partition (ext3 or ext4)
-- Enter the LVM submenu (Logical Volume Manager)
1. Declare md1 a LVM physical volume
2. Create a Volume Group vg1 and add md1 as a member
3. In that volumegroup create the volumes you need:
-- 40 GB for root (ext4) (/)
-- 40++ GB for home (ext4) (/home)
-- ??GB for swap (swap) (size depends on you RAM in the system)
-- ??GB for server-uses (ext4) (/srv)
-- Now back in the main partion menu, define the for logical volumes as shown above
(I will show the filesystem layout at the end)
- Now continue your installation, tasksel does a wonderful job. (I chose Devuan Desktop, XFCE, web server, Console productivity and SSH server)
- !!! at the end, don't just hit continue! chose Back instead and start a shell in the install environment. There you type
grub-install /dev/sda and grub-install /dev /sdb (That makes both disks bootable.
Note: this is absolutely the one and only instance when you must access the RAID members of md0. It only copies one boot block to each disk.
- Here, you can reboot. It worked for me.
- Hint1: remove the anacron scheduler, it's of no use on a server and only causes problems at reboot
- Hint2: add your software as needed, web servers, ftp servers, database servers etc. They all should create and use subdirectories beneath /srv
- Hint3: LVM2 is an excellent way to handle disk space. It allows for snapshots even.
- Hint4: Backup to devices that are formatted (i.e. ext4) and mounted temporarily to /mnt. Never backup to device nodes like /dev/sdxyz. Tape drives did that.

I hope that helps.

A few notes:
Never, in the whole server's life, access RAID or LV members or elements of it. That is a sure way to hell. The only exception is the grub-install after kernel- or GRUB-updates.
RAIDed Volumes are accessed using /dev/md0 (for mounting, fsck etc) but don't touch /dev/md1 in our case.
Logical Volumes are accessed using /dev/mapper/VGx-lv-yyy (for mounting, fsck etc)
All filesystem activities as also backups etc have to access the corresponding mountpoints.

Now the configuration on my test server:
***********************************************************
linuxadmin@devuan:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 7.8G 0 7.8G 0% /dev
tmpfs 1.6G 1.1M 1.6G 1% /run
/dev/mapper/VG0-lv--root 38G 3.8G 32G 11% /
tmpfs 5.0M 8.0K 5.0M 1% /run/lock
tmpfs 3.1G 0 3.1G 0% /dev/shm
/dev/md0 446M 52M 369M 13% /boot
/dev/mapper/VG0-lv--home 47G 1.8M 45G 1% /home
/dev/mapper/VG0-lv--srv 339G 28K 322G 1% /srv
cgroup_root 10M 0 10M 0% /sys/fs/cgroup
tmpfs 1.6G 12K 1.6G 1% /run/user/1000
tmpfs 1.6G 4.0K 1.6G 1% /run/user/109
***********************************************************
linuxadmin@devuan:~$ lsblk -f
NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS
sda
├─sda1 linux_raid_member 1.2 devuan:0 d67d9792-6c1c-c10f-c183-ccdfd63fb72a
│ └─md0 ext3 1.0 BOOT f16202ab-b30f-40df-95e5-9c18998b9b60 368.9M 12% /boot
└─sda2 linux_raid_member 1.2 devuan:1 f1e96c15-2935-8021-39ee-71cba1fab584
└─md1 LVM2_member LVM2 001 NEkYz6-31wg-ymNm-xR09-98J3-RavJ-0TZrZK
├─VG0-lv--root 31.6G 10% /
├─VG0-lv--home 44.2G 0% /home
├─VG0-lv--swap [SWAP]
└─VG0-lv--srv 321.5G 0% /srv
sdb
├─sdb1 linux_raid_member 1.2 devuan:0 d67d9792-6c1c-c10f-c183-ccdfd63fb72a
│ └─md0 ext3 1.0 BOOT f16202ab-b30f-40df-95e5-9c18998b9b60 368.9M 12% /boot
└─sdb2 linux_raid_member 1.2 devuan:1 f1e96c15-2935-8021-39ee-71cba1fab584
└─md1 LVM2_member LVM2 001 NEkYz6-31wg-ymNm-xR09-98J3-RavJ-0TZrZK
├─VG0-lv--root 31.6G 10% /
├─VG0-lv--home 44.2G 0% /home
├─VG0-lv--swap [SWAP]
└─VG0-lv--srv 321.5G 0% /srv

***********************************************************
linuxadmin@devuan:~$ cat /proc/mdstat
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md1 : active raid1 sda2[0] sdb2[1]
468618240 blocks super 1.2 [2/2] [UU]
bitmap: 1/4 pages [4KB], 65536KB chunk

md0 : active raid1 sda1[0] sdb1[1]
497664 blocks super 1.2 [2/2] [UU]

unused devices: <none>
***********************************************************
linuxadmin@devuan:~$ lvshow
-bash: lvshow: command not found
linuxadmin@devuan:~$ man lvm
linuxadmin@devuan:~$ pvscan
-bash: pvscan: command not found
linuxadmin@devuan:~$ sudo pvscan
[sudo] password for linuxadmin:
PV /dev/md1 VG VG0 lvm2 [<446.91 GiB / 0 free]
Total: 1 [<446.91 GiB] / in use: 1 [<446.91 GiB] / in no VG: 0 [0 ]
linuxadmin@devuan:~$ sudo vgscan
Found volume group "VG0" using metadata type lvm2
linuxadmin@devuan:~$ sudo lvscan
ACTIVE '/dev/VG0/lv-root' [38.14 GiB] inherit
ACTIVE '/dev/VG0/lv-home' [47.68 GiB] inherit
ACTIVE '/dev/VG0/lv-swap' [15.83 GiB] inherit
ACTIVE '/dev/VG0/lv-srv' [<345.25 GiB] inherit
linuxadmin@devuan:~$

Link to a excellent article that hints to the intricacies of UEFI-RAID-LVM

https://askubuntu.com/questions/1299978 … -uefi-bios
https://unix.stackexchange.com/question … oft-raid-1

I hope these (lengthy) instructions help.
If you need clarification, I will try to help. Good luck.

Head_on_a_Stick · 2023-01-08 13:00:00

Some notes:

Andre4freedom wrote:

EFI is a good solution to overcome limitations to the classic PC restrictions. It requires you to label and format disks in GPT manner.

The official UEFI standard does actually support booting from an MS-DOS partition table. Whether the UEFI firmware in question supports that option is another matter entirely. I have two recent ThinkPads (E14 & P14s) that will boot in UEFI mode from an MS-DOS partition table.

Andre4freedom wrote:

Now U-EFI, provoked by big tech companies adds complexities to "increase" security against malware, but it's mainly used to lock out any software that is NOT "certified" by Microsoft.

I use SecureBoot with my own keys to lock out Microsoft. See also https://www.rodsbooks.com/efi-bootloade … ng-sb.html.

Andre4freedom wrote:

you have to disable the "Secure Boot" option in the mainboard (or BIOS) setup.

Devuan supports SecureBoot and should work just fine with it enabled. Recent machines may require that "third-party" certificates be allowed though. For example: https://download.lenovo.com/pccbbs/mobi … re_PCs.pdf

dcolburn · 2023-01-08 14:01:34

Wow, @Andre4freedom, that's an awesome bit of documenting the process! Devuan folks might consider adding it (with a few edits) to the Wiki.

@Head_on_a_stick Thanks re. points of clarification.

I gather that you both agree that I need to, essentially, wipe the ssd's and start from scratch?

I have a backup of the file structure in working order. So I won't need to reinvent all of that work.

dcolburn · 2023-01-09 18:24:19

I don't know why there's 1.0 MB at the top of sdb - I selected "Beginning" for the 521 MB partition.

SCSI1 (0,0,0) (sda) - 400.1 GB ATA THNSFxxxxx
#1 primary 510.7 MB K raid
pri/log 399.6 GB FREE SPACE
SCSI3 (0,0,0) (sdb) - 400.1 GB ATA THNSFxxxxx
1.0 MB FREE SPACE
#1 510.7 MB K raid
399.6 GB FREE SPACE

Head_on_a_Stick · 2023-01-09 18:43:36

What is that output from? How did you create this new partition table?

Correct alignment in a disk with a 512 byte sector size does require the partitions to start at sector 2048, which is exactly 1MiB (1,048,576 bytes). It most certainly is not "1.0 MB" though.

Check

# fdisk -l /dev/sdb

That should show the first partition starting at sector 2048.

And also

# parted /dev/sdb align-check opt 1

That should say 1 aligned.

No idea about RAID though, I've never used that.

Last edited by Head_on_a_Stick (2023-01-09 18:44:46)

dcolburn · 2023-01-09 18:57:53

I've gone back the the beginning. Reset BIOS and am now in Install.

I'm at this step in the instructions from @Andre4freedom ...

- Enter the Disk Partitioner:
--    Remove any residual config whatsoever There shall be no LVM volumes, no LV Groups, no RAID partitions, just nothing.
--    Create a 512MB primary partition, the first, on each disk and set the type to type fd (Linux RAID). (sda1 sdb1)

I don't think that I have access to Terminal yet ...

Last edited by dcolburn (2023-01-09 20:19:54)

dcolburn · 2023-01-09 20:34:12

Head_on_a_Stick wrote:

Correct alignment in a disk with a 512 byte sector size does require the partitions to start at sector 2048, which is exactly 1MiB (1,048,576 bytes). It most certainly is not "1.0 MB" though.
Check
# fdisk -l /dev/sdb
That should show the first partition starting at sector 2048.

Why would the Devuan install program allow this?

If it won't work why doesn't it error - or just force the leading blank space as it did for sdb?

Head_on_a_Stick · 2023-01-09 21:03:29

I think the installer has aligned the disk correctly. Please run my suggested commands to confirm this. I don't post shit like that just for fun...

rolfie · 2023-01-09 21:07:46

For such jobs I have a gparted life cd available to boot on usb. Write a new partition table, and then restart the Devuan installer and use manual partitioning.

And to get to another console while installing: use ALT-F2. There you can enter available commands. Back to installer with ALT-F1.

Last edited by rolfie (2023-01-09 21:09:18)

dcolburn · 2023-01-10 00:26:55

Head_on_a_Stick wrote:

I think the installer has aligned the disk correctly. Please run my suggested commands to confirm this. I don't post shit like that just for fun...

I appreciate your support.

So, I should go ahead with the instructions until the new install has progressed to where I can access Terminal - then see how things look (using your suggested commands)?

I'm unaware of a way to use Terminal in the middle of partitioning the disks.

dcolburn · 2023-01-10 00:28:22

rolfie wrote:

For such jobs I have a gparted life cd available to boot on usb. Write a new partition table, and then restart the Devuan installer and use manual partitioning.
And to get to another console while installing: use ALT-F2. There you can enter available commands. Back to installer with ALT-F1.

OK, cool, just read this ... will give it a try and report back.

I seem to have a way of breaking stuff so I tend to be over-cautious ...

dcolburn · 2023-01-10 00:36:37

Head_on_a_Stick wrote:

And also
# parted /dev/sdb align-check opt 1
That should say 1 aligned.

When I ran this # parted /dev/sdb align-check opt 1 it asked about optimal or something (it's scrolled out of sight) - I chose the default - then it asked which partition and I entered 4 and it errored.

When I tried this # parted /dev/sdb align-check opt 1 ailgned it scrolled out the Help menu - so I obviously misunderstood what you wrote here. Sorry - a lot of this is new to me.

dcolburn · 2023-01-10 00:44:23

I took a picture of the output from # fdisk -l /dev/sdb ... https://www.sun2save.com/images/misc1/partitioningerror1.jpg

Head_on_a_Stick · 2023-01-10 06:42:35

The strange parted output is because you're trying to run it from the installer's shell, which uses busybox's gimped version. Why didn't you just load a live ISO image with a graphical desktop? Then you could have started a browser and just pasted the terminal output here directly. Work smarter, not harder.

Anyway the (busybox) fdisk output shows that all of the partitions on /dev/sdb are misaligned. This will result is poor disk I/O performance. I did ask how that partition table was created but you ignored me. Nice.

Whatever you used to create that partition table is a buggy pile of shite. Do not use it again. I would recommend either gdisk for GUID partition tables (as is present on /dev/sdb at the moment) or fdisk for MS-DOS ("MBR" type) partition tables, but don't use the busybox versions. Load up a live ISO image and use the full versions instead.

And what exactly are you using /dev/sdb for? It has an EFI system partition and you should only have one of those per machine. That being the case I would expect the ESP to be present on the "main" system disk, which would usually be /dev/sda.

It is almost impossible to offer any reasonable advice without knowing the exact requirements so help me to help you by supplying more information. This thread is a useless shit-show for anybody else now so we might as well use it as your general help desk to stop you making more useless threads elsewhere.

dcolburn · 2023-01-10 14:29:54

I responded that I created the partition table by following the instructions from Andre4freedom - I didn't know any other way to answer your question. I'm not trying to be difficult - I simply don't know the answers to the level of precision you seek.

As instructed, I'm using the Devuan installer app from the Devuan repo that I burned to a DVD. If it's an awful tool - I have no way to know that. I'm trusting it.

The only thing I know about what I'm doing is that I'm following the instructions from Andre4freedom (see them, posted above).

If those instructions are incorrect then I'll need to start over with better ones. Sigh.

Head_on_a_Stick · 2023-01-10 18:44:48

Sorry dcolburn, I was in a foul mood this morning. My tone was uncalled far and I apologise.

dcolburn · 2023-01-10 19:23:11

Head_on_a_Stick wrote:

Sorry dcolburn, I was in a foul mood this morning. My tone was uncalled far and I apologise.

I'm sorry that you're being stressed like that. It's no fun.

All is forgiven. Life is far too short to be unforgiving.

I know I ask a lot of questions and am somewhat clueless when I need to respond accurately - I'm sure it's frustrating to those who really know this stuff.

I'm anxious to get this up and stable so I can restore all the saved work - and get the web hosting back up - then, to finally actually build the intended content.

I feel as though I need the Raid1 so a primary disk failure doesn't take the server down for a long time.

I'm making snapshots to an external USB drive using Deja Dup.

I also have a second computer that's pretty much identical, except that the drives are HDD vs SSD - which I plan to air-gap (if lightning gets into the house it should survive). The plan so to mirror to it daily (but that project is down the road).

So, what do I need to do next to be sure the partitioning on this one is correct, please?

Head_on_a_Stick · 2023-01-10 20:09:24

If you use fdisk or gdisk they will automatically align the partitions. They're the only partitioning tools I use so I don't know if there are others that work properly in that respect.

dcolburn · 2023-01-11 03:30:18

Andre4freedom wrote:

- Enter the Disk Partitioner:
-- Remove any residual config whatsoever There shall be no LVM volumes, no LV Groups, no RAID partitions, just nothing.
-- Create a 512MB primary partition, the first, on each disk and set the type to type fd (Linux RAID). (sda1 sdb1)
-- Create a very big primary or extended partition. the second, on each disk and set the type to fd (Linux RAID). (sda2 sdb2)
-- Go to the RAID submenu of the partitioner and create the the RAID volumes.
(Assemble sda1 and sdb1 to md0, and sda2 and sdb2 to md1)

OK, so I used fdisk to clear out the old and build the new partitions.

That included Type #29 for Linux RAID.

I then returned to the Install and Disk Partitioner.

https://www.sun2save.com/images/misc1/n … tempt1.jpg

I followed the steps there to "create the RAID volumes" ...

https://www.sun2save.com/images/misc1/n … tempt2.jpg

Do things look OK to this point, please?

Head_on_a_Stick · 2023-01-11 06:38:18

I can't tell from those screens.

Correct alignment requires that the partitions start at sector 2048 (for a disk with 512 byte sectors, as yours seems to be) with subsequent partitions aligned to 1MiB boundaries.

Reference: https://wiki.archlinux.org/title/Advanc … _alignment

Those screens say the partitions start at "1.0 MB" but 1MB is not the same as 1MiB (2048 512 byte sectors or 1,048,576 bytes). I don't know if that screen is actually showing MB or if it is showing MiB but displaying the wrong units. You will have to check that yourself.

Last edited by Head_on_a_Stick (2023-01-11 06:39:55)

dcolburn · 2023-01-11 15:46:46

When I used fdisk the stated first partition sector default was 2048 - so I accepted the default. It should be OK.

I'll now need the folks who use RAID to instruct me if I got that part correct as well.

Thanks!

Andre4freedom · 2023-01-11 16:47:24

Hello dcolburn,
your disk partitioning looks right (https://www.sun2save.com/images/misc1/newpartitionattempt1.jpg)
But how have you assembled the RAID devices? RAID Device 0 should become md0 and contain /dev/sda1 and /dev/sdb1
RAID Device 1 should become md1 and consist of your bigger partitions, sda2 and sdb2
You can do that in your installer program, in the "Configure Software RAID" sub-menu and it does the job right.
Once you have md0 and md1, you can proceed following the instructions. md0 should be your /boot partition. md1 is an excellent base for a LVM2 Volume Group vg0.
Good luck!

The officially official Devuan Forum!

#26 2023-01-07 04:00:06

Re: Server lost changes and partially reverted

#27 2023-01-07 10:24:18

Re: Server lost changes and partially reverted

#28 2023-01-07 14:02:35

Re: Server lost changes and partially reverted

#29 2023-01-08 12:08:34

Re: Server lost changes and partially reverted

#30 2023-01-08 13:00:00

Re: Server lost changes and partially reverted

#31 2023-01-08 14:01:34

Re: Server lost changes and partially reverted

#32 2023-01-09 18:24:19

Re: Server lost changes and partially reverted

#33 2023-01-09 18:43:36

Re: Server lost changes and partially reverted

#34 2023-01-09 18:57:53

Re: Server lost changes and partially reverted

#35 2023-01-09 20:34:12

Re: Server lost changes and partially reverted

#36 2023-01-09 21:03:29

Re: Server lost changes and partially reverted

#37 2023-01-09 21:07:46

Re: Server lost changes and partially reverted

#38 2023-01-10 00:26:55

Re: Server lost changes and partially reverted

#39 2023-01-10 00:28:22

Re: Server lost changes and partially reverted

#40 2023-01-10 00:36:37

Re: Server lost changes and partially reverted

#41 2023-01-10 00:44:23

Re: Server lost changes and partially reverted

#42 2023-01-10 06:42:35

Re: Server lost changes and partially reverted

#43 2023-01-10 14:29:54

Re: Server lost changes and partially reverted

#44 2023-01-10 18:44:48

Re: Server lost changes and partially reverted

#45 2023-01-10 19:23:11

Re: Server lost changes and partially reverted

#46 2023-01-10 20:09:24

Re: Server lost changes and partially reverted

#47 2023-01-11 03:30:18

Re: Server lost changes and partially reverted

#48 2023-01-11 06:38:18

Re: Server lost changes and partially reverted

#49 2023-01-11 15:46:46

Re: Server lost changes and partially reverted

#50 2023-01-11 16:47:24

Re: Server lost changes and partially reverted

Board footer