Server lost changes and partially reverted

dcolburn · 2023-01-05 17:26:57

OK, bad day here.

I'm suspecting a problem in the Raid 1 setup - but it may be elsewhere ...

After all of the work to get the web server running correctly - it's down, again.

I powered it down overnight then brought it back up.

Critical file content has changed - backwards - as I had previously observed/suspected and mentioned.

I deleted "grav" - it's back.

Everything in /etc/nginx including nginx.conf nftables.conf sites-available etc has reverted to what appear be a default versions.

All the work done in /var/www is also gone.

Curiously, ufw and gufw did not return (as before) but nftables is present.

/var/log/nginx access.log.1 is back to December.

access.log is empty

(I still don't know why it keeps creating an access.log.1 and an error.log.1)

error.log and error.log.1 only show my nginx.conf errors today

Ideas, please?

Last edited by dcolburn (2023-01-05 17:32:16)

golinux · 2023-01-05 17:43:35

Hammer?

Sorry couldn't resist . . .

You WILL behave for the machine!!

dcolburn · 2023-01-05 17:55:51

Something, that others have identified in the past, has to be causing this behavior.

I need some guidance as to the places to look.

There's no point in rebuilding until the stability of saved files has been resolved.

Is it possible Raid 1 is causing this?

A hardware problem?

A setting somewhere that's telling it to restore defaults?

rolfie · 2023-01-05 18:01:34

dcolburn wrote:

(I still don't know why it keeps creating an access.log.1 and an error.log.1)

The logger creates a new file when the log reaches a specified size, and adds a number to the older log. e.g. syslog becomes syslog.1, the current contents remains in a new file syslog. if there was a syslog.1 before it becomes syslog.2. As many versions are kept as specified.

This function is called log-rotate.

Last edited by rolfie (2023-01-05 18:13:16)

dcolburn · 2023-01-05 19:17:31

rolfie wrote:

dcolburn wrote:
(I still don't know why it keeps creating an access.log.1 and an error.log.1)
The logger creates a new file when the log reaches a specified size, and adds a number to the older log. e.g. syslog becomes syslog.1, the current contents remains in a new file syslog. if there was a syslog.1 before it becomes syslog.2. As many versions are kept as specified.
This function is called log-rotate.

It's not doing that on my server - it's creating a new one with only a few lines in the first.

It was, prior to today, also still adding to a log that already contained pages of log files - while also updating a new log.1.

I need guidance as to where to start looking for what's broken:

Raid 1

Devuan settings

??

Thanks

rolfie · 2023-01-05 20:42:43

I am not really an expert on logging, I just use whats available every now and then.

Devuan uses its own package called rsyslog. There is also a doc package for this. Maybe that is a start for you. Somehow the stuff can be configured, and maybe yours is off for some reason.

To the raid: I have had my share of experiences with using Raid0 and later on trying Raid1/10 mainly on U320 SCSI, I would call it a disaster. Thats why I personally would walk away from any Raid if I have the choice of using SSDs instead. A single SSD never will cause the kind of hickups you can experience with Raids. A good backup always is required anyhow, and this is the lifesaver, not a Raid1.

And I think good SSDs are more reliable than any disk drive.

dcolburn · 2023-01-05 20:54:00

I am using two SSD drives in a Raid 1 format.

Its purpose is to mirror drive 1 to drive 2.

I'm wondering if it's not working properly and the second drive is overwriting the first drive when I reboot?

dcolburn · 2023-01-05 23:19:38

Anything here look suspect, please?

root@devuan1:/etc/nginx# parted -l
Warning: Not all of the space available to /dev/sda appears to be used, you can
fix the GPT to use all of the space (an extra 6320 blocks) or continue with the
current setting? 
Fix/Ignore? I                                                             
Model: ATA THNSF8400CCSE (scsi)
Disk /dev/sda: 400GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system     Name  Flags
 1      17.4kB  538MB   538MB   fat32                 boot, esp
 2      538MB   48.5GB  48.0GB  ext4
 3      48.5GB  56.5GB  8000MB  linux-swap(v1)  swap  swap
 4      56.5GB  400GB   344GB   ext4

Warning: Not all of the space available to /dev/sdb appears to be used, you can fix the GPT to use all of the space (an extra 6320
blocks) or continue with the current setting? 
Fix/Ignore? I                                                             
Model: ATA THNSF8400CCSE (scsi)
Disk /dev/sdb: 400GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system     Name  Flags
 1      17.4kB  538MB   538MB   fat32                 boot, esp
 2      538MB   48.5GB  48.0GB  ext4
 3      48.5GB  56.5GB  8000MB  linux-swap(v1)  swap  swap
 4      56.5GB  400GB   344GB   ext4

dcolburn · 2023-01-05 23:29:19

Question: Is there any potential security risk if UUID's are posted in-the-clear on this forum?

dcolburn · 2023-01-06 02:45:36

root@devuan1:/etc/nginx# df -h
df: /run/user/0/doc: Operation not permitted
Filesystem      Size  Used Avail Use% Mounted on
udev             32G     0   32G   0% /dev
tmpfs           6.3G  1.2M  6.3G   1% /run
/dev/sdb2        44G  3.8G   38G  10% /
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs            15G     0   15G   0% /dev/shm
/dev/sdb1       513M  5.8M  507M   2% /boot/efi
/dev/sdb4       314G  102M  298G   1% /home
tmpfs           6.3G   12K  6.3G   1% /run/user/0
root@devuan1:/etc/nginx#

dcolburn · 2023-01-06 02:47:04

root@devuan1:/etc/nginx# lsblk -f
NAME   FSTYPE          FSVER            LABEL      UUID                                 FSAVAIL FSUSE% MOUNTPOINT
sda    isw_raid_member 1.1.00                                                                          
sdb    isw_raid_member 1.1.00                                                                          
├─sdb1 vfat            FAT32                       8395-4005                             506.3M     1% /boot/efi
├─sdb2 ext4            1.0                         xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx   37.7G     9% /
├─sdb3 swap            1                          xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx                [SWAP]
└─sdb4 ext4            1.0                         xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  297.8G     0% /home
sr0    iso9660         Joliet Extension Devuan 4.0 2021-10-12-11-25-10-00

dcolburn · 2023-01-06 03:04:27

I've been reading about sync ...

Is it possible that looking here ... https://linuxconfig.org/linux-software-raid-1-setup

... scrolling down to Configure persistent RAID mount ...

that the missing line in /etc/fstab "/dev/md0 /mnt/raid1 ext4 defaults 0 0" is the cause?

Is it correct that without this I don't have a working RAID 1 array that stays mounted even after a system reboot.

Could that cause the lost changes I'm experiencing?

Last edited by dcolburn (2023-01-06 03:05:01)

dcolburn · 2023-01-06 03:23:13

OK, need to shut down ... too many late nights and early mornings in a row.

Sure hope someone on the Forum has some ideas ...

ralph.ronnquist · 2023-01-06 03:56:30

Well, a composite raid1 device would indeed typically be named /dev/md0, but it doesn't necessarily need to be mounted. However it does need to be "assembled" at the first-stage boot (initrd) before this pivots into a raided root filesystem.

And yes, side mounting raided partitions separately is a sure way for interesting experiences.

But I'm not a raid supporter so it's better I'll be quiet and let a friendly raid1 supporter give some guidance.

steve_v · 2023-01-06 04:17:45

dcolburn wrote:

two SSD drives in a Raid 1 format.

Elaborate please.

dcolburn wrote:

root@devuan1:/etc/nginx# lsblk -f
NAME   FSTYPE          FSVER            LABEL      UUID                                 FSAVAIL FSUSE% MOUNTPOINT
sda    isw_raid_member 1.1.00                                                                          
sdb    isw_raid_member 1.1.00                                                                          
├─sdb1 vfat            FAT32                       8395-4005                             506.3M     1% /boot/efi
├─sdb2 ext4            1.0                         xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx   37.7G     9% /
├─sdb3 swap            1                          xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx                [SWAP]
└─sdb4 ext4            1.0                         xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  297.8G     0% /home
sr0    iso9660         Joliet Extension Devuan 4.0 2021-10-12-11-25-10-00

That "isw_raid_member" signature makes me suspect you are using BIOS "RAID", aka. Intel "Rapid Storage", in which case...

dcolburn wrote:

Is it possible that looking here ... https://linuxconfig.org/linux-software-raid-1-setup
... scrolling down to Configure persistent RAID mount ...
that the missing line in /etc/fstab "/dev/md0 /mnt/raid1 ext4 defaults 0 0" is the cause?

mdadm, (unless it has grown Intel RST support recently) has nothing to do with anything here. I also see no mention of any mdadm RAID arrays in that output.

If you really do have a software raid, mdadm (e.g. mdadm --examine --scan) should be able to find it, otherwise...

IIRC, Intel RST is handled by dm-mapper like any other BIOS fakeraid (ask google, I don't use it), and I expect your raid volume should be somewhere under /dev/mapper/... Not /dev/sdb2, which is where your root filesystem currently is.

What mounting an underlying member of a BIOS raid volume would do I don't know (will the BIOS resync on reboot? trusting which drive?), but the symptoms that you are reporting wouldn't surprise me at all.
This is all speculation mind, I don't use BIOS raid myself, and frankly I suggest you ditch it as well (unless you need to access the array from bare-metal windows for some deranged reason). Software (mdadm) RAID is better in every way.

Marjorie · 2023-01-06 13:53:39

I always understood 'fake' RAID/bios RAID doesn't work with Linux.

I do use mdadm RAID1 on my desktop machine, but not on my html/email server and it can help to cover disk failure and some forms of corruption but I would strongly recommend that you do an actual backup of your /root as well as your /home directories regularly to somewhere else. You may choose different solutions/frequencies for /root and /home.

However as you had finally got your nginx htmp server working that is when I would have made a backup of both.

The brtfs filesystem allows for snapshoting of your live system. You can also snapshot a ext4 system if it's inside a LVM2. There are also offline backup solutions where you boot from an alternative root system e.g. from a recovery disk on a usb drive and then run a backup of your unmounted filesystem from that.

Last edited by Marjorie (2023-01-06 13:54:58)

dcolburn · 2023-01-06 15:15:34

OK. So, basically, start from scratch.

@ralph_ronnquist what would be the equivalent-function mirroring method to Raid1 that isn't Raid1?

Is the alternative a complex process and one that requires extra attention? (I really want to get to some long-delayed content. This is my second attempt at a server - I sold the other hardware as it was way beyond what I needed and frustratingly complex.)

Is anyone up to walking me through setting up mdadm RAID1 (and making sure that the BIOS raid is disabled)?

OR, should I just follow this?

https://linuxconfig.org/linux-software-raid-1-setup

Note: I've just dedicated a 1TB external drive for backup purposes. That way the Raid1 array maintains a mirror and the external drive a backup. I guess I'll just go ahead and follow the instructions from the link and then see about the best way to implement the backup (I'd like to run it more often than 1x/day.)

Last edited by dcolburn (2023-01-06 16:41:57)

dcolburn · 2023-01-06 17:56:38

Guess what?

It's working, again!

Here's what I remember doing just now ...

I went into BIOS and toggled-off Raid.

Saved BIOS and let it boot ... files still missing.

I added that line to fstab and rebooted - files still missing.

I added a line to /etc/nginx/nginx.conf and rebooted - change to nginx.conf not there.

I shut down and restarted - cold - files still missing and the change to nginx.conf not there.

I just rebooted and everything is working, again ...

dcolburn · 2023-01-06 18:03:15

QUESTION: What is the recommended app, or command line string, to use to backup to the 1TB external drive, please?

Gnome Disk Utility looks promising.

dd would be OK ... unless over-tired and the wrong string is typed ...

Clonezilla and Tar also look interesting. https://www.maketecheasier.com/back-up-entire-hard-drive-linux/

Timeshift doesn't seem to save everything.

Last edited by dcolburn (2023-01-06 18:25:34)

dcolburn · 2023-01-06 22:02:06

Would this get everything & put it on the external 1TB USB drive? rsync -ra / /dev/sdc/backups/

ralph.ronnquist · 2023-01-06 22:15:03

Currently I have duplicity at the top of my list of timeline backup methods as it's both fast and compact, and it's trivially easy to set up a cron job script that makes an incremental delta as often as I like.

I know of the front-end duply but only by name. (It's so rare that I need to peep into the backup)

Marjorie · 2023-01-06 22:48:19

Backing up.

To backup my /home partition I would use something that supports incremental back where it's possible to restore individual files or directories. I use Déjà Dup which is just a wrapper around duplicity.

To backup my \root partition I use fsarchiver which does a whole partition backup. Although it is possible to backup a live partition (using the -A flag) it's much safer to unmount it and backup from another system (e.g one on a usb stick). If you the have LVM partitions (which I do, on top of RAID1) you can snapshot your live root partition.

Last edited by Marjorie (2023-01-06 22:50:30)

dcolburn · 2023-01-07 01:15:43

ralph.ronnquist wrote:

Currently I have duplicity at the top of my list of timeline backup methods as it's both fast and compact, and it's trivially easy to set up a cron job script that makes an incremental delta as often as I like.
I know of the front-end duply but only by name. (It's so rare that I need to peep into the backup)

Do I read correctly that writing to a "local filesystem" would include a USB drive?

dcolburn · 2023-01-07 01:44:32

Marjorie wrote:

Backing up.
To backup my /home partition I would use something that supports incremental back where it's possible to restore individual files or directories. I use Déjà Dup which is just a wrapper around duplicity.
To backup my \root partition I use fsarchiver which does a whole partition backup. Although it is possible to backup a live partition (using the -A flag) it's much safer to unmount it and backup from another system (e.g one on a usb stick). If you the have LVM partitions (which I do, on top of RAID1) you can snapshot your live root partition.

Deja Dup is awesome. It's a little less than intuitive at points but after thrashing about a bit I have it backing up everything but the Trash folder. lol

Thanks!

dcolburn · 2023-01-07 01:53:55

Anyone using BIND or Knot Resolver as a DNS resolver plus DNSdist as a DNS load balancer in Devuan?

Any reason doing so would be a bad idea in Devuan?

It's her ("linuxbabe") fault that I'm looking into this ... https://www.linuxbabe.com/ubuntu/dns-over-https-doh-resolver-ubuntu-dnsdist lol

Last edited by dcolburn (2023-01-07 16:58:07)

The officially official Devuan Forum!

#1 2023-01-05 17:26:57

Server lost changes and partially reverted

#2 2023-01-05 17:43:35

Re: Server lost changes and partially reverted

#3 2023-01-05 17:55:51

Re: Server lost changes and partially reverted

#4 2023-01-05 18:01:34

Re: Server lost changes and partially reverted

#5 2023-01-05 19:17:31

Re: Server lost changes and partially reverted

#6 2023-01-05 20:42:43

Re: Server lost changes and partially reverted

#7 2023-01-05 20:54:00

Re: Server lost changes and partially reverted

#8 2023-01-05 23:19:38

Re: Server lost changes and partially reverted

#9 2023-01-05 23:29:19

Re: Server lost changes and partially reverted

#10 2023-01-06 02:45:36

Re: Server lost changes and partially reverted

#11 2023-01-06 02:47:04

Re: Server lost changes and partially reverted

#12 2023-01-06 03:04:27

Re: Server lost changes and partially reverted

#13 2023-01-06 03:23:13

Re: Server lost changes and partially reverted

#14 2023-01-06 03:56:30

Re: Server lost changes and partially reverted

#15 2023-01-06 04:17:45

Re: Server lost changes and partially reverted

#16 2023-01-06 13:53:39

Re: Server lost changes and partially reverted

#17 2023-01-06 15:15:34

Re: Server lost changes and partially reverted

#18 2023-01-06 17:56:38

Re: Server lost changes and partially reverted

#19 2023-01-06 18:03:15

Re: Server lost changes and partially reverted

#20 2023-01-06 22:02:06

Re: Server lost changes and partially reverted

#21 2023-01-06 22:15:03

Re: Server lost changes and partially reverted

#22 2023-01-06 22:48:19

Re: Server lost changes and partially reverted

#23 2023-01-07 01:15:43

Re: Server lost changes and partially reverted

#24 2023-01-07 01:44:32

Re: Server lost changes and partially reverted

#25 2023-01-07 01:53:55

Re: Server lost changes and partially reverted

Board footer