Is Refracta-snapshot and installer available/working in Excalibur?

greenjeans · 2024-11-28 17:41:47

I have the last net install iso downloaded (arrgh at debian installer), was thinking about installing, updating, then rolling up a current hybrid iso.

Was thinking it might be helpful for the average Devuan user, an easy way to do some testing and report back.

fsmithred · 2024-11-28 19:13:24

Yes, snapshot and installer versions in excalibur/ceres work. I've been making desktop-live isos about once a month, but I have not released any because I was hoping that installer isos would appear and get tested first. We're still waiting for that to happen.

greenjeans · 2024-12-01 20:03:04

Got to try it last night (Daedalus), worked great as always!

One question, in Refractasnapshot.conf, both options to use xz were commented out, I uncommented the lower one and used that as that is what I always used in Vuu-do, but if I had left both commented out what would it have used for compression?

fsmithred · 2024-12-01 20:53:27

Yeah, that second one is the one you provided. Default is to use gzip.

Squashfs is a highly compressed read-only filesystem for Linux. It uses either gzip/xz/lzo/lz4/zstd compression to compress both files, inodes and directories.

greenjeans · 2025-01-02 18:06:32

One last question about snapshot. I noticed zstd is in the list quoted above, is it possible to use that for compression in snapshot?

My thinking may be way off base here, but was reading up on different compression methods and noticed that while it won't compress quite as small as xz, that it's decompression speeds are several orders of magnitude faster.

Use case scenario for that, that i'm interested in, is on a liveUSB, after it boots there is a lag time when you first open the menu or any programs, after that it's cached and plenty fast, but was just wondering if zstd would cut down that lag time.

Again, on a modern fast machine probably never notice, but on the 10-12 year old low-spec machines I typically have to deal with it seems like it might help?

ETA: I know the real bottleneck is going to be transfer speed from usb 2, but just looking for small speed tweaks here.

Last edited by greenjeans (2025-01-02 18:11:50)

fsmithred · 2025-01-02 22:00:30

You can change the compression type for the squashfs in /etc/refractasnapshot.conf. Let me know if zstd works. I'm on chimaera at the moment and it doesn't seem to be an option.

For faster response on opening programs/files, use the toram option. For a live usb use toram=/path/to/filesystem.squashfs
Go make coffee while you're waiting for it to boot.

greenjeans · 2025-01-02 23:47:30

Go make coffee while you're waiting for it to boot.

Lol, I usually go have a smoke. Yeah I usually load into ram for my stuff, the iso's are small and I have 4 gigs of ram, I was thinking more specifically for someone with less ram, like 1 gig, it'll fit in there but you won't be able to do much if anything else, so for those folks the regular load is better and might work fine. It's just a tiny tweak but with a lot of others they add up.

I'd like to give zstd a whirl, and check all times vs. xz. I know it's gonna be a little bigger iso, probably close to 4% bigger, but I have a little breathing room in my size goals for now.

mksq_opt="-comp zstd --ultra -22 -T0"

seems about right. Wish me luck, i'll report back here in a bit.

greenjeans · 2025-01-03 02:55:40

Okay, posting now from an iso I just made using zstd, the Vuu-do openbox mini, running off a stick (usb 2) that I loaded Chromium into during this live-session. Totally worked to make a useable iso. Command I posted above and tried first error'ed out, I should have been paying more attention to the mksquashfs man page than the zstd man page. Thankfully the Refracta-snapshot log showed me not only the error of my ways but also how to fix it:

mksq_opt="-comp zstd -Xcompression-level 22"

All done on my Compaq CQ-58 circa 2012, amd dual-core APU @ 1.0 ghz.

In practice on the mini iso, 2.4 gigs unpacked, the highest level of compression for zstd made an iso about 10% larger than xz -Xbcj x86 did.

651 mb vs 712 mb.

The next part is subjective, I don't have a stopwatch to check small intervals. But after loading the iso to a USB, and using the regular boot option, NOT load-to-ram, the boot time was faster, the lag times when opening the menu and programs practically disappeared and it was easily as fast as I normally get loading the whole thing to ram. Fast as hell.

And after bootup before running anything, I was at 479 mb of total ram use as opposed to the 1.14 gigs when I loaded the system entirely to ram. The task manager showed actual usage by the services running at about 300 mb.

Obviously I need to do a whole lot more testing, and get some actual timing for things, I could have some confirmation-bias.

But, impressions:

1. zstd most definitely does not compress as small as xz when doing an iso. About a 10% difference.
2. The bottleneck of using an optical disc (CD in this case) is too much for this to make a difference.
3. A liveUSB2 though has about 5 times faster transfer speed than my disc-drive, and in practice it's still a small bottleneck, but most programs like the file manager, text-editor, terminal etc. are pretty small, and get transferred fast enough, and de-compressed noticeably faster than xz does.
4. I should be able to run an iso that's a gig in size on a liveUSB stick, and actually use it with only a gig of ram since it doesn't all need to get loaded and the speed doesn't suffer as a result of not loading it all to ram.

Fun stuff, gotta go do some more testing!

greenjeans · 2025-01-04 05:03:01

Running right now on a Corsair 32 gb stick several years old, but looks pretty much the same as the ones they sell now at wally world where I got it. Ran an iso of the Vuu-do max using zstd, came out as expected about 10% larger at 1.1 gb.

This usb is smoking the hard-drive, it boots so much faster, and here's the thing that's blowing my mind and making me super happy: Even when I load it all to ram it's still faster than the HD boot.

I looked up specs on this very stick and tests showed an average 22 mbs read and a 10 mbs write. But when I load it to ram I can watch it loading the largest file (squashfs) and it's loading at 30 mbs. The HD on this old machine usually runs around 35-45 in practice, so loading from usb is only about 10-30% slower, but it's loading 3 times the data.

It's compressed data, but zstd decompresses it 13 times faster than xz does. I didn't have time for a smoke or to make a cup of coffee before it booted.

Procedure I followed:

Add entry to /etc/refractasnapshot.conf, enable, and comment out others. mksq_opt="-comp zstd -Xcompression-level 22" Run iso.

Run Gparted on usb stick: new partition table, two new primary partitions both fat32, first slightly larger than iso with boot flag, second rest of stick labeled "DATA".

Refracta2usb: option iso2 findiso, install syslinux, edit boot menu to add toram and failsafe entries and remove others, syslinux boot splash ugly, replaced with boot splash from onboard isolinux that I made. Syslinux highlight color, also ugly, changed that.

@fsmithred you gotta try this, you're gonna love it.

fsmithred · 2025-01-04 12:48:44

I already did try it yesterday. The iso with zstd booted about 5 seconds faster than xz in qemu. That was about 1 minute, 20 seconds of which were at the boot menu. Building the iso was about 2 minutes faster (9 vs. 11). I haven't tried it on usb yet.

This will be in the next build of refractasnapshot. I already added support for zstd in the initramfs.

greenjeans · 2025-01-04 16:42:49

Nice! I really begrudge the extra size, you know how I am, but the extra speed is really nice especially on a usb, haven't installed one of the z-iso's yet from a live-stick but i'm guessing there might even be some time picked up there too.

greenjeans · 2025-01-06 19:07:09

So back to subject of original post, I have made a working version of the openbox-mini in excalibur. Rolled up a couple iso's with it and they work, but the iso size is much larger than anticipated, like it's adding empty space.

System size is right at 2 gigs, which should have gotten me around 600 mb, but it turned out an 825 mb iso instead. (zstd)

The daedalus version of same system is actually larger at 2.4 gigs, and it crunched to 717 mb. (zstd)

825 mb iso is indicative of around a 2.8-2.9 gig system uncompressed, where am I getting a 800-900 mb increase added to iso? Can't find evidence of it anywhere on the iso.

I wonder if it's something to do with the usrmerge that took place before I upgraded everything to excalibur? I have not uninstalled usrmerge in the new system.

fsmithred · 2025-01-06 21:39:38

I don't know what's going on with excalibur size. I just made a refracta excalibur i386 No-X iso with the last 686-pae kernel and it comes out to 1.5G using xz compression instead of 700MB like the daedalus iso. Looks like the excalibur iso has fewer packages installed, too.

Every release gets bigger, but this is the biggest jump I've seen.

greenjeans · 2025-01-06 21:54:02

It's kind of freaky. Something's not right with something. The size of the data on the partition i'm concerned with is not in question, it's straight-up 2 gigs, no reason why it would squash much larger, the excalibur install is unquestionably smaller.

In fact it's blowing my mind, I just pulled out a ton more of stuff that was left in after the upgrade, old versions of libraries and binaries for things. Important stuff too. And now system size down to 1.9 gigs, same system in daedalus is 2.4.

The 6.12.6-amd64 kernel is listed at 109 mb in excalibur. The previous kernel I used in daedalus is listed at 408. WTH?

Kinda at a loss here...

Seems like something is getting rolled in to squashfs that should be excluded.

Off-topic: Getting a hella delay during boot right after "waiting for /dev to be populated".

Last edited by greenjeans (2025-01-07 03:00:16)

The officially official Devuan Forum!

#1 2024-11-28 17:41:47

Is Refracta-snapshot and installer available/working in Excalibur?

#2 2024-11-28 19:13:24

Re: Is Refracta-snapshot and installer available/working in Excalibur?

#3 2024-12-01 20:03:04

Re: Is Refracta-snapshot and installer available/working in Excalibur?

#4 2024-12-01 20:53:27

Re: Is Refracta-snapshot and installer available/working in Excalibur?

#5 2025-01-02 18:06:32

Re: Is Refracta-snapshot and installer available/working in Excalibur?

#6 2025-01-02 22:00:30

Re: Is Refracta-snapshot and installer available/working in Excalibur?

#7 2025-01-02 23:47:30

Re: Is Refracta-snapshot and installer available/working in Excalibur?

#8 2025-01-03 02:55:40

Re: Is Refracta-snapshot and installer available/working in Excalibur?

#9 2025-01-04 05:03:01

Re: Is Refracta-snapshot and installer available/working in Excalibur?

#10 2025-01-04 12:48:44

Re: Is Refracta-snapshot and installer available/working in Excalibur?

#11 2025-01-04 16:42:49

Re: Is Refracta-snapshot and installer available/working in Excalibur?

#12 2025-01-06 19:07:09

Re: Is Refracta-snapshot and installer available/working in Excalibur?

#13 2025-01-06 21:39:38

Re: Is Refracta-snapshot and installer available/working in Excalibur?

#14 2025-01-06 21:54:02

Re: Is Refracta-snapshot and installer available/working in Excalibur?

Board footer