Escaping the Kiosk

Micronaut · 2022-01-06 21:19:29

This is a difficult topic to classify, so I'm not sure where to post it. As Windows becomes more and more locked down, forcing the 'users' to live with Microsoft management decisions on their 'personal' computers, I'm getting more motivated to stop using it altogether. Win 8.1 goes out of support next year, and I don't want to 'upgrade' to 10 or 11. They have clearly learned from the example of Google and Android successfully manipulating everyone into using their 'service' layered on top of access to the phone system, with all the money-extracting conditions they have imposed. MS is also in the process of converting our "personal computers" into 'computing as a utility' and billing for monthly usage. Oh and making even more money by surveillance of the user base. I'd like to ditch the MS kiosk and have control of my own system. Escaping the Google kiosk looks more difficult, since Android alternatives are scarcer and installing them on tiny little devices is trickier, but I do hope to get there also. Escaping MS seems within reach now.

There are some unresolved issues, though. And maybe a few I have not thought of.

First, e.g. -- How do people here (presuming most of you are MS-free and rely on Linux or other independent OSes) store their personal data? On your PCs? Or do you have independent storage using something like FreeNAS? All my data would have to move from NTFS, which is at least stable for all these years, to something else. Is EXT4 a good FS for storing large amounts of data? Despite the claim that Linux does not need defrag, it eventually does when you have drives filled close to capacity and are gradually adding and deleting files (as you do with personal archives). Does EXT4 have defrag? Does any *ix FS have defrag? Is some other FS considered best for long-term storage of personal data?

That's one issue that occurs to me immediately. There are probably others. Does anyone with experience moving their 'lives' from the MS Kiosk to something else have any thoughts?

rolfie · 2022-01-06 21:35:03

For private use you may set up a file server with Linux, holding the data on any file system you like. I run mine with ext4. Some specialists claim that zfs may be better.

I have Samba installed to be able to serve Windows PCs via network, the same files are available to my Linux workstations via nfs.

I have started my file server nearly 20 years ago based on Sarge, still working on Windows. 10 years ago I switched my private family network to Linux workstations, still keeping the ability to use WinXP, Win7 on older PCs and in VMs. The file server in the background still does the job, though I have migrated it to Chimaera. The Samba setup is ages old, just no more SMB1 supported.

I don't care about defrag, thats a Windows topic for me nowadays. I see no need to care ...

HTH, rolfie

Last edited by rolfie (2022-01-06 21:35:41)

steve_v · 2022-01-07 01:48:27

Micronaut wrote:

How do people here (presuming most of you are MS-free and rely on Linux or other independent OSes) store their personal data? On your PCs? Or do you have independent storage using something like FreeNAS?

On a Devuan-powered server, of course.

More specifically, on a largish (24TB usable) RAIDZ6 pool, served via NFS, SMB, and nextcloud. That machine lives in my house, where it should.
No Google, no Microsoft, no random "cloud" filestores. Includes webmail, dropbox-like filehosting & link sharing, PIM (contacts, calendar, tasks, notes) synchronisation to desktop and mobile, browser bookmark sync, password management, automated full-system backups for several other machines, the whole nine yards.

Micronaut wrote:

All my data would have to move from NTFS, which is at least stable for all these years, to something else.

It doesn't have to, but it's probably a good idea.

Micronaut wrote:

Is EXT4 a good FS for storing large amounts of data? Despite the claim that Linux does not need defrag, it eventually does when you have drives filled close to capacity and are gradually adding and deleting files (as you do with personal archives). Does EXT4 have defrag? Does any *ix FS have defrag?

There used to be a couple of defragmentation tools for ext4, but in 20+ years as a GNU/Linux user and unrepentant data-hoarder, I've never needed or wanted them.
Yes, you will get some fragmentation if you run your storage at 90+% capacity. The easiest answer is "don't do that", but even if you do the performance impact is nothing like the nightmare you get on a fragmented NTFS volume.

Micronaut wrote:

Is some other FS considered best for long-term storage of personal data?

I will plug ZFS at any opportunity. It's kinda heavy on hardware requirements (particularly memory if you want good performance), but it more than makes up for that in featureset, reliability, and data integrity. No other filesystem comes even remotely close.
If you don't want to deal with ZFS (and out-of tree kernel modules), there's nothing wrong with ext4 on top of RAID and/or a sensible backup strategy either.

Micronaut wrote:

Does anyone with experience moving their 'lives' from the MS Kiosk to something else have any thoughts?

I never really had a life in the MS ecosystem, at least not the way it is now. The last version of Windows that stored any of my personal data was '98.
To be fair, I do still have an XP install on my old netbook, but that's strictly a tool for dealing with ancient hardware and holds no personal information whatsoever.

rolfie wrote:

I run mine with ext4. Some specialists claim that zfs may be better.

Oh, but it is

rolfie wrote:

I have started my file server nearly 20 years ago based on Sarge

23-ish years here, and mine started out as a Slackware install, on a 486, in a (literal) pizza box.
Now it's an ebay-special dual xeon, shoehorned into a desktop case. New times, same old recycled hardware policy.

rolfie wrote:

I don't care about defrag, thats a Windows topic for me nowadays. I see no need to care ...

Indeed. Seconded.

Last edited by steve_v (2022-01-07 01:54:06)

PedroReina · 2022-01-07 10:17:49

Micronaut wrote:

How do people here ... store their personal data?

My main desktop has a 2TB spinning disk that holds /home; I backup it on a weekly basis, rotating four external disks plugged through SATA. All partitions are ext*. No trouble in more that 20 years; exceptions: disks eventually fail, you HAD to backup all your data beforehand.

I've two servers at home: same policy as desktop, also no problems. 20 years and counting.

Defrag? Nope.

Last edited by PedroReina (2022-01-07 12:18:32)

Camtaf · 2022-01-07 10:18:32

I dumped the 'blue screen of death' way back in 1999.

Linux or BSD will do everything a normal user requires.

There is no 'defrag' necessary, because of the way Linux handles the data using inodes.

I always keep my data on my discs, having a back up is also advisory, for any O/S.

Go for it!

zephyr · 2022-01-07 11:22:06

steve_v wrote:

There used to be a couple of defragmentation tools for ext4, but in 20+ years as a GNU/Linux user and unrepentant data-hoarder

Found this sometime back, I never knew about defragging Linux until I came across it. Have no idea if it's of any benefit but the commands are listed.

How to defrag your Linux system

There is a common misconception among GNU/Linux users that our systems never ever need to be defragmented. This stems from the success of the journalized filesystems used by most distributions including EXT2,3 and 4, JFS, ZFS, XFS, ReiserFS and BTRFS. All of these boast smart ways and techniques in regards to the files allocation in the disks, minimizing the fragmentation problem to a point that there is practically no reason to defrag even after many years of installing and uninstalling applications and libraries in the same system. Fragmentation though can still be an issue though, especially for users that use space limited disks that may not offer many file allocation options.

Here's a bulk description of how the (Linux) file allocation procedure works: files are stored in multiple places in the disk, leaving huge unwritten space between them, allowing them to grow unobstructed over time if needed. This is in contrary to filesystems like the Windows' NTFS which places files next to each other consecutively. If the disk gets more crowded and a file needs more space to grow by staying in one piece, Linux filesystems attempt to re-write it completely on another sector that has enough space to store it as a whole. This way, everything is kept clean, tidy and in one piece each. Confined space though, causes this file “maneuvering” to get more challenging with time. Here's how to deal with this problem and how to actually defrag your Linux system.

Now, the first thing that you'll need to do is get a defragment tool installed. There are many defragmenters available for Linux filesystems but I will use “e4defrag” as it is one that will most probably be already installed in your system. Using this tool, you can determine if you have fragmented files and how serious this fragmentation is. To do this, open a terminal and type: sudo e4defrag -c /location or /dev/device. Below, I have scanned my /home folder for fragmented files and actually found five of them. My fragmentation score though is quite low so defragging won't do much different in my system's performance in that case. If this outputs a score over “30”, then defragging would be a good idea.

At first, I will demonstrate how to defrag using e4defrag, by defragging my fragmented files one by one. The e4defrag program is part of the e2fsprogs package which should already be installed on your computer. In case the program is missing, install it with this command on Ubuntu:

sudo apt-get install e2fsprogs

To do this I use the following command:

sudo e4defrag

followed by the location and name of the file as shown in the screenshot below:

This is good when you have to defrag just a couple of files like I did, but if you want to defrag your whole system then you should first unmount all partitions and run the following command:

sudo e4defrag /dev/*

If you want to perform defrag without unmounting, then:

sudo e4defrag /

would be a safe choice.

Since many users nowadays use SSDs and not HDDs, it is important to note that the defragmentation procedure is only beneficial for the later. If you own an SSD, there is simply no point in worrying about fragmented files as those disks can access their storage randomly, wheres HDDs access sequentially. Defragging your SSD will only increase the read/write count and thus reduce the lifetime of your disk. SSD owners should convey their interest on the TRIM function instead, which is not covered in this tutorial.
Bill Toulas

About Bill Toulas

Over five years of experience writing about Linux and open source software on blogs and news websites. As part of the community, this is my way to give back as well as to promote what I perceive as the most amazing development in the area of software and operation systems.

view as pdf | print
Share this page:

Suggested articles
20 Comment(s)
Add comment
Name *
Email *
p
Comments
By: Curtis
Reply  

If you are a btrfs user, you can use the btrfs command to defragment.  This works on a directory level and has to be told to defrag recursively.  So if you wanted to defrag the whole system, you would do something like:

     #  btrfs filesystem defragment -r /

By: Van
Reply  

Thank you for this and nice wallpaper.

By: Davide Repetto
Reply  

Actually "sudo e4defrag /dev/*" will not work. Partitions need to be mounted for e4defrag to do its work.

By: Scott
Reply  

I just ran this on my /home partition, which was created more than three years ago when I first got this computer.  I have re-installed or upgraded the OS in a separate partition many times since.  Out of tens of thousands of files only five are fragmented.  I have never heard credible stories from any source claiming that their Linux partitions were in need of defragmenting, and this check of my /home partition just adds credibility to the claim that Linux filesystems do not need defragmenting.  It's nice that there are tools available should that ever become necessary, though.  I can understand this need if someone is dedicating their computer to something like a database and after many thousands of transactions it becomes somehow fragmented and degrades performance.  But this demonstration flies in the face of your opening claim that it is a misconception on the part of Linux users that their systems do not need to be defragmented.  In the days of MS-DOS it was common to dedicate an evening to defragmenting the FAT filesystem.  It would take hours on "large" (60+ MB) partitions to defragment them.  If only they would just stay defragmented.  The Norton Utilities were a best-seller in large part because they included a defragmenting program.  But for Linux, I see this utility as an effort in completeness rather than necessity.

By: Bill_Toulas
Reply  

There are use cases such as "extensive use of bittorrent tools" that may result in a lot of fragmented files. In most common scenarios though, yes there is no reason to defrag. That's what I wrote in the very first paragraph: "minimizing the fragmentation problem to a point that there is practically no reason to defrag even after many years of installing and uninstalling applications and libraries in the same system" which is exactly what you describe as well. :)

By: Gene
Reply  

Totally off topic. What DE are you running? Is that Gnome Shell? On top of Fedora? Looks much better than the last time I checked it out...

By: Bill_Toulas
Reply  

The DE is a GNOME-Shell fork called Panthen developed by Elementary OS developers. 

By: present_arns
Reply  

Nope that looks like ElementaryOS :)

By: tehnikpc
Reply  

Not relevant, because all servers have long been SSD.

By: till
Reply  

Most servers still use normal hard disks, especially when you have to store large amounts of data.

By: Bryan
Reply  

It should point out that defragging an SD card is important if you know the contents are less than, say, 4G and you want to copy that content to a 4G device.  For example, I have an 8G sdcard.  I know this card contains less than 4G of data.  I want to move these contents to a 4G flash part using 'dd'.   DD will fail, obviously, but at least it'll copy the contents over (only to require a little fsck fix up later, no biggy).  But, if the 8G part is not defragged data could exist at the 4G+ space and will never get copied over.

By: david
Reply  

This is horrible advice, please nobody think that using dd from a larger device to a smaller one is ever a good idea.

If you really want to do it, first shrink the partition to match the smaller device, then dd just the partition, not the entire device.

By: don
Reply  

It would have sure been nice if fonts in the picture of your screen had been large enough to have been readable!

By: Scott
Reply  

It is a common misconception, but Pantheon is not a fork of Gnome Shell.  "...notable difference between GNOME Shell and Pantheon is choice of programming language. Pantheon is written in Vala, while GNOME Shell is written in javascript (Specifically, GJS). Just to reiterate, these are totally separate unrelated code bases and are not derived from each other at all. We’re committed to using a single language across our entire desktop and apps. This allows us to share code, enforce a consistent code style, and ensure developers writing their own apps can learn from and adapt our code to new purposes."   https://medium.com/elementaryos/busting-major-myths-around-elementary-os-bd966402a9c2

By: Inukaze
Reply  

In the tittle you should pur "How to defrag your ext4 Linux partition" because, under GNU/Linux the users can select between diferents file systems like : btrfs , ext2/3/4, zfs, xfs, reiserfs, reiser4, jfs, etc . . .

And defrag for each filesystem is diferent

By: Bill
Reply  

Thanks!

By: r.l.
Reply  

Thanks for the very good job!

By: raoul
Reply  

Thanx Bill, the linux mint system has slowed down over 1,5 years and i was looking for a method of defragging which is a step in the right direction.

i used to be a unix man - sco. most stable operating system in the world.

cool - raoul

By: Joe Zeff
Reply  

Using defrag on an ext* partition is never a good idea.  Not only that, using the command "sudo e4defrag dev/*" is only going to cause trouble as it tries to defrag your keyboard, your modem, your monitor...

By: Barnaby Jonez
Reply  

Thank you.  This is great!  Mu home/ partition that is nearly full and it was heavily fragmented.  The defrag took awhile and it made things faster.  And the I did a few more passes and it degraged more.  The content on that partion is moving much smoother now!

Home
How to defrag your Linux system
>
Sign up now!

Tutorial Info

Author:
    Bill Toulas 

Tags:
    arch linux, debian, fedora, linux, ubuntu 

Share This Page

40.2k Followers

Popular Tutorials

    Securing Your Server With A Host-based Intrusion Detection System
    How to Compile A Kernel - The Debian Way
    ISPConfig Perfect Multiserver setup on Ubuntu 20.04 and Debian 10
    Install Varnish Cache 6 for Apache/Nginx on CentOS 8
    How to Install Elastic Stack (Elasticsearch, Logstash and Kibana) on CentOS 8
    How to use grep to search for strings in files on the shell
    How to Install Rust Programming Language on Ubuntu 20.04
    How to Install WordPress with Nginx on AlmaLinux 8
    A Beginner's Guide To LVM
    Installing a Web, Email & MySQL Database Cluster on Debian 8.4 Jessie with ISPConfig 3.1

Xenforo skin by Xenfocus

    Contribute
    Contact
    Help
    Imprint and Legal Notice
    Top
    RSS-Feed

    Terms and Rules Privacy Policy 

Howtoforge © projektfarm GmbH.

Hope this helps!

cheers

zephyr

Head_on_a_Stick · 2022-01-07 13:23:17

Micronaut wrote:

Is EXT4 a good FS for storing large amounts of data?

I don't have the expertise to answer that directly but XFS was expressly designed for large storage and can deal with larger devices than ext4. Red Hat (RHEL) defaults to XFS as their filesystem.

Micronaut wrote:

Does EXT4 have defrag?

See e4defrag(8) (as noted by zephyr).

Micronaut wrote:

Does any *ix FS have defrag?

https://wiki.archlinux.org/title/btrfs#Defragmentation

Micronaut wrote:

Does anyone with experience moving their 'lives' from the MS Kiosk to something else have any thoughts?

Steam Play & Proton has let me dump MS completely now but I only use my laptop for recreational purposes.

steve_v wrote:

rolfie wrote:
I run mine with ext4. Some specialists claim that zfs may be better.
Oh, but it is

Only if you don't mind the ridiculous memory usage. Have you tried it with de-duplication enabled?

steve_v · 2022-01-07 14:20:48

Head_on_a_Stick wrote:

Only if you don't mind the ridiculous memory usage. Have you tried it with de-duplication enabled?

That's why there are so many warnings WRT the storage saving vs memory usage trade-off of dedup. In my case that trade isn't worth it, so I don't enable it.
IIRC the 64GB of RAM in my machine cost me ~120USD, and frankly I only installed 64 because I could. For a reasonable home-server pool size, ZFS (without dedup) will run just fine with 4.
More is better of course, as with most things. ZFS will use whatever it can get for its adaptive cache.

Of bigger concern hardware-wise is ECC and a UPS, though if you're serious about data integrity you'll have those already, regardless of filesystem choice.

Micronaut · 2022-01-08 17:57:56

steve_v wrote:

There used to be a couple of defragmentation tools for ext4, but in 20+ years as a GNU/Linux user and unrepentant data-hoarder, I've never needed or wanted them.
Yes, you will get some fragmentation if you run your storage at 90+% capacity. The easiest answer is "don't do that", but even if you do the performance impact is nothing like the nightmare you get on a fragmented NTFS volume.

Granted, it's not a good idea to run very close to capacity on any FS for a variety of reasons, but I think defrag is more necessary than most of the *ix purists like to admit. Keeping the system partitions with actual programs separate from the mass data storage partitions helps, too.

steve_v wrote:

I will plug ZFS at any opportunity. It's kinda heavy on hardware requirements (particularly memory if you want good performance), but it more than makes up for that in featureset, reliability, and data integrity. No other filesystem comes even remotely close.
If you don't want to deal with ZFS (and out-of tree kernel modules), there's nothing wrong with ext4 on top of RAID and/or a sensible backup strategy either.

Yeah, that's supposed to be the really outstanding FS for protecting data. MS is currently struggling to duplicate it with their "Resilient File System" (ReFS), but still recommends even servers use NTFS most of the time, I believe. Since ZFS is *BSD native, there is some sort of a license conflict and it's not easily available in Linux. I would probably have to use a dedicated NAS hardware device with one of those *BSD OSes to use that.

I've been in the habit of keeping multiple backups for years anyway, even with NTFS. That would certainly not change if I moved to Linux, but I might have to occasionally delete and re-copy the entire contents of a volume to defrag it. Seems likely to cause excessive wear.

Micronaut · 2022-01-08 18:03:51

Head_on_a_Stick wrote:

Steam Play & Proton has let me dump MS completely now but I only use my laptop for recreational purposes.

Yeah, I hadn't got to that part yet, just talking about routine usage like word-processing, email, and web browsing, but there is a game component to the migration away from MS. I've got some MMO habits and don't want to give them up. In a worst case scenario, I might have to keep a Win 8.1 machine only for gaming and never use it for web browsing once the security support ends. But eventually games will stop supporting 8.1 and then it's get things to run in Linux or keep a special machine running 10/11/whatever just for gaming.

Proton is supposed to be good, but can you use it without signing up for Steam? I'm thinking Play on Linux or Crossover if Proton is tightly bound to a Steam subscription.

Head_on_a_Stick · 2022-01-08 18:17:08

Micronaut wrote:

Proton is supposed to be good, but can you use it without signing up for Steam?

A quick search suggests perhaps. But I've never tried it.

steve_v · 2022-01-08 20:22:29

Micronaut wrote:

I think defrag is more necessary than most of the *ix purists like to admit.

You're entitled to think whatever you want, of course.
My opinion on defragging ext* remains unshaken all the same though, and it's nothing to do with being a "purist". Ext* simply doesn't have the same rampant fragmentation issues that FAT and NTFS do.

Micronaut wrote:

MS is currently struggling to duplicate it with their "Resilient File System" (ReFS), but still recommends even servers use NTFS most of the time, I believe.

At this point, I'm really starting to suspect ReFS is vaporware.

Micronaut wrote:

Since ZFS is *BSD native, there is some sort of a license conflict and it's not easily available in Linux.

Solaris native, actually. But it does have a very nice BSD port.

Sun Microsystems released the ZFS source under their own take on a FOSS licence (the CDDL) shortly before they were devoured by Oracle, and actively developed ports are available for most BSD variants, MacOS and Linux.
Unfortunately It's not entirely clear whether the CDDL is compatible with the GPL, and there have been claims both ways. The FSF says it isn't, while the SFLC and Ubuntu claim that it is... As such, most GNU/Linux distros don't ship ZFS out of the box, requiring users to install it later.

For personal use the licence shenanigans is largely irrelevant, and full support on debian-based distros is just an 'apt install zfs-dkms zfsutils' away. You can even boot a GNU/Linux system from a ZFS pool, though I can't comment on Devuan's support for that specifically.

Micronaut wrote:

I might have to occasionally delete and re-copy the entire contents of a volume to defrag it. Seems likely to cause excessive wear.

Uhh, defragging a volume causes "excessive" wear as well... But hey, if you really want to do that kind of thing, go nuts.

Micronaut wrote:

Proton is supposed to be good, but can you use it without signing up for Steam?

Yes and no, and it's a complete pain in the ass.
Fortunately, TK-Glitch maintains a wine build-system with features backported from proton, and compatibility is usually just as good as the real thing.
AFAIK there are no convenient binaries for debian-based distros (TK-Glitch is an Arch user BTW), but if you don't want to compile stuff yourself you can get also get them through Lutris. This is probably the easiest, most compatible option outside of installing Steam, and it's in the Devuan repos...
Or just swipe the Lutris wine tarballs for use with your frontend of choice. I get the distinct impression the Lutris devs don't like people doing this (an attitude that pisses me off no end), but they aren't particularly well hidden.

Aside, if you want optimal gaming performance with Wine, you'll probably want a kernel with the fsync and/or futex2 patchset. These can also be found on TK-Glitch's github, and IIRC the Xanmod and Liquorix kernels include them if you don't want to patch/build by hand.
I can't comment on how well those kernels behave with Devuan though, since the only machine I actually game on runs Gentoo.

IME the biggest problem with gaming on GNU/Linux isn't compatibility or performance any more (I get framerates within 1-2% of native), it's the rise of intrusive kernel-level anti-cheat / DRM.
Unfortunately there's no easy fix for that, so if your MMO habits include games that install ring-0 malware for "security" purposes or auto-ban wine users for "cheating", you'll probably want to keep a windows partition around...
Or better yet, vote with your wallet and just refuse to buy games that spy on you or try to hijack your system.

Last edited by steve_v (2022-01-08 21:16:28)

The officially official Devuan Forum!

#1 2022-01-06 21:19:29

Escaping the Kiosk

#2 2022-01-06 21:35:03

Re: Escaping the Kiosk

#3 2022-01-07 01:48:27

Re: Escaping the Kiosk

#4 2022-01-07 10:17:49

Re: Escaping the Kiosk

#5 2022-01-07 10:18:32

Re: Escaping the Kiosk

#6 2022-01-07 11:22:06

Re: Escaping the Kiosk

#7 2022-01-07 13:23:17

Re: Escaping the Kiosk

#8 2022-01-07 14:20:48

Re: Escaping the Kiosk

#9 2022-01-08 17:57:56

Re: Escaping the Kiosk

#10 2022-01-08 18:03:51

Re: Escaping the Kiosk

#11 2022-01-08 18:17:08

Re: Escaping the Kiosk

#12 2022-01-08 20:22:29

Re: Escaping the Kiosk

Board footer