We're migrating our distributed network to Devuan

thezeit · 2018-09-04 14:33:43

Let me see if I can summarize succinctly. This may read more like bullet points.

I'm the sysadmin for a small network of globally distributed computers. The network is always slowly ticking up in size.

The core function of the network is to support, load, provision, and heal content on Freenet.

At this stage about 50% of the network has been converted from Debian to Devuan.

Security of the computers on the network is essential since one of the primary objectives of the project is censorship resistance.

I've had some concerns about the encroachment of systemd for awhile and was keeping an eye on Devuan. A number of things started coming up in Debian Stretch. And not only in the move from Jessie to Stretch but also between Stretch versions.

Some examples:

* Umounting encrypted drives suddenly became intermittent. Something not reported in lsof sometimes has a hold on the mount point. Worked fine in jessie and now my automatic provisioners trip intermittently.

* Installer scripts stopped working correctly between stretch versions. Systemd took over the DNS config and would replace or truncate the settings on reboot. I didn't take the time to figure out which systemd changed expressly caused it.

* Freenet arrays started running rough. Logs were spitting out "could not spawn thread" errors. After 2 days of hunting and scouring the internet I finally found reference to the "DefaultTasksMax" setting in the "/etc/systemd/system.conf" file. This setting appeared between two systemd versions and didn't adjust thread/task limits at the kernel. So all queries to the kernel reported we had more than enough resources and I didn't see any flaws in the Freenet code either.

I have an up & coming sysadmin picking up skills from me. He already knew about systemd tripping us before but once he found out about how it's encroaching on the D-Bus and firewall... he demanded we switch to Devuan immediately.

I conceded and started making the switch.

My experience so far is: Devuan works brilliantly.

We migrated a number of boxes in place and performance noticeably improved.

I've begun to hassle server farms to provide a Devuan distro. I anticipate many of them will refuse and I'll need to perform the migration myself after they provide a Debian image. I continue to hassle them anyway. I did actually have a server company meet me half way. They booted Devuan off a USB stick and allowed me to install it remotely.

I deliberately removed my personal tirades from this message to conclude with this: No more invisible walls.

Centurion Dan · 2018-09-04 20:05:40

What server farms in particular?

I'm working towards having a mechanism to produce cloud-init based images for various cloud and vps providers as a part to of our release processes. Be nice to gather information on the various platforms that it would be good to support.

golinux · 2018-09-04 22:50:12

Welcome to Devuan. I predict that we will be hearing more and more stories like yours in the future. Would you consider dropping a modified version of this over at Distrowatch? Might inspire someone else to take the leap.

Your personal tirades might provide some always welcome amusement . . .

thezeit · 2018-09-05 12:04:57

What server farms in particular?
I'm working towards having a mechanism to produce cloud-init based images for various cloud and vps providers as a part to of our release processes. Be nice to gather information on the various platforms that it would be good to support.

I'm biting my tongue hard right now. I would really like to give you intel on that. The nature of our "censorship resistance" project has us facing the very real risk of service termination due to ambiguous ToS agreements though. Most agreements translate in plain English to, "We'll terminate you if you cost us too much."

Would you consider dropping a modified version of this over at Distrowatch? Might inspire someone else to take the leap.

Done! Do they manually review the posts?

It brought me back to the same page with the settings cleared and no visible post. I'll follow up if they don't post the review.

Something I should probably emphasize to people is the ease of the migration. I was worried the network settings were going to totally implode but it was very smooth and painless. You just have to remember that the interface name will change from something random-looking to "eth0". Yes, I left the old interface name in the first time. Yes, the new sysadmin teased me and told me to RTFM.

Your personal tirades might provide some always welcome amusement . . .

I was a little tiny bit absolutely livid.

Something that's driving me up the wall is all of the new systemd background services that are active with each new Debian/Raspbian release.

A lot of project contributors are interested in using raspberry pis or the new models of libres (they look like Pis) to help power Freenet. And since Raspbian tracks with Debian many of those systemd background services are littered all over the project contributor nodes. Magic cancer dust.

I have two things I'll tack on here.

Devuan Youtube Exposure

I have a YT channel with a few thousand subscribers. I'm part of a wider net of men that are facing deplatforming headwind and what might best be described as "financial harassment" (e.g. people will donate runs of small amounts and then issue chargebacks through the banks in an effort to bury the authors financially).

The side effect of both of those things is it's forcing authors to get off the Microsoft/Apple bandwagon and many of them are turning to cryptocurrencies. The side effect of that is many of them are just now starting to learn about Linux (Ubuntu/Debian usually) and they're just now starting to learn about DNS hijacking, Tor, and Cloudflare.

I anticipate many of them aren't going to know what systemd is. So while they're in this process of learning/transitioning I have an opportunity to stick my foot in the door here and encourage them to get ahead of the curve all at once rather than staying persistently behind it. My popularity at YT seems to go up whenever the risks (lack of foresight?) decide to manifest. So I'm planning on holding rank with funneling people toward Devuan as they get incrementally "woke".

Tails OS

I'm a big fan of Tails OS. I'm not a big fan of it hooking itself up to GNOME/systemd.

Their FAQ says they won't consider leaving GNOME and their helpdesk shows they're moving to leverage "systemd security features".

Moving forward I might take their principle concepts (kernel settings, mac rotation/spoofing with each new connection, hardened firewall that rebuilds itself, Tor browser, app armor profiles, onion circuits, onion share, etc...) and apply them to a custom build of Devuan. The way I normally handle this is a fresh LXC container that takes an existing ISO, straps in the goodies, and then bakes out a hybrid bootable ISO (i.e. thumb drive or DVD). Usually the whole process is just one or two scripts and can even be done from a live boot (provided enough ram and/or swap).

I'm wondering if the Devuan community would be interested in something like that. Normally I just use it for myself internally... but with Tails OS seemingly marrying itself to the GNOME/systemd model... I could see it as a useful building block. I probably don't have the time to "be my own distro". But putting out the scripts I use might help jumpstart hardened live-boots of Devuan.

Feel free to let me know if anyone's already ported Tails to Devuan and I'll just shut up and support their work instead.

golinux · 2018-09-05 14:46:45

In a bit of a rush now so just this . . .

thezeit wrote:

Feel free to let me know if anyone's already ported Tails to Devuan and I'll just shut up and support their work instead.

We have had "heads" available for years!

chris2be8 · 2018-09-05 18:11:22

thezeit wrote:

* Umounting encrypted drives suddenly became intermittent. Something not reported in lsof sometimes has a hold on the mount point. Worked fine in jessie and now my automatic provisioners trip intermittently.

That could be something accessing the mount point for short intervals. So it's gone when you look for it with lsof. But finding out what it is could be difficult.

You might be able to get round that with a script that retries the unmount a few times if it fails. But that should not be necessary.

Chris

thezeit · 2018-09-05 19:10:28

We have had "heads" available for years!

A grsec-enabled, non-systemd, Tails analogue...

Man crush developing...

You might be able to get round that with a script that retries the unmount a few times if it fails. But that should not be necessary.

Tried that. Indefinite umount looping hangs indefinitely. Delayed/deferred and forced umount also fail.

The only way to free the mounts points ultimately was to restart the system. Granted this was always on a graphical system (running GNOME) running the auto scripts. So it's conceivable that GNOME might be jamming something too.

The officially official Devuan Forum!

#1 2018-09-04 14:33:43

We're migrating our distributed network to Devuan

#2 2018-09-04 20:05:40

Re: We're migrating our distributed network to Devuan

#3 2018-09-04 22:50:12

Re: We're migrating our distributed network to Devuan

#4 2018-09-05 12:04:57

Re: We're migrating our distributed network to Devuan

#5 2018-09-05 14:46:45

Re: We're migrating our distributed network to Devuan

#6 2018-09-05 18:11:22

Re: We're migrating our distributed network to Devuan

#7 2018-09-05 19:10:28

Re: We're migrating our distributed network to Devuan

Board footer