Took me a while to get around to trying to do this, as i was not sure of how much of a hack the whole root on zfs install would be. But minus the odd install part of the system(due to some license  incompatibility) Which is on par with doing root on NFS which is then managed trough zfs snapshots and cloning on server. No custom scrips, patches, or even extensive modifications to any part of system will be used in this setup .

Here is my starting point References for ROOT on ZOL first one is what i hoped to achieve.

• http://www.thecrosseroads.net/2016/02/b … on-debian/
  https://openzfs.github.io/openzfs-docs/ … 20ZFS.html
  https://www.funtoo.org/ZFS_as_Root_Filesystem
  https://www.combustible.me/blog/linux-m … -swap.html
  https://wiki.archlinux.org/index.php/In … bootloader

I wanted my rpool on whole disk and not how most  howtos split up in to bpool and rpool (my understanding is, it is due to limitations in grub and some zpool features but if we get it running and not enable them later, all is well ?) This is most i could find at the moment  but a good lead. https://unix.stackexchange.com/question … b-can-read

Aditional info on BSD and Indiana multibooting

• https://blog.karpador.xyz/articles/one- … -them-all/
  https://ericmccorkleblog.wordpress.com/ … fs-volume/

Starting with devuan Live image was discarded. It is not persistent, you will have to reinstall and reconfigure zfs and other things to get your system back up  if something goes wrong. I chose a hard drive installation for a rescue  system that is ready to go. Can also boot systems in the pool from outside if needed. Other options can be a usb stick, msata drive, anything that devuan can be installed on and booted.

Interesting option  could be an ssd drive that could serve multiple functions.

• 1 Rescue System
  2 Swap on ssd for hibernation which is not support on zfs zvol yet
  3 And a persistent l2arc on same ssd when hibernation is used

But that is a bit of a strech for data integrity ideal of zfs. (Moving data, here to swap, outside of zfs control)

To get started

I am doing net install of beowulf,

• 300 meg boot partition

• 10Gig root

• 5Gig swap

Keep it small and manageable. We will use this as our maintenance/rescue system and as a starting clone for first system on zfs root.

Grub-legacy booting only for now

• no need for X or desktops etc...

• minimal advanced install  only selecting "standard system utilities".

Some nice things to help out , but not essential

• To help with copy paste the instructions from another machine on networked system might be a good idea at this point as it will save time later once we get other clones going. moving in and out of different installs will be much faster. (and for some reason without --no-install-recommends it tries to pull in things from icon themes to x11-common , basicaly half the install of X without the X)

  apt-get install --no-install-recommends openssh-server  

  And on a neworked machine that will access this installation, (configure autologin for ssh(and a desktop launcher to make it really easy))
  replace username@machineIP to match your install.

  ssh-keygen -t rsa
  ssh-copy-id user@10.10.50.x

• to help copy and paste inside the console if needed

  apt-get install gpm

• if needed

  apt-get install nfs-common

Once the system is up and configured to your liking.

• Install headers for your kernel. (apt get int the next step seems to pull in wrong ones, so do it manually )
  then install 0.7.12 zfs  if you plan on staying with 4.X kernel.

  apt-get install zfs-dkms zfsutils-linux

• or keep going and install 0.8.4 version of zfs, uncomment or add beowulf-backports to /etc/apt/sources.list
  and make sure contrib is also there and install.

  apt-get install -t beowulf-backports zfs-dkms zfsutils-linux

  and install 5.X kernel and headers ,(if done before installing backports version of zfs i get install errors of it trying to compile old version of zfs-dkms to 5.x kernel.)

Going from .7 zfs to .8  is a big step that is worth it in features and functionality. Lots of things started working in .8
that did not before, like hotspare drive functionality started working in .8 . I tested it by unplugging drives or
dd if=/dev/urandom of=/dev/sdX garbage to a drive in zpool.

But you do not have to upgrade the pool version itself while upgrading to .8 from .7 unless you need the extra flags
or functions inside the pool , without upgrading the pool version you can keep it backwards compatable. to older linux
kernels or BSD and even Solaris/Indiana (citations ?)

• Now , clean up, add your favorite aliases in .bashrc etc.. Maybe delete apt deb caches to make it even smaller.
  Every iteration of clones from this point will start adding up and then throw in snapshots on top of that and you could
  double or triple the sizes if no cleanings are done.

If anything from this point on is not clear here are great ZFS specific resources 

• https://pthree.org/2012/04/17/install-z … -gnulinux/
  https://wiki.archlinux.org/index.php/ZFS
  https://wiki.debian.org/ZFS
  https://docs.oracle.com/cd/E36784_01/ht … cinfo.html
  https://www.freebsd.org/doc/handbook/zfs.html

Create a pool

I am not setting altroot, the non persisten / mountpoint. As we should start using tempmounts for zfs
from the get go, its what i will use later for cloning and managing. Its does the same thing but with  tempmount you
know exactly what you mounted and where.

ls -al /dev/disk/by-id/ 

and

lsblk

to make sure not to grab the system disk by mistake

pick your disks , choose raidz1, 2, 3, mirror, stripe , mirror/stripe,
or a mirror of raidz3 stripes and -o copies=5 of data ? for special occasions !

Setting  ashift=12 is highly recommended  during the creation(please investigate for yourself)

zpool create rpool mirror \
ata-WDC_WD1600AAJS-75M0A0_WD-WMAV31415162 \
ata-WDC_WD1600AAJS-75M0A0_WD-WMAV31415738 \
mirror \
ata-WDC_WD1600AAJS-75M0A0_WD-WMAV31432691 \
ata-WDC_WD1600AAJS-75M0A0_WD-WMAV31376665

Set some basics that we want to propagate to child datasets (unless set localy), this is mostly  options and depends on use case ( tuning zfs ) most things can be set later, while others  will only take effect for new files unless you recopy files or simpler to send and recieve dataset with new options. Like recompress  to lz4 from gzip etc...Some options can only be set once during pool creation like ashift, or casesensitivity  on datasets for smb/cifs shares.

can embed this during pool creation but i keep it separate.

zfs set mountpoint=none rpool

Also this can be moved down to rpool/ROOT As i will have other datasets under rpool and probably do not want them inheriting these options

zfs set atime=off rpool
zfs set relatime=on rpool
zfs set compression=lz4 rpool

info on proper zvol swap use https://openzfs.github.io/openzfs-docs/ … wap-device

zfs create -V 10G -b $(getconf PAGESIZE) \
    -o logbias=throughput -o sync=always \
    -o primarycache=metadata -o compression=off\
     rpool/swap

mkswap -L swap /dev/zvol/rpool/swap

Start making datasets

This sets up the dataset tree that will make managing this easier (i guess format is from solaris).

zfs create -o mountpoint=none rpool/ROOT
zfs create -o mountpoint=/ rpool/ROOT/debian1

Not very clear on this one yet, (but lets use it, till we know better)

zpool set bootfs=rpool/ROOT/debian1 rpool

Now need to make a mountpoint for our first system. Usualy zfs does not need mountpoints, It creates them if none exist, and refuses to mount on mountpoints with files unless overridden , but when using tempmount or legacy, it will balk at not having one.

mkdir /mnt/debian1
mount -t zfs -o zfsutil rpool/ROOT/debian1 /mnt/debian1

to make sure rpool/ROOT/debian1 is indeed mounted

df -h 
zfs get all rpool/ROOT/debian1 | grep mount  

Cloning the system in to the zpool

now rpool is ready to accept our first system

apt-get install rsync

Since this is a new system we are not excluding more things that need to be on other systems for cloning like /media etc.... could take out srv too  and mnt since we are staying on one filesystem but for safetys sake.... lets not loop it ( and if you do not have a separate /boot partition , disreguard the second  command)

rsync -aAHXx / --exclude={"/dev/*","/proc/*","/sys/*","/run/*","/mnt/*","/srv/*"} /mnt/debian1/
rsync -aAHXx /boot/* /mnt/debian1/boot/

Now chroot to get in to the system and do a few tasks

mount --rbind /dev /mnt/debian1/dev
mount --rbind /proc /mnt/debian1/proc
mount --rbind /sys /mnt/debian1/sys
mount --rbind /run /mnt/debian1/run

chroot /mnt/debian1 /bin/bash --login

A config file/db of sorts, if does not exist yet make.
Some say use it , some say go without, as all info is within
the pool drives anyways (need to clear up)

mkdir -p /etc/zfs
zpool set cachefile=/etc/zfs/zpool.cache rpool

Will need this step to make sure zfs can mount / right after grub, in managment system we are not runing root on zfs so it can mount the pool later during boot by loading a kernel modue. But here we will need initramfs to do that.

apt-get install -t beowulf-backports zfs-initramfs

Test if grub sees that its zfs ?

grub-probe /boot

• this will write updates to /boot/grub/grub.cfg which will be called by grub chain starting from mbr.
  At the moment it would still boot the initial system unless we do this step.

  update-grub

• Using cfdisk change 9th partition  on each zfs pool drive to BIOS boot. Which zfs made during pool creation,small 8 meg patition on my system, (need citation for this) but its seems to be for uefi  boot compatibility. And if it was in active part of zfs it would have balked at me using during first zpool scrub or on imports.Here is some info https://www.reddit.com/r/zfs/comments/g … ved_space/
  (or bios_grub flag in gparted)

if previous step is incorrect or not done , this will happend on grub mbr install, replace X with drive(s) in the pool

if

grub-install /dev/sdX 

goes without error , repeat for each drive in the pool this way if one drive fails in the pool it can alway boot from another provided they are set sequentially in bios for boot drive order .

edit /etc/fstab
coment out everything about old filesystems as zfs will handle that for now,
but we will return here if we start using legasy mountpoints. And add
/dev/zvol/rpool/swap    none    swap    sw    0    0

edit /etc/initramfs-tools/conf.d/resume
as zfs does not support  hibernation on zvol yet and will hang if you leave old drive there
RESUME=none
(and i wonder if this can be separate to from zvol swap, but not tested yet)

Update initramfs to get  resume update to none.

update-initramfs -c -k all  

and

exit

from chroot

At this point reboot and change the bios boot drive to one of the drives from the pool, and if everything went to plan, the system should be up and running

df -h  to see that the root is mounted from rpool/ROOT/systemXXX that you expect. Do that a few times on every new system you clone, as mistakes in fstab or forgeting update-grub will boot in to a clone source system.

some zfs errors about it not being unable to mount / are ok as it is normal when initrd already mounted it, like on NFS root boot i get same errors.

If booted and everything ok, do

zfs snapshot rpool/ROOT/debian1@initial

do not rush on installing anything here yet as this system is another maintenance system , only inside zpool now.
and a new initial clone source for next steps, plus chain loading  all other systems will come from this systems grub.

New to the board, wanted to post a howto or two.

Just got zfs on root going starting with beowulf and through to ceres
on whole drive moror/stripe pool, 6 or so clones with various configs.
snapshots, clones, shared datasets etc..
Maybe some input on things that need ironing out or i missed.

And just wanted to see if i can edit my posts later or do i need to prepare
the whole thing as a single shot.

move this post if necessary , just for testing purposes