Update : PeerTube 2.0.0 on OpenBSD

I wrote this article after migrating my PeerTube instance from FreeBSD to OpenBSD. Doing so, I hope to inspire other to jump in there. I believe to be the first and only one to have done that yet (unable to check that though).

The PeerTube release used is 1.4.1 due to newer releases bugging at install.

But I wanted to do it. I wanted to setup PeerTube on OpenBSD and write this article and now I have done it. So we can do it. And we can spot and fix bugs on newer releases.

It is also my hope that we can start to work on improving the security of PeerTube using an OpenBSD system and mindset. I hope that some will comment the article, saying me where I did a mistake, or how I could have done better.

We can run PeerTube on OpenBSD !

The start

I had my PeerTube instance running on FreeBSD hosted on a VPS at ARN, which is a french non-profit community-driven ISP. I had just tested and succeeded in installing and running an instance on my home server. So now I can do the same in production.

As it's the occasion, I wish to thank again ARN and its members, especially those who helped me personly during this adventure.

First mist… step

I check with ARN that it goes fine, then we find a a day where we can work together.

Valentin, ARN's chief manager, who was the one I had to work with tells me he is going to setup an SSD aside my current VM, I should just install OpenBSD on it, mount my previous FreeBSD partition in it and that's it.

But actually, no.

It was though a pretty optimization try. Except no.

The VPS

The VPS given by ARN is a 10 G SSD and a 200 G HDD. Obviously most of it is purely abstract to me, but it's how the OS sees that.

I divided the SSD this way (actually I just used the sizes given by the automatic allocation process) :

#                size        fstype
a:             1.2G          4.2BSD    # /
b:             0.2G          swap      # none
d:             3.0G          4.2BSD    # /usr
e:             5.6G          4.2BSD    # /var

I hope not to have made mistakes with that, but actually, you cannot know before. I am the first one to build a PeerTube instance on OpenBSD, so I guess I have a few pots and eggs to break.

As I write those lines, /usr still has roughly 500 M free space and should stay this way. /var is bigger but will host the database and PeerTube itself which is quite big.

The HDD will be the /home partition. I shall use it mainly to host videos files but also as personal backup.

First OpenBSD installation

I won't go into details over the OpenBSD installation process nor on the weird network configuration they use.

It's a bit hard, but well described in their documentation. You just need a full iso as you won't have any network access during the installation process. The install itself goes fine. A well-known system that I know so good. Indeed, what to ask more ?

The machine shall receive the sweet name (hostname) of dina. Dina is the kitten in Alice in Wonderland. The kitten which stays out Wonderland. Because all my domain hosts inside my home network are named after Wonderland's characters.

I set the former HDD as an OpenBSD partition in disklabel but I don't do anything more.

At reboot… Crash. The thing needed to understand what is going on. So five attempts. Each time something else went wrong.

Finally I could make the thing boot up correctly. Just to figure out the thing cannot mount the HDD.

It's this way I understood despite their common name and origins, FreeBSD and OpenBSD's filesystems are not at all the same anymore.

Second try

So let's try on a full VPS. The configuration will be brand new. A SSD + HDD. Same as before, but a brand new IP too. All good.

Now I fire up a live FreeBSD, fsck the former disk and mount it to access data and transfer them on the new VPS.

Once again, no.

The two VPS could not communicate. No ping between them. I suspect the bug comes from the fact that the FreeBSD was live, which meant I could not fire up sshd or anything else.

And I could not boot FreeBSD in its former state.

Third try

Once again, let's setup FreeBSD. But just in a hurry to do the job and it's done.

Valentin mount a 10 G SSD to host a temporary FreeBSD system. I install FreeBSD in a hurry and mount the former disk in /mnt. It is just a oneshot anyway.

Then I manage to make the two VPS to communicate. But only on ipv6. Hurrah for ipv6 !

The dumb thing is that I cannot create an NFS share, which would have been so fast and powerful. But OpenBSD does not do NFS over ipv6.

PeerTube installation

From now on, we have a stable system on both virtual machines.

I can actually install PeerTube on OpenBSD. First, I create a dedicated system user, _peertube, with /var/www/peertube as home repository. I also have a section in login.conf :

peertube:\
    :openfiles=1024:\
    :tc=daemon:

The transfert

To move gigabits of videos data between the two VPS, I am simply going to launch sshd on FreeBSD and rsync from the OpenBSD VM.

But … take care ! I control both VM via ssh from a third machine (my desktop computer, at home). I have to make sure operations don't go crazy if and when the ssh connection from my desktop gets broken.

So let's connect to the OpenBSD VM, launch tmux and from within, rsync. It looked like that :

tmux
cd /home/peertube
doas -u _peertube openrsync -e ssh -6 root@rescue:/var/www/peertube/storage/ .
  1. doas -u _peertube : Let's work as _peertube so that files belong to him from now on.
  2. openrsync : I use OpenBSD native rsync. So that if transfer crash, I just have to relaunch it and rsync will make sure no file is missing and that it's not working twice.
  3. ssh -6 : connexion via ssh with ipv6 rather than ipv4, as it's the one true way of communicating between them.
  4. root@rescue : I wrote the original FreeBSD VPS's ipv6 address as rescue in /etc/hosts on dina, and I connect there as root so I ca get all files from there. Anyway, they will belong to _peertube like written before.
  5. in /home/peertube : files coming in directly in their final place.

The /home partition

I wanted to truly integrate PeerTube in the operating system. So my plan was to do this :

/etc/peertube/...
/home/peertube/...
/var/www/peertube/peertube-latest
                /versions/$VERSION

But actually, it's not possible. Because nginx, the webserver, will do a big part of the work and it is chrooted. As a result, it will not be able to send the video files outside /var nor to update its configuration in /etc.

It is therefore necessary to return to the documentation's original instructions, which requires to install PeerTube as described below. Peertube-latest is just a link to the version we will actually be launching :

/var/www/PeerTube/storage
                /config
                /peertube-latest
                /versions/$VERSION

What to do for storage, ie video files, subtitles, images, etc? For now, they are in /home/peertube. Should not I dedicate this partition entirely to PeerTube storage?

No actually I really want to use some of this space for other personal reasons (backup space), probably with a /home/backup. So I will keep things like this. The ideal at this time would be to be able to bind mount the directory /home/peertube in /var/www/peertube/storage.

Except that OpenBSD does not have this mount option.

So, in fact, I exported the /home/peertube directory via NFS and mounted it locally in /var/www/peertube/storage.

/etc/exports

/home/peertube   -maproot=root   -alldirs localhost

/etc/fstab

# SSD
32d7755f555b0d5d.b none swap sw
32d7755f555b0d5d.a / ffs rw 1 1
32d7755f555b0d5d.d /usr ffs rw,wxallowed,nodev 1 2
32d7755f555b0d5d.e /var ffs rw,nodev,nosuid 1 2

# HDD
ffec183db71141e2.a /home ffs rw,nodev,nosuid 1 2

swap            /tmp mfs rw,nosuid,nodev,-s=250M 0 0

127.0.0.1:/home/peertube /var/www/peertube/storage   nfs rw,nodev,nosuid,noexec 0 0

/etc/rc.conf.local

check_quotas=NO
portmap_flags=
mountd_flags=
nfsd_flags="-tun 4" 

Notice that a logical link (ln -s) in nginx’ chroot would not be better. Logical links are just software shortcuts between two points of the filesystem. And from nginx’ point of view, there is no /home/peertube on which the link points out.

Dependencies

PeerTube's dependencies are well documented :

doas pkg_add ffmpeg redis postgresql-server postgresql-contrib node nginx

Notice that I needed python3, eventhough I cannot read it back now in the current installation manual :

doas pkg_add python3

Don't forget to add that in rc.conf.local:

pkg_scripts=postgresql redis peertube nginx

PostgreSQL

You have to initialize PostgreSQL correctly and setup the contrib.

Do not just copy-paste, take your time to understand what I did. Read a few texts or documentations about PostgreSQL.

# su - _postgresql
$ mkdir /var/postgresql/data
$ initdb -D /var/postgresql/data -U postgres -A scram-sha-256 -E UTF8 -W
$ createuser -P peertube
$ createdb -O peertube peertube_prod
$ psql -c "CREATE EXTENSION pg_trgm;" peertube_prod
$ psql -c "CREATE EXTENSION unaccent;" peertube_prod

Just take five minutes to read the doc from the port and check if you need something more or to ajust kernel parameters or whatever…

less /usr/local/share/doc/pkg-readmes/postgresql-server

libvips

There is also libvips as a dependency and I wrote a port to install it properly. I proposed it for integration in the ports tree, but have not received a final answer.

The port is to be installed in /usr/ports/multimedia/.

# $OpenBSD: Makefile,v 1.84 2019/08/08 23:04:22 kmos Exp $

COMMENT =       fast image processing library with low memory needs

VERSION = 8.8.3
DISTNAME =      vips-${VERSION}
CATEGORIES =    multimedia

SHARED_LIBS +=  vips                 0.0   # 53.1
SHARED_LIBS +=  vips-cpp             0.0    # 53.1

HOMEPAGE = https://libvips.github.io/libvips/

#MAINTAINER =           ???

# LGPLv2+
PERMIT_PACKAGE =        Yes

WANTLIB += ${COMPILER_LIBCXX} c expat fontconfig freetype m z
WANTLIB += ffi glib-2.0 gmodule-2.0 gobject-2.0 gthread-2.0 iconv
WANTLIB += gif jpeg png tiff webp webpdemux webpmux
WANTLIB += intl pcre

MASTER_SITES =          https://github.com/libvips/libvips/releases/download/v${VERSION}/

# Dependencies
LIB_DEPENDS +=          devel/glib2 graphics/png graphics/jpeg
LIB_DEPENDS +=          graphics/libwebp graphics/giflib graphics/tiff

SEPARATE_BUILD =        Yes
USE_GMAKE =             Yes

COMPILER = base-clang ports-gcc
CONFIGURE_STYLE =       gnu

.include <bsd.port.mk>

PeerTube download

export VERSION=…
cd /var/www/peertube/versions
doas -u _peertube wget "https://github.com/Chocobozzz/PeerTube/releases/download/${VERSION}/peertube-${VERSION}.zip"
doas -u _peertube unzip peertube-${VERSION}.zip && doas -u _peertube rm peertube-${VERSION}.zip

Node and Yarn

You need yarn to install.

doas npm install --global yarn

Sharp

Each time I installed / reinstalled / updated PeerTube, the thing which was buggy, was Sharp. Guess what ? Yeah, there too.

The Sharp module installation has pain with the libvips library. If things go wrong about it, run it this way:

cd /var/www/peertube/peertube-latest
SHARP_IGNORE_GLOBAL_LIBVIPS=1 doas -u _peertube npm add sharp

You can then finish it quite normally:

cd /var/www/peertube/peertube-latest
doas -u _peertube yarn install --production --pure-lockfile

The rc script

Je me suis efforcé de coller au mieux au script rc fourni en template.

#!/bin/ksh

daemon_user="_peertube"

HOME_DIR="/var/www/peertube/"
CONF_DIR="/var/www/peertube/config/"

daemon="HOME=${HOME_DIR} NODE_CONFIG_DIR=${CONF_DIR} NODE_ENV=production USER=${daemon_user} \
        /usr/local/bin/node ${HOME_DIR}/peertube-latest/dist/server"

. /etc/rc.d/rc.subr

daemon_timeout=60

pexp="node: peertube"

rc_reload=NO

rc_cmd $1

Auxilary daemons

First thing first, PeerTube will check connectivity to the databases (PostgreSQL and Redis) and smtp servers written in its configuration. If there is any trouble there, it will fail.

OpenBSD has an smtp server in base, so it might be worth using it instead of Google & co.

It also means that PeerTube shall not start before those same daemons.

At the moment, PeerTube cannot connect via any local socket to database servers.

Redis was accepting connections on ::1 but not on 127.0.0.1. Check that up.

Nginx

PeerTube uses Nginx a lot.

Nginx will serve directly static files and sometimes videos. It shall also relay torrent flows to the tracker integrated in the instance.

You should definitely pay attention to both PeerTube and Nginx configuration files provided in the installation zip. I read them fast and used them without much tweaking.

I could not figure out which of Nginx or PeerTube shall start up first. It happens often that Nginx does not relay flows to PeerTube despite both process running smoothly and in that case, you get the famous “Bad Gateway” mistake. It is best to just restart Nginx itself.

I did not mentioned anything about TLS and acme-client, nor logs. On that matter, PeerTube is just another web server : there will be tons of logs (from PeerTube and Nginx) and you shall insure their backup and rotation and renewing of tls certs.

PF and ipv6

Same with PF, you shall accept incoming connections on the usual ports. But take care because PeerTube will open lots of connections to other servers and shall receive connections from browsers all other internet.

Ipv4 and ipv6.

So if you have ipv6 connectivity but not much stable, you can have weird troubles. You can click on a video and see the page loading, but the video would never stream. Or thumbnails would never load.

That's because some connections would be broken while other (ipv4 ones) are still running.

Finally and to thank you for reading all of this, I wish to show you this much joyfull video from Australia, because hey, I did it, I installed PeerTube on OpenBSD, and I am full of joy !