Hex S The Good The Bad The Ugly

I am in the fortuned situation to have a fiber7 directly to my home. This means unfortunately goodbye to my apu4d2 board from pcengines. Because I couldn't figure out why the performance was capt at ~300 Mbps. And there is no way to connect a sfp module without a media convert to a apu board. Which is a extra device meaning an extra thing which can fail. So I asked @wauwuff for recommendations on what to get. An this is why I got the MikroTik hEX S. Here are the specs I was most excited about:

  • first of all @wauwuff promised me that it would deliver 1Gbits speed
  • Passive PoE up to 57V out port
  • SFP port
  • price point

What this means for me is that I can reduce from Router, PoE injector and Media converter to just one device, the hEX S.

The Good

The Hardware is amazing. Don't get me wrong it is a plastic box. If you are not the type to get exited by a plastic box don't get your hopes up. But it is a plastic box which delivered on all the things I hopped it would. Most important of all it is capable to do 1Gbits.

The Bad

For some reason I wasn't able to connect to the router because the password wasn't reset properly. The bright side of that? It's time to figure out how the reset works. Which is not that hard if you can follow written instructions (Netinstall). Obviously I failed my first 2 attempts. Because reading is hard. On big downside of this process for me is that it is windows software.

The Ugly

The software, with one small exceptions. DDNS was super easy to setup.

/ip cloud set ddns-enabled=yes
/ip cloud print

Everything else was and is a pain to setup and configure. The software UX is less than ideal. The problem is not the the UI, which is not pretty but who cares. What do I mean by this? Mostly the software is built around functions and not workflows. Let's take OpenVPN as example to set it up you need to navigate through at least 3 sub-menus to configure. This is on top of a OpenVPN implementation which is very limited. No UDP, LZO compression and limited Cryptography support.

In general many things which should be in the same place are very disconnected.

And then there are minor issue. I needed to force PoE to power my Ubnt access point like this:

/interface ethernet poe set ether5 poe-out=forced-on

As well as the Terminal which has a weird auto-completion feature which completes without pressing tab.

Streaming Setup

This is an continuation of Podcasting With Pulse.

Building darkice

Years ago when I first compiled darkice it was complicated to get it compiled and running with mp3 support.

Amazingly darkice still exists and it's easier than ever to compile it with mp3 support.

The first step is to get the latest (darkice-1.4.tar.gz) version from www.darkice.org. And unpack it.

tar xf ~/Downloads/darkice-1.4.tar.gz -C .

And to get it with mp3 support just install the headers.

sudo dnf install lame-devel

And then build it. (Make sure you see something like checking for lame library at /usr ... found at /usr on configure)

./configure
make
make install

Start a stream

And to make life even simpler I created 2 scripts to setup the interfaces and start the stream.

setup-stream.sh

#!/bin/sh

pactl load-module module-remap-source master=alsa_input.usb-Focusrite_Scarlett_Solo_USB-00.analog-stereo master_channel_map=front-left,front-right channels=2 channel_map=mono,mono

pactl load-module module-null-sink sink_name=stream sink_properties=device.description="Streaming"

pactl load-module  module-loopback source=alsa_output.usb-Focusrite_Scarlett_Solo_USB-00.analog-stereo.monitor sink=stream latency_msec=1
pactl load-module  module-loopback source=alsa_input.usb-Focusrite_Scarlett_Solo_USB-00.analog-stereo.remapped sink=stream latency_msec=1

start-stream.sh

#!/bin/sh

darkice -c stream.cfg

Now podcasting is just:

./setup-stream.sh
./start-stream.sh

Apu Bios Upgrade

I read How to enable Core Performance Boost on AMD platforms?. Which lead me to the question did I upgrade my apu board bios. And the answer is a conclusive maybe.

So here are the steps to upgrade a apu board bios on OPNsense (or basically every FreeBSD). To check the current bios version there is a tool called dmidecode.

# dmidecode -t bios
Scanning /dev/mem for entry point.
SMBIOS 2.7 present.

Handle 0x0000, DMI type 0, 26 bytes
BIOS Information
	Vendor: coreboot
	Version: v4.9.0.3
	Release Date: 03/08/2019
	ROM Size: 8192 kB
	Characteristics:
		PCI is supported
		PC Card (PCMCIA) is supported
		BIOS is upgradeable
		Selectable boot is supported
		ACPI is supported
		Targeted content distribution is supported
	BIOS Revision: 4.9
	Firmware Revision: 0.0

The next step is to check https://pcengines.github.io/ for new bios versions. Now it is very important to download the correct bios version which matches your hardware version.

And then just one flashrom command is needed (this needs root permissions):

# pkg install flashrom
# fetch https://3mdeb.com/open-source-firmware/pcengines/apu3/apu3_v4.9.0.5.rom
# flashrom -w apu3_v4.9.0.5.rom -p internal
flashrom v1.0 on FreeBSD 11.2-RELEASE-p9-HBSD (amd64)
flashrom is free software, get the source code at https://flashrom.org

Using clock_gettime for delay loops (clk_id: 4, resolution: 2ns).
coreboot table found at 0x7eed0000.
Found chipset "AMD FCH".
Enabling flash write... OK.
Found Winbond flash chip "W25Q64.V" (8192 kB, SPI) mapped at physical address 0x00000000ff800000.
Reading old flash chip contents... done.
Erasing and writing flash chip... Erase/write done.
Verifying flash... VERIFIED.

Now you can reboot and enjoy your new bios!

A common issue is that the the mainboard tag does not match the tag in the rom file:

This coreboot image (PC Engines:apu3) does not appear to
be correct for the detected mainboard (PC Engines:PCEngines apu3).
Aborting. You can override this with -p internal:boardmismatch=force.

If that happens make sure you downloaded the right rom file and then force it:

# flashrom -w apu3_v4.9.0.3.rom -p internal:boardmismatch=force
flashrom v1.0 on FreeBSD 11.2-RELEASE-p9-HBSD (amd64)
flashrom is free software, get the source code at https://flashrom.org

Using clock_gettime for delay loops (clk_id: 4, resolution: 2ns).
coreboot table found at 0x77fae000.
Found chipset "AMD FCH".
Enabling flash write... OK.
Found Winbond flash chip "W25Q64.V" (8192 kB, SPI) mapped at physical address 0x00000000ff800000.
This coreboot image (PC Engines:apu3) does not appear to
be correct for the detected mainboard (PC Engines:PCEngines apu3).
Proceeding anyway because user forced us to.
Reading old flash chip contents... done.
Erasing and writing flash chip... Erase/write done.
Verifying flash... VERIFIED.

10gbe Cc2 N320e Sr

I finally connected my NAS with my build server with 10 Gigabit Ethernet.

The network card I used is the: Chelsio CC2-N320E-SR. You get this cards around 30$ from ebay and similar places. Plus the direct attach cable needed, which costs around 8-20$, we are looking at a total of less than 100$ to connect 2 computers with 10GbE. Which is very impressive to me.

The coolest part about this cards is that they just work out of the box with FreeBSD. This was harder than expected. Since I put the first card in slot 1 of my Dell T20 and it never showed up. According to the documentation the 1 slot is: One full-height, half-length x16 PCIe Gen3 card slot connected to processor. It is unclear to me why the card never showed up, not in the BIOS, not with pciconf -lv. So I moved it to slot 4: One full-height, half-length x16 (x4) PCIe Gen2 card slot connected to PCH. And it showed up.

When the card is recognized the drive is loaded automatically. And you can check dmesg if the card needs a firmware upgrade.

kernel: cxgbc0: using MSI-X interrupts (9 vectors)
kernel: found old FW minor version(5.0), driver compiled for version 7.11
kernel: cxgbc0: firmware needs to be updated to version 7.11.0

The firmware binary: t3fw-7.11.0.bin is found on https://service.chelsio.com/legacy.html. This can then be unpacked and flashed with the help of cxgbtool. (Which is shipped with FreeBSD but needs to be compiled)

cd /usr/src/tools/tools/cxgbtool
make install
cxgbtool cxgb0 loadfw t3fw-7.11.0.bin

And that's it, reboot and check dmesg again. There is a great blog post by Boris Tassou which explains many of these steps in detail if you are interested.

Now to the big question how fast is it? According to my very primitive iperf testing:

[ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 7.82 GBytes 6.71 Gbits/sec

World Backup Day

Today is the world backup day. This is a wonderful excuse to talk about how I do backups (again).

And I can happily report that the solution I build and blogged about is working perfectly. (If you missed that here is the blog post from two years ago ZFS remote backups).

The biggest change since then is the hardware. You might remember that I own a Dell T20 which is still very cool hardware but with a Intel(R) Xeon(R) CPU E3-1225 v3 it uses quit a bit of power. That is why I switched to a self built solution with a Intel(R) Xeon(R) CPU D-1528. This is 35 watt TDP instead of 84 watt TDP. But you can read all about that in my previous blog post Self built NAS

The big improvement I implements since last year is the monitoring. I switched from observium to telegraf, grafana and influx for monitoring. The next logical step was to create a dashboard for my backup status.

This is how it looks (you can find the template in this gist):

grafana backup dashboard

Which is so much better than just write the infos to a log file in /tmp and check it manually.

It is built with the Line Protocol of influx. Which allows to post data with curl.

curl -i -XPOST -u username:password 'https://hostname:8086/write?db=databasename' \
        --data-binary "backup,host=backuphost status=${code}i
backuptime,host=backuphost value=${SECONDS}i"

It is not perfect but it is the best solution I ever built. And here is your reminder: do backups, check if your backup was executed successful (visibility in a dashboard helps immensely) and last but not least try to restore it. A backup with out restoring is useless.