(Forced) Move from zfsonlinux to Ubuntu ZFS

I upgraded the server from 14.04 to 16.04 and slowly discovered everything was not working perfectly with the ZFS functionality. Researching it appears zfsonlinux doesn’t support 16.04 since ZFS is a part of 16.04. However I couldn’t get the Ubuntu version to load and got this error. It gave me the pointer that the old trusty (14.04) version was the issue.

The following packages have unmet dependencies:
 zfsutils-linux : Depends: zfs-doc (= but is to be installed
                  Depends: libnvpair1linux but it is not going to be installed
                  Depends: libuutil1linux but it is not going to be installed
                  Depends: libzfs2linux but it is not going to be installed
                  Depends: libzpool2linux but it is not going to be installed
                  Recommends: zfs-zed but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

So had to purge the non Ubuntu trusty packages and start over. This will find the old installed zfs packages:

dpkg -l|grep zfs

Then you can purge the packages old trusty (14.04) to remove them:

sudo apt-get purge zfs-doc

Make sure nothing odd is in /etc/apt. Then install the native Ubuntu version with the normal:

sudo apt install zfsutils-linux

Black Mirror in real life – Chinese public credit scores

China, and specifically Shanghai for now, is instituting public credit scores to rank citizens by various factors. The end goal to improve the behavior of people by ranking them publicly. This has to have been in the works for awhile. Did Black Mirror just steal the plot for Nosedive from what China is actually doing?

Quite a few places are reporting it and here’s the NRP version. I read it first in the Economist weeks ago so they should get credit for breaking it really but NPR reminded me. A quote from NPR’s version:

“We want to make Shanghai a global city of excellence,” says Shao Zhiqing, deputy director of Shanghai’s Commission of Economy and Informatization, which oversees the Honest Shanghai app. “Through this app, we hope our residents learn they’ll be rewarded if they’re honest. That will lead to a positive energy in society.”

Shao says Honest Shanghai draws on up to 3,000 items of information collected from nearly 100 government entities to determine an individual’s public credit score.

A good score allows users to collect rewards like discounted airline tickets, and a bad score could one day lead to problems getting loans and getting seats on planes and trains.

The wonders of Let’s Encrypt and Certbot in relation to StartSSL/StartCom issues

All websites should be encrypted. Definitely don’t do anything secure over an unencrypted connection.

To support that the site has always had SSL setup. However I just discovered that the SSL certs the site uses by StartSSL have been revoked by all major browsers it appears!

A little digging for a new free SSL cert site came up with Let’s Encrypt. That in combo with Certbot is SUPER EASY to get certs and keep them up to date. All for free! I’ll probably even donate it’s so easy now!

The trick for certbot for me was to use the standalone check so I didn’t have to mess with the NGINX server’s folder security.

$ certbot certonly --standalone -d example.com -d www.example.com

Then you just point NGINX to the PEM files it creates and you’re set!

ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;

EASY PEASY! Now Chrome and Safari work again!

Then to renew all the certs at once you can run this

certbot renew --pre-hook "service nginx stop" --post-hook "service nginx start"

Duplicity restore

As a part of restoring all the posts from the old Drupal server I finally had to use Duplicity to restore a backup. Always wondered how it’d actually work. Twas easy! This restores a backup from 9/1/2014 to /tmp/restore.

duplicity restore –time 2014-09-01 s3+http://rack_deb/drupal /tmp/restore2

The Arduino is really quite handy to have around. Right now we’ve got one reporting the temperature and humidity of a chicken incubater. It could also control the temperature, humidity, and egg rotation but we haven’t gone there yet. Even so it’s nice to get the measurements with the audible warning. Then you can always repurpose all this stuff to do something else once you are done incubating.

You just need (unvalidated list):

Add in a stepper motor, voltage switch, and some sort of fluid controller and you could do it all. Unglorious image of this – the OLED is very nice:

Here’s the relevant code – it of course plays the 1st 4 notes of Beethoven’s 5th as a warning (or at least an attempt at that not having found the true notes):

#include <Wire.h>
#include "Adafruit_GFX.h"
#include "Adafruit_SSD1306.h"
#include <Adafruit_Sensor.h>
#include "DHT.h"
#define OLED_DC 11
#define OLED_CS 12
#define OLED_CLK 10
#define OLED_MOSI 9
#define OLED_RESET 13

#define DHTTYPE DHT22   // DHT 22  (AM2302)
#define DHTPIN 2     // what pin we're connected to

#define speakerPin 4

// TONES  ==========================================
// Start by defining the relationship between 
//       note, period, &  frequency. 
#define  c     3830    // 261 Hz 
#define  d     3400    // 294 Hz 
#define  e     3038    // 329 Hz 
#define  f     2864    // 349 Hz 
#define  g     2550    // 392 Hz 
#define  a     2272    // 440 Hz 
#define  b     2028    // 493 Hz 
#define  C     1912    // 523 Hz 

//thermometer - humidity

float temp;
float humidity;

void setup() {


void loop() {
  display.clearDisplay();   // clears the screen and buffer

void getWeather() {
  //read DHT
  float humidity = dht.readHumidity();
  float t = dht.readTemperature();
  t = t * 9.0 / 5.0 + 32.0;
  // check if returns are valid, if they are NaN (not a number) then something went wrong!
  if (isnan(t) || isnan(humidity)) {
    Serial.println("Failed to read from DHT");
  else {
    Serial.print("Humidity: "); 
    Serial.print(" %\t");
    Serial.print("Temperature: "); 
    Serial.println(" *C");


  delay(10000);   //10 seconds seems often enough

void updateDisplay(float h, float t) {
  display.print("Temp:  ");
  display.println(" F");
  display.println(" ");
  display.print("Humidity:  ");
  display.println(" ");

  // warnings
  if (t < 98.8 || t > 100) {
    display.println("CHECK TEMP!!!");
    display.println(" ");

  if (h < 50 || h > 70) {
    display.println("CHECK HUMIDITY!!!");

void playWarning() {
  tone(speakerPin, g);
  tone(speakerPin, g);
  tone(speakerPin, g);
  tone(speakerPin, C);


Here’s a quick python script to use the camera you can attach to a Raspberry Pi and take a picture whenever it detects motion. This is pretty cool, there are all sorts of silly things you can do if you can do motion detection. And to do it with the Raspberry Pi and attached camera is relatively inexpensive (less than $100) to boot! Next time someone knocks our mailbox down we’ll have ’em!

I started from the script here and modified it to just take one picture to both test whether anything changed as well as save the picture if something changed. Seems a bit better than the original taking two pictures – at least this doesn’t cause the Pi to freeze. That means the Pi have to scan a bit larger image for a change but skipping pixels seems to keep it under control. It also takes better pictures as takes some time (the -t 500 in the raspistill call) to calibrate the camera.

#!/usr/bin/env python

import StringIO
import subprocess
import os
import time
from datetime import datetime
from PIL import Image

# Motion detection settings:
# Threshold (how much a pixel has to change by to be marked as "changed")
# Sensitivity (how many changed pixels before capturing an image)
# ForceCapture (whether to force an image to be captured every forceCaptureTime seconds)
threshold = 10
sensitivity = 2000
forceCapture = True
forceCaptureTime = 60 * 60 # Once an hour

# File settings
saveWidth = 1280
saveHeight = 960
diskSpaceToReserve = 500 * 1024 * 1024 # Keep 500 mb free on disk

# Capture a small test image (for motion detection)
def captureTestImage():
    command = "raspistill -w %s -h %s -t 500 -e bmp -o -" % (1280, 960)
    imageData = StringIO.StringIO()
    imageData.write(subprocess.check_output(command, shell=True))
    im = Image.open(imageData)
    buffer = im.load()
    return im, buffer

# Keep free space above given level
def keepDiskSpaceFree(bytesToReserve):
    if (getFreeSpace() < bytesToReserve):
        for filename in sorted(os.listdir(".")):
            if filename.startswith("capture") and filename.endswith(".jpg"):
                print "Deleted %s to avoid filling disk" % filename
                if (getFreeSpace() > bytesToReserve):

# Get available disk space
def getFreeSpace():
    st = os.statvfs(".")
    du = st.f_bavail * st.f_frsize
    return du

# Get first image
image1, buffer1 = captureTestImage()

# Reset last capture time
lastCapture = time.time()

while (True):
    # Get comparison image
    image2, buffer2 = captureTestImage()

    # Count changed pixels
    changedPixels = 0

    for x in xrange(0, 1280, 4):
        for y in xrange(0, 960, 4):
            # Just check green channel as it's the highest quality channel
            pixdiff = abs(buffer1[x,y][1] - buffer2[x,y][1])
            if pixdiff > threshold:
                changedPixels += 1

    # Check force capture
    if forceCapture:
        if time.time() - lastCapture > forceCaptureTime:
            changedPixels = sensitivity + 1

    # Save an image if pixels changed
    if changedPixels > sensitivity:
        lastCapture = time.time()
        timeN = datetime.now()
        print "Save jpg"
        filename = "capture-%04d%02d%02d-%02d%02d%02d.jpg" % (timeN.year, timeN.month, timeN.day, timeN.hour, timeN.minute, timeN.second)
        image2.save(filename, "JPEG")
    # Swap comparison buffers
    image1 = image2
    buffer1 = buffer2
    print "Done waiting"


So looking at XKCD you’d think ‘How could they possibly let this bug go?’ That’s so obvious:

Looking at this other blog that purports to show the code causing the issue (and I do look at various forms of computer code daily) you’d say ‘How the heck would they ever know this is a problem?’ Perhaps we shouldn’t be allowed to code in C. The fix:

The fix

The most important part of the fix was this:

/* Read type and payload length first */
if (1 + 2 + 16 > s->s3->rrec.length)
    return 0; /* silently discard */
hbtype = *p++;
n2s(p, payload);
if (1 + 2 + payload + 16 > s->s3->rrec.length)
    return 0; /* silently discard per RFC 6520 sec. 4 */
pl = p;

This does two things: the first check stops zero-length heartbeats. The second check checks to make sure that the actual record length is sufficiently long. That’s it.

OK, maybe I see somewhere there a related fix they descibe. I suppose if one gets over the ‘s->s3->rrec.length’ it makes sense as long as they are doing that (I assume) pointer *p++ right too. And pl=p, that’s totally obvious!

If you look here it wasn’t looking to request lengths at all before the fix. Some QA! If the technology is to allow you to request a number of letters and you don’t test that you can get more than you should that is still sort of bad.

Thank goodness for SQL, Java, and Python! Even thank goodness for Mumps (the programming language). At least (not in Mumps) then you could make the request an object of the requested length and drop everything else.

This site was secure I found thanks to it using forward secrecy (at least since the conversion to Debian – seems like the old version of FreeBSD was unaffected as well). No idea how it was using that but phew! Since ran some SSL check and we’re an even more tight ship now!

Debian and Exim4

The one bit of switching to Debian that was difficult was setting up Exim and SMTP. I tried using the configuration off of FreeBSD but Debian just has it’s own special setup. For whatever reason the FreeBSD configuration resulted in SMTP reject errors.

This is a helpful command for debugging the exim configuration – it really helps to know what Exim4 is actually doing with all those config files!

exim -bP

So I tried the Debian approach and that worked ok for receiving mail (which the FreeBSD also did). However I still couldn’t connect with the mail client. Looking at it again if I’d read that wiki closely and followed all the instructions it would have saved me a couple of hours… I just needed to uncomment the SASL bit in /etc/exim4/exim4.conf.template. So always follow the instructions closely I suppose!

FreeBSD, ZFS, Rackspace, and to Debian

So the old server has always crashed quite a bit and been a bit slow for no obvious reason otherwise. So I spent some time trying to optimize the server with various things but then it occurred to me to consider this message on login:

### ZFS Tuning ###

ZFS is NOT tuned for instances with less than 1GB RAM.

For tuning 256MB and 512MB instances, the following link is recommended reading


Now the image I’m using has less than 1GB RAM so I went to that wiki and tried to do some stuff and totally messed up the performance. At least that somewhat proved to me that it was ZFS and low memory causing the random hangs. The wiki notes the ‘bursting’ behaviour of the ZFS filesystem sometimes. That’s what the server had been doing! Trying to tweak it a bit I just made it worse. There was nothing in swap and any disk based activity was REALLY SLOW. Just tarring things up after my tweaks freezes the system… Darn fancy filesystems. Combine that with I was starting to get some package issues with a mixture of compiled packages and trying ‘pkgng’ I thought I’d given Debian a try again.

Debian has no slowness issues! I’m 99% sure it was the ZFS and low memory. So as much as I like FreeBSD this I’ll use Debian. I do like apt better than pkgng anyhow, much more refined. I was almost getting into a version of RPM hell with FreeBSD there… (At least it’d never ruin the whole system with BSD though)

Anyhow switching to Debian worked fine, I can’t get the SMTP server working for some reason but everything else was easy to port. And there is no more random hanging. Also  no swap space is used at all for the same configuration, so either it very likely was ZFS that was the issue on FreeBSD.