As a part of restoring all the posts from the old Drupal server I finally had to use Duplicity to restore a backup. Always wondered how it’d actually work. Twas easy! This restores a backup from 9/1/2014 to /tmp/restore.
duplicity restore –time 2014-09-01 s3+http://rack_deb/drupal /tmp/restore2
Ignore this post if you have no idea what ZFS is.
ZFS related notes for posterity. I’ve got it up and running as well as BTRFS. ZFS seems better. So that’s settled.
Continue reading “ZFS notes”
Here’s a quick python script to use the camera you can attach to a Raspberry Pi and take a picture whenever it detects motion. This is pretty cool, there are all sorts of silly things you can do if you can do motion detection. And to do it with the Raspberry Pi and attached camera is relatively inexpensive (less than $100) to boot! Next time someone knocks our mailbox down we’ll have ’em!
I started from the script here and modified it to just take one picture to both test whether anything changed as well as save the picture if something changed. Seems a bit better than the original taking two pictures – at least this doesn’t cause the Pi to freeze. That means the Pi have to scan a bit larger image for a change but skipping pixels seems to keep it under control. It also takes better pictures as takes some time (the -t 500 in the raspistill call) to calibrate the camera.
from datetime import datetime
from PIL import Image
# Motion detection settings:
# Threshold (how much a pixel has to change by to be marked as "changed")
# Sensitivity (how many changed pixels before capturing an image)
# ForceCapture (whether to force an image to be captured every forceCaptureTime seconds)
threshold = 10
sensitivity = 2000
forceCapture = True
forceCaptureTime = 60 * 60 # Once an hour
# File settings
saveWidth = 1280
saveHeight = 960
diskSpaceToReserve = 500 * 1024 * 1024 # Keep 500 mb free on disk
# Capture a small test image (for motion detection)
command = "raspistill -w %s -h %s -t 500 -e bmp -o -" % (1280, 960)
imageData = StringIO.StringIO()
im = Image.open(imageData)
buffer = im.load()
return im, buffer
# Keep free space above given level
if (getFreeSpace() < bytesToReserve):
for filename in sorted(os.listdir(".")):
if filename.startswith("capture") and filename.endswith(".jpg"):
print "Deleted %s to avoid filling disk" % filename
if (getFreeSpace() > bytesToReserve):
# Get available disk space
st = os.statvfs(".")
du = st.f_bavail * st.f_frsize
# Get first image
image1, buffer1 = captureTestImage()
# Reset last capture time
lastCapture = time.time()
# Get comparison image
image2, buffer2 = captureTestImage()
# Count changed pixels
changedPixels = 0
for x in xrange(0, 1280, 4):
for y in xrange(0, 960, 4):
# Just check green channel as it's the highest quality channel
pixdiff = abs(buffer1[x,y] - buffer2[x,y])
if pixdiff > threshold:
changedPixels += 1
# Check force capture
if time.time() - lastCapture > forceCaptureTime:
changedPixels = sensitivity + 1
# Save an image if pixels changed
if changedPixels > sensitivity:
lastCapture = time.time()
timeN = datetime.now()
print "Save jpg"
filename = "capture-%04d%02d%02d-%02d%02d%02d.jpg" % (timeN.year, timeN.month, timeN.day, timeN.hour, timeN.minute, timeN.second)
# Swap comparison buffers
image1 = image2
buffer1 = buffer2
print "Done waiting"
So looking at XKCD you’d think ‘How could they possibly let this bug go?’ That’s so obvious:
Looking at this other blog that purports to show the code causing the issue (and I do look at various forms of computer code daily) you’d say ‘How the heck would they ever know this is a problem?’ Perhaps we shouldn’t be allowed to code in C. The fix:
The most important part of the fix was this:
/* Read type and payload length first */
if (1 + 2 + 16 > s->s3->rrec.length)
return 0; /* silently discard */
hbtype = *p++;
if (1 + 2 + payload + 16 > s->s3->rrec.length)
return 0; /* silently discard per RFC 6520 sec. 4 */
pl = p;
This does two things: the first check stops zero-length heartbeats. The second check checks to make sure that the actual record length is sufficiently long. That’s it.
OK, maybe I see somewhere there a related fix they descibe. I suppose if one gets over the ‘s->s3->rrec.length’ it makes sense as long as they are doing that (I assume) pointer *p++ right too. And pl=p, that’s totally obvious!
If you look here it wasn’t looking to request lengths at all before the fix. Some QA! If the technology is to allow you to request a number of letters and you don’t test that you can get more than you should that is still sort of bad.
Thank goodness for SQL, Java, and Python! Even thank goodness for Mumps (the programming language). At least (not in Mumps) then you could make the request an object of the requested length and drop everything else.
This site was secure I found thanks to it using forward secrecy (at least since the conversion to Debian – seems like the old version of FreeBSD was unaffected as well). No idea how it was using that but phew! Since ran some SSL check and we’re an even more tight ship now!
The one bit of switching to Debian that was difficult was setting up Exim and SMTP. I tried using the configuration off of FreeBSD but Debian just has it’s own special setup. For whatever reason the FreeBSD configuration resulted in SMTP reject errors.
This is a helpful command for debugging the exim configuration – it really helps to know what Exim4 is actually doing with all those config files!
So I tried the Debian approach and that worked ok for receiving mail (which the FreeBSD also did). However I still couldn’t connect with the mail client. Looking at it again if I’d read that wiki closely and followed all the instructions it would have saved me a couple of hours… I just needed to uncomment the SASL bit in /etc/exim4/exim4.conf.template. So always follow the instructions closely I suppose!
So the old server has always crashed quite a bit and been a bit slow for no obvious reason otherwise. So I spent some time trying to optimize the server with various things but then it occurred to me to consider this message on login:
### ZFS Tuning ###
ZFS is NOT tuned for instances with less than 1GB RAM.
For tuning 256MB and 512MB instances, the following link is recommended reading
Now the image I’m using has less than 1GB RAM so I went to that wiki and tried to do some stuff and totally messed up the performance. At least that somewhat proved to me that it was ZFS and low memory causing the random hangs. The wiki notes the ‘bursting’ behaviour of the ZFS filesystem sometimes. That’s what the server had been doing! Trying to tweak it a bit I just made it worse. There was nothing in swap and any disk based activity was REALLY SLOW. Just tarring things up after my tweaks freezes the system… Darn fancy filesystems. Combine that with I was starting to get some package issues with a mixture of compiled packages and trying ‘pkgng’ I thought I’d given Debian a try again.
Debian has no slowness issues! I’m 99% sure it was the ZFS and low memory. So as much as I like FreeBSD this I’ll use Debian. I do like apt better than pkgng anyhow, much more refined. I was almost getting into a version of RPM hell with FreeBSD there… (At least it’d never ruin the whole system with BSD though)
Anyhow switching to Debian worked fine, I can’t get the SMTP server working for some reason but everything else was easy to port. And there is no more random hanging. Also no swap space is used at all for the same configuration, so either it very likely was ZFS that was the issue on FreeBSD.
I have to search for this periodically being sites like to change their SSL certificates every once and a while. So here’s another place to look for it!
If you don’t know what Fetchmail is its a little program (maybe its gigatic, I don’t know) that fetches email for you from many different email servers. This allows me to have one place where I store all my email across all my email addresses and then I can use an IMAP email server to view it all in one place. Very nice if you’ve many different email address.
In order to do this magic you have to tell fetchmail how to login to your email accounts and its always better to do this securely over SSL. To do it over SSL you have to tell fetchmail the site’s SSL certificiate (or you could weaken fetchmail to not care to validate the certificate but that seems silly). And to do that you have to tell it in hex or other computer sprach.
So first you need to get the SSL certificate in computer sprach:
openssl s_client -connect pop3.live.com:995 -showcerts | openssl x509 -fingerprint -noout -md5
Running this will get you the certificate computer sprach entry itself that’ll look something like this:
Second you need to tell fecthmail to use this id when it connects to the server in the fetchmailrc file:
poll pop3.live.com uidl protocol POP3 user “email@example.com” there with password “ohSuchASecurePassword!@#$” is xxxx@tttt here ssl sslfingerprint ‘91:63:CF:6F:DF:03:3C:B9:3E:19:B3:1C:FF:EB:3C:25‘
There you have it! That’s almost a good memorable, secure password I just made up there.
One sad day our old modem failed. The phone company gave us a new DSL modem/router. I had figured out how to setup portforwarding and such on the old modem. I can’t get into the new one at all! After briefly attempting to get info out of the phone company about how to access the new modem’s admin interface to no success I gave up on all my nice old port forwards.
Continue reading “Port forwarding VNC connection”
I’ve got a pretty nice free setup for a webserver I’ve been taking for granted now. So for anyone starting off wanting to setup a server with email, webserver, and database here are some applications I’ve found reliable and somewhat easy to figure out.
Continue reading “Notes for a general webserver”
A couple of really handy things that may not be obvious when using the RaspberryPi.
Continue reading “Pi bits”