Monthly Archives: September 2014

Backing up OpenVZ ploop snapshots

As I mentioned in an earlier post I have been using snapshots to back up my OpenVZ servers. At first I used a slightly modified version of the backup script found on the OpenVZ wiki to create backups, and that worked fine except for one missing feature: logging.

This seemed as good an opportunity as any to start learning some Python (which to my great embarrassment I have not used much before). The result is a small Python script OVZ-Backup that backs up all OpenVZ containers (or a subset of them) using ploop snapshots and rsync. If there are errors it logs them with syslog and (optionally) sends out error messages to a list of email addresses/users.

The script sends out email using the ‘mail’ command and can only send them to local users as is. My last HowTo explains how you can use postfix to forward user email to an external address and relay emails sent from local email clients without SMTP support through your email provider’s SMTP server. This would let you send error messages to any email address you want.

Encrypting and forwarding local email to an external email address

Last week I set out to find a reliable and permanent solution to my problems with unread system mail and undelivered cron error messages, and I can now see that sending, delivering, and receiving email is far more complicated than I had thought. I have had to learn more about SMTP, SSL/TLS, and email delivery than I ever wanted to know, and I still only know the bare basics.

When I started I had two requirements:

  1. All local email sent to the root user (and preferably everyone else) must be forwarded to an external SMTP server and email account
  2. All email that leave my network must be encrypted

After experimenting with number of different applications (GNU Anubis, Nullmailer, ESMTP, MSMTP, etc) I finally found something that worked; Postfix with GPG-Mailgate. If all you need is the ability to send email to an external account then there are other applications you could use instead (I had some success with MSMTP and Nullmailer). The advantage of using Postfix is its flexibility and maturity. All other applications I tried had some small thing they could not do reliably or would fail in some edge cases, such as only forwarding some of the emails sent to root, but not quite all of them.

This guide will show each step needed to set this up, and a few mistakes to look out for. The first part will cover Postfix and how to configure it to forward all emails sent to a local user to an external email account. The second part will show how to set up GPG-Mailgate so that all emails that are sent to certain accounts are encrypted with GPG before they leave the server. If all you need is mail forwarding then you can stop after the first part, and if you already have Postfix configured you can jump straight to the second part.

A few assumptions will be made in this guide.

  1. You are using the root account. If you are not then you will have to prepend ‘sudo’ to some commands.
  2. You are using Debian. Everything should still work if you are using a different Linux distribution, but you may have to make some minor changes.
Names and servers

I will be using the following user names, email accounts, and servers in this guide.

User names

root : The local root user

gpgmap : The user account used for GPG-Mailgate

Email accounts

[email protected] : The fully qualified email address of the root user

[email protected] : An external email account that we want to send email through

[email protected] : An external email account that we want to forward all local emails to

Domains : Your local domain : The domain of your email provider : The domain of the email provider for the account you want to forward all local email to

Servers : The SMTP server of your email provider : The computer you want to forward local emails from

Part 1 – Forwarding local email to an external SMTP server

The first step is to install and configure Postfix to forward all emails sent to the root user (or any user you want) to an external SMTP server and email account. This is not difficult, but it does require a few steps.

First, if you do not already have Postfix install it and some dependencies.

apt-get install postfix libsasl2-modules

Fill in any domain names the installer asks for and choose the “satellite” option.

If you are not already using Postfix then you are probably running Sendmail. Before starting Postfix you will have to turn it off. To avoid future error messages and warnings I also recommend you completely uninstall it.

service sendmail stop

apt-get remove sendmail-base sendmail-cf sendmail-doc

Next, start postfix.

service postfix start

Make sure that it started correctly.

service postfix status

Next comes the hard part (not really); configuring Postfix. The exact details depends on your email provider. Everything I describe here will be completely compatible with Gmail.

Most of the settings you need should have been configured for you by the installer.

Open ‘/etc/postfix/’ and make sure that the relay host is set to your email provider’s SMTP server.

relayhost =

And add these lines:

smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous
smtp_sasl_tls_security_options = noanonymous
smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt

Next, create a password file ‘/etc/postfix/sasl_passwd’ and add the following to it: [email protected]:yourpassword

The SMTP hostname and port must match relayhost in the file.

Update Postfix’s password database and settings.

postmap /etc/postfix/sasl_passwd
service postfix reload

Everything you need to send emails through Postfix using your email provider’s SMTP server should now be in place. Before proceeding you should test that it is working.

echo "Testing Postfix email delivery" | mail -s "Test email" [email protected]

If the email doesn’t arrive check in ‘/var/log/sysIog’ for errors and make sure that you can successfully send emails using the same settings from an email client (such as Thunderbird).

Now that that is working we only need to tell Postfix to forward all emails sent to the root user to an external email account. Open ‘/etc/aliases’ and add the following line:

root: [email protected]

If you want to forward emails for other users than root you simply add them to the list as well. When you are done run:


That’s it. If everything is configured properly all emails sent to the root user will now be forwarded to [email protected]

echo "Testing Postfix forwarding" | mail -s "Test email" root

If that was all you needed then you can stop here, but you are now sending unencrypted emails through the internet containing potentially sensitive information. If you think that sounds dangerous then continue to the second part were we will set up some automated GPG encryption.

A warning about restrictive SMTP servers and the From header

It is possible that these settings won’t work with your particular email provider. I’m using a Gmail account to forward email. It isn’t my primary email provider, but there are practical reasons for using it in this case. Unlike some other email providers Gmail does not care about what you specify in the ‘From’ header. It will overwrite whatever is in it with the Gmail user you used. This simplifies things greatly when forwarding local emails since you would otherwise have to change the contents of the header before sending it out. Postfix has the ability to replace addresses in outbound email (just google smtp_generic_maps or sender_canonical_maps and you should find instructions for how to configure Postfix), however, depending on how an application fills out the headers in emails they send out this may not work reliably in all situations. I tried to set this up with a more restrictive email account, but I could not get it to forward cron error messages.

If you need this for your email provider, and if you manage to set it up successfully, please leave a comment explaining how you did it. For everyone else; just use a Gmail account. It’s easier.

Part 2 – Encrypting outgoing email with GPG

It is possible that some of the applications sending email to your local accounts might send data that you don’t want any stranger on the internet to see. In my paranoid mind that just isn’t acceptable, and so we will be adding a layer of encryption to Postfix using GPG-Mailgate. GPG-Mailgate is a content filter script for Postfix that will encrypt a received email if there is a public GPG key available for its recipient, and if the email is not already encrypted. It is relatively easy to install and configure, but be warned that if you do not configure it correctly it will probably fail silently and send out empty emails.

First, either download the source code from Github or clone it.

git clone

Next, manually put everything where it needs to be. Replace python2.6 with your python version.

cd gpg-mailgate
cp /usr/local/bin/
cp -r GnuPG /usr/lib/python2.6/

Make sure that all permissions are correct.

chown root:root /usr/local/bin/
chmod 755 /usr/local/bin/
chown -R root:root /usr/lib/python2.6/GnuPG
chmod 755 /usr/lib/python2.6/GnuPG
chmod 644 /usr/lib/python2.6/GnuPG/

Create a user to run the GPG-Mailgate script as and import the public key you want to encrypt forwarded email with.

useradd -s /bin/false -d /var/gpg -M gpgmap
mkdir -p /var/gpg/.gnupg
chown -R gpgmap:gpgmap /var/gpg/.gnupg
chmod 700 /var/gpg/.gnupg
sudo -u gpgmap /usr/bin/gpg --import yourpublic.key

yourpublic.key is the public part of your GPG key pair. If you don’t have one already then your will need to create one. How to do that is beyond the scope of this guide, but it isn’t difficult.

Check that everything worked and that the key is in place.

sudo -u gpgmap /usr/bin/gpg --list-keys  --keyid-format long

This should give you something like this.

pub 4096R/0123456789ABCDEF 2014-09-17
uid                        Your Name <[email protected]>
sub 4096R/FEDCBA9876543210 2014-09-17

Save whatever you have instead of ‘0123456789ABCDEF’. This is the identifier of your public key and you will need it later.

Add the following to the end of ‘/etc/postfix/’:

gpg-mailgate unix - n n - - pipe
flags= user=gpgmap argv=/usr/local/bin/ ${recipient} inet n - n - 10 smtpd
  -o content_filter=
  -o receive_override_options=no_unknown_recipient_checks,no_header_body_checks
  -o smtpd_helo_restrictions=
  -o smtpd_client_restrictions=
  -o smtpd_sender_restrictions=
  -o smtpd_recipient_restrictions=permit_mynetworks,reject
  -o mynetworks=
  -o smtpd_authorized_xforward_hosts=

Make certain that the user name and script location matches what you used above.

Add the following to ‘/etc/postfix/’:

content_filter = gpg-mailgate

And reload Postfix’s settings.

service postfix reload

Create a configuration file for GPG-Mailgate at ‘/etc/gpg-mailgate.conf’ and add the following to it:

# whether gpg-mailgate should add a header after it has processed an email
# this may be useful for debugging purposes
add_header = yes

# whether we should only sign emails if they are explicitly defined in
# the key mappings below ([keymap] section)
# this means gpg-mailgate won't automatically detect PGP recipients
keymap_only = yes

# the directory where gpg-mailgate public keys are stored
# (see INSTALL for details)
keyhome = /var/gpg/.gnupg

# For logging to syslog. 'file = syslog', otherwise use path to the file.
file = syslog
verbose = no

# the relay settings to use for Postfix
# gpg-mailgate will submit email to this relay after it is done processing
# unless you alter the default Postfix configuration, you won't have to modify this
host =
port = 10028

# You can find these by running the following command:
# gpg --list-keys --keyid-format long [email protected]
# Which will return output similar to:
# pub 1024D/AAAAAAAAAAAAAAAA 2007-10-22
# uid Joe User <[email protected]>
# sub 2048g/BBBBBBBBBBBBBBBB 2007-10-22
#[email protected] = <gpg key id>
[email protected] = 0123456789ABCDEF
[email protected] = 0123456789ABCDEF

Replace the addresses and keys on the last two lines with the email address you want to forward emails to and the identifier of the public key you imported earlier.

The final line is a bit of a hack. GPG-Mailgate uses the ‘To’ header to find the GPG key to use when encrypting email. If it can’t find a matching entry in the configuration file it will not encrypt it. This creates a minor complication since Postfix doesn’t rewrite the ‘To’ header when forwarding local email. Any email sent to the root user will therefore have a ‘To’ address of ‘[email protected]’; which GPG-Mailgate won’t recognise. To fix this we add the local email address to the configuration file as well.

This should be all. Now test that everything is working.

echo "Testing GPG encryption" | mail -s "Test GPG" [email protected]
echo "Testing GPG encryption to root user" | mail -s "Test GPG root" root

These commands should both send an encrypted email to [email protected] If it doesn’t work check ‘/var/log/syslog’ for errors.

If GPG-Mailgate did not encrypt your emails then it is likely that it did not find a matching public key. Make sure that the keys and addresses in ‘/etc/gpg-mailgate.conf’ are correct.

If you get empty messages then it is likely that GPG returned an error to GPG-Mailgate. This will cause GPG-Mailgate to fail silently. Make sure that the gpgmap user has the permissions needed to use the GPG keys in ‘/var/gpg/.gnupg’ and that gpgmap is used by Postfix when running GPG-Mailgate.

If you get an error about a missing GnuPG module then you either set the wrong permissions for the GnuPG folder and its contents, or you placed it in the wrong python folder.

If everything is working than this should be it.

  • You can now send email through an external SMTP server from the command line
  • All emails sent to the root user are forwarded to an external email address
  • All emails sent to [email protected] are automatically encrypted before they leave the server


I’m in the process of moving all of my servers to a small OpenVZ cluster. I had hoped that this would be painless, but some of my servers are not behaving well. The web server appears to working fine, but If I can’t figure out what the problem is I may have to take everything down for maintenance. Don’t expect the blog to be available for the next couple of days.

Update 1:

Turns out that most of my problems didn’t have anything to do with the move. It was only some old (and in one case very old) misconfigurations that didn’t take effect until I rebooted one of my servers; that, and I had forgotten just how much memory Gitlab uses.

Update 2:

The cluster is almost complete. I only need to configure Heartbeat and it should be done. Unfortunately, moving my servers revealed some rather alarming deficiencies in how some of the older servers have been configured, how backups are handled, and especially in how they are monitored. One server (albeit not a very important one) had not been backed up for months, and I had not been alerted of this. This is not acceptable!

The cluster will have to wait until I have dealt with this.

Update 3:

I have learnt a few things.

First, backup systems will (in this case at least) only work if all clocks are in sync, and if one of the servers doesn’t have a CMOS battery then any NTP failures will break the backup system. If the server in question also doesn’t have a proper hard drive to store logs on, can’t run OSSEC, can’t store logs remotely using rsyslog, and doesn’t send system mail to an external mail server, then the first you will hear of this is when you need those backups.

Second, a DRBD cluster is not worth the effort of managing properly for a couple of private servers. It was very easy to set up, but robust it was not. My first attempt to try taking a node down ended with the whole cluster going down and refusing to synchronize. It was probably my fault (it was in the middle of the night, so I’m pretty sure it must have been my fault in some way), but when the DRBD service started giving error messages and crashing, I decided it was time to try something else while I knew the data on one node was still good.

I did not want to be forced to restore my servers from backups. The reason I did this in the first place was because I’m unhappy with my backup system and I needed a stop-gap I could use until I can set up something more robust. Perhaps I will try this again in the future, but for now I will fall back on simple snapshots.

So ends my latest adventure with needlessly complex solutions to simple problems; in disgraceful failure and the use of a more appropriate tool. Next on the list; soldering and configuration management systems.

writetoserial – A small program that sends binary data to a serial port

While waiting for the hardware I need to get any further on my little programming project I have been experimenting with sending and receiving data from the FRDM-KL25Z through a serial port. Mostly to prepare the tools I will need to debug the system I’m planning on building, and in particular the code and algorithms I will use for file encryption. Unfortunately, sending and receiving files and binary data over a serial port proved to be a little harder to do than I had anticipated. It was possible to do this to some extent by using simple tools like ‘cat’ to write data to the serial port’s device file, but I could not get it to work reliably for binary data or files over a few kilobytes in size. It would sometimes drop data or stop halfway for no apparent reason.

Because of this I wrote a simple little program ‘writetoserial’ that reads the contents of file and writes it to a serial port as binary data. This works far more reliably and I’m now able to send both large text files and binary files to the FRDM-KL25Z.