And while MySQL comes with replication functionality out of the box, automated failover and topology management is not part of its feature set. On top of that, it turns out that it is rather difficult to not shoot yourself in the foot when configuring replication.

In fact, without careful configuration and the right tools, a failover from a source to a replica server will almost certainly lose transactions that have been acknowledged as committed to the application.

This is a blog post about setting up lossless MySQL replication with automated failover, i.e. ensuring that not a single transaction is lost during a failover, and that failovers happen entirely without human intervention.

Terminology & Disclaimer

This blog post uses MySQL’s new and less offensive terminology as it was introduced in version 8.0.26. We refer to primary hosts as “source” and to their replication hosts as “replicas”. Configuration parameters may still use offensive terms though.

I am a software engineer and not a DBA. Please excuse errors and let me know. I’m happy to correct any mistakes.

Asynchronous replication is best effort and will lose transactions during a failover

MySQL’s default replication mechanism is asynchronous replication, meaning that a transaction committed by the application is replicated by a background thread to all connected replica hosts. As long as the source host does not crash, this is perfectly fine. If it does, however, there are no guarantees that all transactions already made it to the replica host(s) before the crash.

Consider a topology of three hosts, with A being the source and B and C being replicas of A:

Assuming sync_binlog=1 on the source host (A), the way asynchronous replication works is like this:

1. The application commits transaction to source (A)
2. A writes the transaction to its binary log
3. A executes the transaction in the storage engine (InnoDB)
4. A acknowledges transaction to the application (What if we crash here?)
5. B and C retrieve the transaction from A’s binary log, write it to their relay log via the IO thread and then apply the transaction via the SQL thread

This is obviously dangerous because the application gets an acknowledgement of a transaction before it has been replicated. If the source (A) crashes after that, the replication may not have been replicated to B and/or C and you lose transactions.

Semi-sync replication to the rescue, sort of

Clearly, asynchronous replication is not the answer if losing transactions is not acceptable for your application. Semi-sync replication solves this problem, at least partially: Unlike with asynchronous replication, the source does not acknowledge the transaction to the application until the replica acknowledges the receipt (but not the execution!) of the transaction.

Here’s how the order of operations changes with semi-sync enabled on the source and the replicas, i.e. with rpl_semi_sync_master_enabled=1 on A, and rpl_semi_sync_slave_enabled=1 on B and C:

1. The application commits transaction to source (A)
2. A writes the transaction to its binary log – Crash point 1: what if A crashes here?
3. B and C both retrieve the transaction and write it to their relay log
4. B and C acknowledge the receipt of the transaction – Crash point 2: what if A crashes here?
5. A acknowledges transaction to the application
6. A executes transaction in the storage engine (InnoDB)
7. B and C apply the transaction in the storage engine (InnoDB)

(This example flow assumes that rpl_semi_sync_source_wait_point=AFTER_SYNC and ignores rpl_semi_sync_master_timeout and rpl_semi_sync_master_wait_for_slave_count. We’ll get to that later.)

This looks much better:

If the source crashes before the transaction has been shipped to B and C (crash point 1), our application is in a consistent state: A has not acknowledged the transaction to the application before it crashed, so it’s as if it had never been committed. As long as we fail over to B or C and don’t ever restart the source (more about that later!), we’re fine.

The other crash point (crash point 2) is more interesting: If A crashes after the transaction has been transferred to the relay log on B and C, the application will lose its database connection to A and think that the transaction failed. If we were to restart A (which we shouldn’t), the transaction would still exist in A’s binary log and it’d be rolled forward in during crash recovery. In both cases the transaction won’t be lost.

Assuming we fail over to B or C, the transaction will however still be executed there, despite the connection error. Lossless replication means we don’t lose anything, so this is actually an acceptable scenario, though incredibly unexpected from a developer perspective. We have seen these throws-error-but-still-executed cases many times, so you have to be able to deal with them.

There are obviously more than two crash points, but I think you get the idea: semi-sync replication ensures that each transaction has been written to at least one other replica before sending an acknowledgement to the application.

Dangerous semi-sync pitfalls

Unfortunately, setting up semi-sync is not as dead-simple as one might hope. There are a number of inconspicuous, yet quite consequential config parameters that you have to get just right.

Accidental fallback to asynchronous replication

The rpl_semi_sync_master_timeout parameter controls how long a source will wait on the replica(s) to respond and acknowledge the receipt of a transaction before giving up and falling back to asynchronous replication. The default value is 10,000 ms (10 seconds), meaning that in its default configuration, semi-sync replication will fall back to the best effort async replication behavior if the replica(s) don’t respond in that time.

While 10 seconds may seem like a long time for a single transaction, we all know that there are dozens of things that can cause delays like that even on modern hardware: (temporary) network failures, bad disks, or heavy load – all are reasons for why replicas could (at least temporarily) not respond within 10 seconds.

Here’s an illustration of the example above, but this time we time out (3 & 4) when trying to deliver/write to the replica. Note that the transaction is still acknowledged to the application (5), despite the downstream timeout, so if a failover were to occur now, we’d lose this transaction:

We have actually been bitten by this quite severely in one of our products: A temporary network blip caused a fallback to async replication, and a subsequent failover to a replica (also due to a temporary network failure) led to the loss of 4 seconds of transactions. That may not seem like much, but it took us multiple days to repair the affected tables.

If you want to make sure that you never fall back to asynchronous replication, you must setrpl_semi_sync_master_timeout to something outrageously high, e.g., one hour or even 24 hours.

While this will guarantee that transactions won’t get written to the source without being at least received by one replica, the implication is that if your replica(s) are down (even just for maintenance), your application will block “forever”. This is a desired state if your most important requirement is not losing a single transaction, but still something that may not be entirely clear to administrators or developers when setting up semi-sync replication.

(Note that there is a way to avoid the forever-blocking nature of semi-sync using two replicas, and dynamic semi-sync flag management using orchestrator. I’ll talk about this further down.)

Promotion confusion due to incorrect replica count

The rpl_semi_sync_master_wait_for_slave_count config option controls how many replicas MySQL will wait for before acknowledging the transaction to the application. The default value is 1, meaning that even if you have two configured semi-sync replicas, MySQL will only wait for one of them to respond before assuming that things are fine.

Assuming our three host example above has rpl_semi_sync_master_wait_for_slave_count=1 set, we won’t know if B or C acknowledged the transaction. As long as the source (A) doesn’t crash this is not a problem, of course. If A crashes, however, our failover script (or our topology manager) has to decide which replica to fail over to and promote to be the new source: B or C.

Depending on the sophistication of the failover script or the topology manager, it is of course possible to figure out which replica has received the latest transactions (using GTIDs and/or the binary log position), but it makes the whole scenario much more complex.

For our setup, we have chosen to enable semi-sync on only one of the replicas and never promote the second asynchronous replica. This can be achieved with orchestrator’s brand new EnforceExactSemiSyncReplicas option (see below).

Re-using a failed source

Even though the semi-sync documentation clearly states that you should never re-use a failed source, I feel obligated to repeat it here, because the implications of re-using a failed source are quite significant.

Here’s what the docs say:

“With semisynchronous replication, if the source crashes and a failover to a replica is carried out, the failed source should not be reused as the replication source, andshould be discarded. It could have transactions that were not acknowledged by any replica, which were therefore not committed before the failover.“

This paragraph talks about the crash point 1 scenario from above. When a transaction is written to the binary log on the source (A), but it didn’t make it to B or C, this transaction officially never happened from the application perspective and from the perspective of B and C.

If you now re-use A (even as a replica of B or C), there will be a renegade transaction in A. In the best case, A won’t start up properly. In the worst case, you won’t notice for a while, but the state of source and replicas won’t be identical.

Bottom line: Don’t ever re-use a failed source. Rebuild it using xtrabackup from the promoted new source.

Automated failover is hard

While replication is part of MySQL’s core feature set, topology management and automated failover is not. What that means is that in the case a source host goes down, or has to be taken down for maintenance, MySQL won’t decide on a new source, change replication targets and it certainly won’t tell your application which of the replicas the new source host is.

To achieve an automated failover from a failed source to a replica, you have to employ the services of other tools. Unfortunately, the tool landscape for MySQL failover has changed over the years and it’s still a little bit of a wild wild west out there.

keepalived, VIPs and roll-your-own scripts: not for lossless failover!

For a very long time, we were using a failover mechanism that was based on keepalived, a floating virtual IP address (VIP) and some custom monitoring and failover scripts.

In this setup, the application talks to the MySQL source host via a floating IP address, which is controlled and managed by keepalived. Each MySQL host runs keepalived, which regularly runs a monitoring script that performs basic checks (1). If the health check of the MySQL source fails, it moves the VIP to the replica host that will become the new source (2) and then (!) triggers a notify script which will flip the host to read/write to become the new source (3).

This setup actually works quite well, requiring few moving pieces and no extra hosts. However, in our experience it is much less robust than a proxy based setup and not suitable if you require lossless failover with semi-sync. Here’s a (non-exhaustive) list of issues we ran into:

keepalived doesn’t know anything about MySQL. It just manages a VIP and isn’t aware of the MySQL topology. It doesn’t know who the source is, what the read_only flag is set to or if semi-sync is enabled. All of that has to be done by you in your own scripts. Not only is that a lot of work, it is also prone to error. You have to manually implement a mutex lock to make sure only one script runs at a time, you have to implement waiting for the relay logs to be processed, fencing, and you have to manually flip the read only and semi-sync state. Lots to do, lots to go wrong.

keepalived is decentralized. It works by all participating nodes communicating with one another. What that means is that if the communication between the nodes is (even momentarily) interrupted, there is a significant risk ofsplit-brain and with that, the risk that two nodes will try to grab the VIP and declare themselves the source with read_only=0. While the network layer will obviously ensure that only one host has the VIP, it could still flap back and forth between the hosts. The consequences of that are quite catastrophic, as it can lead to two MySQL hosts applying transactions. The perfect storm.

I could go on, but I’ll leave it at that. We had lots of trouble with this solution when we really started playing Chaos Monkey so I advise against this if you like your data.

HAproxy/ProxySQL, orchestrator, Consul and Consul Template

Luckily, there are a few popular setups out there that work quite nicely and don’t experience the above mentioned problems. Pretty much all of them revolve around using a combination of ProxySQL or HAproxy, orchestrator, Consul, and Consul Template. There are a number of great resources (4 links!) out there describing them, so I’ll be brief.

(Note that in this illustration, “proxy” can be HAproxy or ProxySQL. See below for details.)

All of the linked setups rely on monitoring the MySQL hosts for their health (responsiveness, replication, lag, …) using the topology manager orchestrator, and proxying SQL traffic through either HAproxy (TCP-level proxy) or ProxySQL (application-level proxy) to the current MySQL source host.

When orchestrator detects a failure on the source host (A), it first determines which of the replicas will become the new source, either B or C. Once it has figured that out, it repoints MySQL replication (stop slave; change master to …; start slave) of the remaining hosts and isolates the failed source (A). After that, it tells the proxy (HAproxy or ProxySQL) to repoint to a different host. In the most popular setup, this happens by dynamically updating the configuration through Consul and Consul Template.

Dangerous failover pitfalls

Even more so than setting up MySQL semi-sync replication, setting up failover as described above is quite tricky. There are lots of tools involved, and naturally each of them have tons of configuration options. Many things to do wrong.

Always wait for the relay log before accepting write traffic

Semi-sync replication guarantees the delivery of a transaction to the replica(s), but not the execution. It makes sure that the relay log on the replica(s) has a copy of every transaction, but still handles the execution asynchronously.

That means that it is very easy for MySQL replicas to fall behind on the execution of transactions and create a replication lag, if the SQL thread cannot execute transactions fast enough in the storage engine (7).

That’s why it’s called semi-sync replication, and not synchronous replication. (It sure would be nice if it was fully synchronous, though.)

If you develop your own failover solution, it is vital that you ensure that you fully process all existing relay logs on the replica that has been designated the new source before turning on write traffic, i.e. before setting read_only=0. If you don’t, you’ll start writing transactions to the storage engine out of order. In the best case scenario they will be unrelated, but most likely you’ll get pretty horrible duplicate key exceptions and will spend hours or days trying to repair the state.

When using orchestrator, waiting for the relay logs can be enabled by setting DelayMasterPromotionIfSQLThreadNotUpToDate=true. Since orchestrator manages the read-only state, you don’t have to worry about that separately.

Be sure to shoot the other node in the head

There are many different failure modes that may lead to the decision to fail over to a replica. For the sake of simplicity, we as engineers often only talk about a hard crash of a source and sometimes forget that temporary failures such as network blips, a hanging process or even just a service restart. Not considering temporary failures like this can be a huge mistake, because we always want to make sure that our application is only ever talking to the correct source, and not to the back-from-the-dead host that we failed over from.

When the decision is made to fail over from a failed source to a newly promoted replica, it is important to fence off the failed node from the application before appointing a replica to be the new source. This process is called Shoot The Other Node in the Head (STONITH).

Here’s an example of a failover with orchestrator that includes a STONITH pre-hook:

In this illustration, we see orchestrator has discovered a failure on the source (A) and decides to fail over to B. Using orchestrator’s pre-failover hook PreFailoverProcesses, we fence off A by pointing the proxy to “nowhere” while the failover is in progress. After orchestrator has completed the failover to the new source B (3), and repointed the other replica C to the new source (4), it updates the proxy configuration again (5) to send traffic to the new source (6).

Without the STONITH step (2), the proxy would point to the failed source until orchestrator is done with the failover. If the host comes back online in the meantime, you will have written to the wrong source host, and in the worst case you’ll lose those transactions.

Unfortunately, orchestrator does not support STONITH out of the box, so you’ll have to write your own pre-hook script to accomplish it. In HAproxy, you can simply update a listen block to point to 127.0.0.1:1337 (a non-existing target). In ProxySQL, you can achieve this by marking the host as OFFLINE as this blog post describes.

(Please note that the STONITH approach is not without controversy. You may read more in this blog post.)

Putting it all together

As of today, we’ve deployed our lossless semi-sync setup and the automated failover solution on hundreds of hosts, managing hundreds of millions of tables. Despite a few (quite severe) hiccups and the hundreds of hours we’ve spent optimizing and automating things, I’d say overall we’re pretty happy with the setup.

We’re using the exact setup I already talked about, namely HAproxy, orchestrator, Consul and Consul Template. I’ve uploaded a representative set of configuration files to GitHub, so you don’t have to start from zero when setting this up yourself.

Here are some details and important configurations:

MySQL:

We use Percona Server 8, which works pretty well, despite the extreme scale that we use it with. We’ve seen a number of horrible crashes related to too many replication restarts (which we fixed by telling orchestrator not to restart replication so often through a higher ReasonableReplicationLagSeconds setting), but other than that I’d say it’s been solid.

The semi-sync settings in our mysqld.cnf file match what was discussed above. Long source timeout to prevent async fallback, and semi-sync enabled by default:

loose-rpl_semi_sync_master_enabled              = 1
loose-rpl_semi_sync_master_timeout              = 3600000
loose-rpl_semi_sync_slave_enabled               = 1
loose-rpl_semi_sync_master_wait_for_slave_count = 1

It is worth noting that if you enable the semi-sync source setting on all hosts (including replicas) at startup, you may run into oddities in your stats. We solved this with a custom script (not in the repository) that disables the setting on replicas dynamically (rpl_semi_sync_master_enabled=0).

HAproxy:

After an unsuccessful attempt to use ProxySQL (it kept crashing constantly), we decided to use HAproxy for proxying SQL traffic to the currently active source host(s). The main haproxy.cfg file is pretty basic and not really worth talking about.

The mysql.cfg file, however, contains listen blocks that route to the currently active source hosts. This file is autogenerated and kept up-to-date by Consul Template using the template file haproxy_mysql.cfg.tpl. The template will generate a listen block like this for every MySQL source host:

listen mysql-g0
   bind 10.0.13.1:3400
   server db-g0-1.example.com 10.0.14.1:3306
   mode tcp
   option tcplog

It also implements one part of our STONITH approach: If the Consul key mysql/master/$cluster/failed exists, it will black-hole all traffic to this cluster by pointing it to 127.0.0.1:1337, a non-existing host.

Consul Template:

The template above is updated and re-rendered by Consul Template using the config file haproxy.hcl. After it re-renders the file, we reload HAproxy to refresh the configuration.

In our real production environment, we don’t just call out to systemctl reload haproxy. Instead we have a tiny Python script (not included in the repository) that kills all active connections to the cluster and then reloads HAproxy.

Consul:

Consul is the single source of truth for which MySQL host(s) are the source. The installation is pretty straightforward (see configuration files).

orchestrator:

We run orchestrator in a cluster of three with a MySQL backend and with Raft support (see configuration files).

Most importantly, we enabled DelayMasterPromotionIfSQLThreadNotUpToDate=true, which makes sure that orchestrator waits for the relay logs before promoting a replica and turning read-write on. See above for a detailed discussion on this.

We also enabled the brand new EnforceExactSemiSyncReplica=true setting (which I am very proud to have contributed to orchestrator), which completely manages the semi-sync replica state: with this setting, orchestrator will enforce the correct semi-sync flag on the replicas during failover (i.e. enabling and disabling it), matching the wait count set in rpl_semi_sync_master_wait_for_slave_count. For us this is especially important, because we’d like to only ever have one semi-sync replica, even though we have two replicas, so that we know which host to fail over to, and so can do maintenance or one of them can crash.

Looking at the PreFailoverProcesses in the orchestrator.json file, you can see that we’re calling a script called orchestrator-pre-failover (not included in the repository). This script sets the mysql/master/$cluster/failed key, which triggers Consul Template to update the HAproxy config and back-hole the cluster for new traffic. This is the other side of the STONITH mechanism I talked about above.

Wrapping it up

As you can tell from the length of this post, our journey to lossless MySQL replication and failover has been quite interesting and we’ve learned a lot. If you’ve read the whole thing, I applaud you and thank you for staying with me.

In this post, I discussed all the technologies and tools that are necessary to provide a MySQL replication environment with lossless automated failover. I compared async replication to semi-sync replication and discussed some gotchas. I then presented different approaches for automated failover and highlighted the pieces that are relevant for a lossless setup, concluding in a section showing our own fully functional setup.

MySQL replication and failover is complicated. It’s really easy to mess it up and lose data. I hope this post helped prevent data loss in your organization.

Despite its incredible features though, it has its rough edges. And no, I don’t mean the horrific query language (honestly, who thought that was a good idea?). I mean the fact that without external tools it’s quite impossible to import, export, copy, move or re-shard an Elasticsearch index. Indices are very final, unfortunately.

This is quite often very inconvenient if you have a growing index for which each Elasticsearch shard is outgrowing its recommended size (2 billion documents). Or even if you have the opposite problem: if you have an ES cluster that has too many shards (~800 shards per host is the recommendation I think), because you have too many indices.

This is why I wrote elastictl: elastictl is a simple tool to import/export Elasticsearch indices into a file, and/or reshard an index. In this short post, I’ll show a few examples of how it can be used.

Usage

elastictl can be used for:

• Backup/restore of an Elasticsearch index
• Performance test an Elasticsearch cluster (import with high concurrency, see --workers)
• Change the shard/replica count of an index (see elastictl reshard command)

It’s a tiny utility, so don’t expect too much, but it’s helped our work quite a bit. It allows you to easily copy an index, or move it, or test the index concurrency supported by your cluster. In my local cluster, I was able to import ~10k documents per second.

Here’s a short usage overview:

$ elastictl
NAME:
   elastictl - Elasticsearch toolkit

USAGE:
   elastictl COMMAND [OPTION..] [ARG..]

COMMANDS:
   export, e   Export an entire index to STDOUT
   import, i   Write to ES index from STDIN
   reshard, r  Reshard index using different shard/replica counts

Try 'elastictl COMMAND --help' for more information.

elastictl 0.0.5 (e645803), runtime go1.16, built at 2021-04-14T15:05:42Z
Copyright (C) 2021 Philipp C. Heckel, distributed under the Apache License 2.0

Export/dump an index to a file

To back up an index into a file, including its mapping and all the documents, you can use the elastictl export command. It will write out JSON to STDOUT. The file format is pretty simple: the first line of the output format is the mapping, the rest are the documents. You can even export only a subset of an index using the --search/-q option (that is, if you can master the query language).

# Entire index (assumes that ES is running at localhost:9200)
elastictl export my-index | gzip > my-index.json.gz

# Only a subset of documents
elastictl export \
  --host 10.0.1.2:9200 \
  --search '{"query":{"bool":{"must_not":{"match":{"eventType":"Success"}}}}}' \
  my-index > my-index-no-successes.json

If you’re wondering “isn’t this just like elasticdump”? The answer is yes and no. I naturally tried elasticdump first, but it didn’t really work for me: I had issues installing it via npm, and it was quite frankly rather slow. elasticdump also doesn’t support resharding, though it has many other cool features.

Import to new index

The elastictl import command will read from STDIN and write a previously exported file to a new or existing index with configurable concurrency. Using a high number of --workers, you can really hammer the ES cluster. It’s actually quite easy to make even large clusters fall over like this (assuming of course that you’re pointing --host to a load balancer):

# With high concurrency
zcat my-index.json.gz | elastictl import --workers 100 my-index-copy

# Just copy and index
elastictl export my-index | elastictl import my-index-copy2

There are other options you can pass to the elastictl import command to modify the mapping slightly (mostly the number of replicas and the number of shards):

$ elastictl import --help
NAME:
   elastictl import - Write to ES index from STDIN

USAGE:
   elastictl import INDEX

OPTIONS:
   --host value, -H value      override default host (default: localhost:9200)
   --workers value, -w value   number of concurrent workers (default: 50)
   --shards value, -s value    override the number of shards on index creation (default: no change)
   --replicas value, -r value  override the number of replicas on index creation (default: no change)
   --no-create, -N             do not create index (default: false)
   --help, -h                  show help (default: false)

Re-shard an index

The elastictl reshard command is a combination of the two above commands: it first exports an index into a file and then re-imports it with a different number of shards and/or replicas.

# Set number of shards of the "my-index" index to 10 and the number of replicas to 1
elastictl reshard \
  --shards 10 \
  --replicas 1 \
  my-index

# Export a subset of the "my-index" index and re-import it with a smaller number of shards/replicas
elastictl reshard \
  --search '{"query":{"bool":{"must_not":{"match":{"eventType":"Success"}}}}}' \
  --shards 1 \
  --replicas 1 \
  my-index

Note: Similar to the _reindex API in Elasticsearch, this command should be used while the index is not being written to, because documents coming in after the command was kicked off will otherwise be lost. Please also note that the command does DELETE the index after exporting it. A copy will be available on disk though.

Feedback is welcome

elastictl is a tiny little tool, and I’m sure there are others that do a similar job. The tool is open source and available under the Apache 2.0 license, so please feel free to send contributions via pull a request on GitHub.

However, the tool that I wrote, a recursive search/replace tool which I intelligently dubbed re, is actually incredibly useful: to my own surprise, I use it every day. I haven’t made a single modification to it in all that time (until today for this post). And since I’m in the sharing mood today, I thought I’d share it with the millions of people (cough) that come here every day. Ha!

Why is there no recursive sed out there?

It’s a little odd, isn’t it? When you search for recursive search replace linux in your favorite search engine, you get all sorts of find | xarg sed-type solutions (like here), or find -exec sed ... (like here). And yes, of course, that’s the Linux way: combine the tools that you already have to do what you want. Still, searching and replacing seems like a pretty basic thing to me.

So anyway, for some reason, there is no standard Linux utility that can do it out of the box – something where you can just do re oldvalue newvalue to replace and old value with a new value in all files in the current folder (and all its subfolders).

Introducing the fabulous re

Well you’re in luck. That’s what re is. A tool that can recursively replace strings in one or many directories. By default, it will walk the entire directory tree and replace the search string in all files recursively. You can provide include/exclude wildcards for the filename, e.g. to only replace strings in “*.java” or “.sh” files.

Here’s the basic syntax:

Syntax: re [options] SEARCH REPLACEMENT [DIR ...]
Options:
  -e string
    	Comma-separated list of excluded files, wildcards supported (default ".bzr,CVS,.git,.hg,.svn")
  -f	Apply changes
  -i string
    	Comma-separated list of included files, wildcards supported, e.g "*.js,*.html,*index.*"

And here are a few examples of how I used it in the past (straight from my history):

# Shows files that would have been replaced. Without the -f flag, 
# nothing is actually replaced.
$ re CacheDir ClipboardDir

# Replace CacheDir with ClipboardDir recursively
$ re -f CacheDir ClipboardDir

# Replace "Foo(Bar)" with "Foo(Baz)" recursively in all scala files
$ re -f -i "*.scala" "Foo(Bar)" "Foo(Baz)" 

Here’s what the output will look like. In this example, I’m not passing the -f flag, so no change will be applied:

$ re -i "*.go" var const
No changes will be applied unless -f is given.
Skipping excluded directory .git
+ LEGACY/client.go
+ LEGACY/udp_client.go
+ LEGACY/udp_server.go
+ build/lib/_obj/_cgo_gotypes.go
+ client_conn.go
+ client_forward.go
+ cmd/natter/main.go
+ internal/natter.pb.go
+ protocol.go
9 file(s) WOULD have been updated.

You get the idea. It’s simple. That’s it.

If you like it, you can get it from the Github page. Either build it yourself (go build) or check out the releases page.

A. About this post

This post is super short because it’s just a Code Snippet. You can find other short, code-focused posts like this in that section.

Well today I am changing that. I like writing, but not necessarily so much that I always want to write a super long post. Sometimes, things should be short. Like this one.

So in this super short post I’m gonna show you a cool thing I figured out: How to calculate the the value that curls --pinnedpubkey option needs in Go.

curl’s --insecure flag is insecure – shocker!

For my tiny side project pcopy, I wanted to be able to be able to allow people to easily install/join a remote clipboard using a curl | sh-type install, even when the shared clipboard is internal to a company network and doesn’t have a proper SSL certificate, i.e. when the cert is self-signed.

The only way to make curl work with self-signed certs is with the -k flag (--insecure):

$ pcopy invite
# Instructions for clipboard 'default'

# Install pcopy on other computers (as root):
curl -sSLk https://10.0.160.67:1986/install | sudo sh

# Join this clipboard on other computers:
curl -sSLk https://10.0.160.67:1986/join | sh

And that’s obviously not cool, because that flag allows replacing the cert entirely in man-in-the-middle attacks (see my post on mitmproxy for a practical example; wow that post is from 7 years ago – I’m telling you time flies …).

--pinnedpubkey to the rescue

So what to do, what to do? Should we just not care about that? No of course not! I care deeply about security, so let’s figure out if we can use public key pinning in curl (note that this link points to “HTTP public key pinning” not the concept of public key pinning in general; there is strangely no Wikipedia article about that).

Public key pinning allows us to pin a specific public key for a given request, so that even if a certificate is self-signed it can’t be replaced without raising an exception. This technique is pretty useful if you want to support self-signed certs, but still be secure.

And surely enough, curl has a --pinnedpubkey option. From the man page:

--pinnedpubkey <hashes>
  (TLS)  Tells  curl to use the specified public key file (or hashes) to verify the peer. 
  This can be a path to a file which contains a single public key in PEM or DER format, 
  or any number of base64 encoded sha256 hashes preceded by ´sha256//´ and separated by ´;´

  When negotiating a TLS or SSL connection, the server sends a certificate indicating its
  identity. A public key is extracted from this certificate and if it does not exactly match
  the public key provided to this option, curl will abort the connection before sending or
  receiving any data.
</hashes>

Now this description wasn’t really helpful when I was trying to figure out what exactly curl was expecting here. Public keys can be encoded as PEM or DER, and then there’s also PKIX and PKCS1. Plus somehow ASN.1 is involved in the whole thing. It’s all pretty confusing, and of course, none of this is mentioned in the tiny man page entry. So it took a while to figure it out.

So with the help of the curl docs on CURLOPT_PINNEDPUBLICKEY and the lovely #security channel on the Go Slack, I figured out that curl is expecting the base64-encoded SHA-256 checksum of the PKIX, ASN.1 DER encoded public key. That’s a mouthful, isn’t it?

In bash (using openssl), that looks like this:

# Assuming default.crt is a PEM-encoded cert, this extracts the public key
# converts it to DER form, hashes it with SHA-256, then base64-encodes it
# and prepends "sha256//"

echo sha256//$(openssl x509 -in default.crt -pubkey -noout \
   | openssl asn1parse -inform PEM -in - -noout -out - \
   | openssl dgst -sha256 -binary - \
   | openssl base64)

# Outputs something like sha256//Y/CGGnkaoZwUgOqArQs12llyoaX0bkjSIgHCPtXba+c=

In Go, it looks like this:

func calculatePublicKeyHashes(certs []*x509.Certificate) ([]string, error) {
  hashes := make([]string, len(certs))

  for i, cert := range certs {
    derCert, err := x509.MarshalPKIXPublicKey(cert.PublicKey)
    if err != nil {
      return nil, err
    }
    hash := sha256.New()
    hash.Write(derCert)
    hashes[i] = fmt.Sprintf("sha256//%s", base64.StdEncoding.EncodeToString(hash.Sum(nil)))
  }

  return hashes, nil
}

It took me a while to figure out that I needed to use x509.MarshalPKIXPublicKey (PKIX, ASN.1 DER form), and not x509.MarshalPKCS1PublicKey (PKCS#1, ASN.1 DER form). Apparently, the PKIX format also includes the public key algorithm (RSA, EC), and not just the raw bytes, and curl expects this form. There is a good explanation of Stack Overflow.

Long story short, now pcopy invite (see source code) can output a secure curl command, even for self-signed certs:

$ pcopy invite
# Instructions for clipboard 'default'

# Install pcopy on other computers (as root):
curl -sSLk --pinnedpubkey sha256//Y/CGGnkaoZwUgOqArQs12llyoaX0bkjSIgHCPtXba+c= https://10.0.160.67:1986/install | sudo sh

# Join this clipboard on other computers:
curl -sSLk --pinnedpubkey sha256//Y/CGGnkaoZwUgOqArQs12llyoaX0bkjSIgHCPtXba+c= https://10.0.160.67:1986/join | sh

Note that despite the -k flag still being there, the command cannot be intercepted without curl erroring, because the pinned public key hash won’t match. However, removing the flag will make curl complain with curl: (60) SSL certificate problem: self signed certificate. All that means is that the --pinnedpubkey verification happens after the self-signed cert verification.

I know I’m being unfair, because more often than not, rebooting Ubuntu actually completes successfully. However, my incredibly unscientific estimate of how often things get stuck forever on shutdown or reboot is this: 1-3%. That’s how often I believe reboots hang. That’s shockingly high, right? Well, I pulled that out of my hat, but that estimate is based on many hundred thousands of reboots I’ve witnessed in our fleet of backup devices. That number is not too terrible when you deal with a handful of machines that you rarely ever reboot. It is, however, incredibly terrible if you reboot tens of thousands of devices running Ubuntu every two weeks as part of an upgrade process (I wrote about our image based upgrade mechanism in another post).

This post describes the short story of how we managed to make Ubuntu machines reliably reboot.

1. What’s the problem?

When we first rolled out Image Based Upgrades, we encountered tons of stuck reboots. The first few thousand device reboots were terrifying; so many devices just didn’t come back up. For our customers, that meant driving on-site or using a remote power strip – both of which are annoying and costly.

Either things didn’t even begin the reboot process, like here:

Or they got stuck at random points during the shutdown procedure:

Think about how hard it is to debug shutdown problems like this when you have IPMI access in your own datacenter. Now imagine how hard it is to debug without physical access to the machines, and having to rely on the good grace of your own customers and support team to work with you. We all worked very closely together with our customers, taking hundreds of pictures of hung machines trying to pinpoint the root cause. We spent days trying to reproduce the problems in-house, upgrading devices over and over again many times a day to force the issue. We analyzed ticket after ticket, but in the end we were none the wiser: Ubuntu seemed to have gotten stuck just at random positions in the shutdown process. There was no pattern.

So what now? Give up? Of course not. We didn’t give up. We just merely chose to give up diagnosing the root cause and opted for the nuclear option instead: hard resetting.

2. How about Sysrq?

We first thought about using the Sysrq triggers that the Linux kernel provides to immediately trigger a hard-reset. Instead of calling reboot, we’d call this magical command to instruct the kernel to hard-reset the computer.

echo b > /proc/sysrq-trigger   # Trigger hard-reset

This works wonderfully and actually is a viable option if you are 100% sure that you’ve written everything to disk that you need to, and that all the buffers are flushed to disk. So if you run a read-only Linux, this option may be for you. In all other cases, using just the echo b trigger is not enough, because you run the risk of corrupting random files, your file system, or other in-flight operations.

So what about this then?

echo s > /proc/sysrq-trigger   # Sync all mounted filesystems to disk
echo b > /proc/sysrq-trigger   # Trigger hard-reset

Well, yes, that’s better, because it flushes all the buffers to disk before pulling the cord. The chance of losing or corrupting data is very slim with this one. It’s still pretty nuclear if you ask me, and doesn’t make me feel warm and fuzzy inside. And as it turns out, echo s can hang indefinitely (much like sync) if the hardware doesn’t respond – the infamous uninterruptible sleep (D) strikes again. (Side note: I am far from an expert in this field, but the amount of times TASK_UNINTERRUPTIBLE has bitten us left and right leaves me to believe that there is something wrong with having a state like that in the kernel at all. That’s a story for another time, though.)

So because echo s can hang, we briefly experimented with something like this:

timeout 10s bash -c ‘echo s > /proc/sysrq-trigger’  # Wait max 10s for sync
echo b > /proc/sysrq-trigger 

We tested this for a while, but saw enough corrupt file systems (despite the echo s) that it scared us away from this approach. It also felt wrong to punish the 98% of devices that were rebooting just fine using reboot. What we really needed was a reboot mechanism that first tried to run a proper reboot, and only triggered a hard-reset when things got stuck for too long.

We reviewed how reboot is actually implemented and then thought about implementing our own reboot mechanism with a hard-reset fallback. How hard can it be? It’s just sending SIGTERM to all processes, and then waiting a while for things to gracefully shut down before triggering a hard-reset, right?

Before you raise your eyebrows too much, I’m kidding. We realized pretty quickly that if Ubuntu can’t get it right, we’d probably have a hard time too. So on to the next idea.

3. Introducing watchdogs

The Linux kernel provides a watchdog API whose entire purpose in life is to hard-reset systems that get stuck or hang due to unrecoverable errors such as a kernel panic. Turns out that many modern CPUs and some IPMI/BMC systems ship with a hardware-level watchdog implementation that hard-resets the host system unless a heartbeat is received regularly. When a watchdog is enabled, and a system freezes or otherwise hangs so that it cannot send a heartbeat, the watchdog resets the computer.

This is exactly what we needed!

Even though watchdogs certainly aren’t meant to be used to ensure a proper reboot, they absolutely can be used for what we need: enable a hardware-level watchdog that hard resets the machine in case the soft reboot via reboot fails.

There are multiple watchdog implementations, each of which comes with its own quirks. We have found the Intel CPU-basediTCO watchdog to be the most reliable, so we use that as a default. Unfortunately, some motherboard vendors disable this functionality in hardware. In that case, and on non-Intel devices, we have two fallback options:

• IPMI/BMC based watchdog: aside from the iTCO watchdog, we have found the ipmi_watchdog kernel module to be very reliable. Since it’s IPMI-based, you obviously need to have a BMC that supports it and make sure that it actually works. We have found some mainboards have faulty watchdogs, so beware of that and do extensive testing before using it.

• Software-based watchdog: the kernel provides a softdog module, which can be used if no hardware supported watchdog is available. The softdog is obviously not as reliable because it relies on the kernel to not have crashed entirely. It’s better than nothing though, that’s for sure.

The watchdog driver provides a /dev/watchdog device that needs to be written to every X seconds. If nothing writes to it until it times out, the system is hard reset. Usually some process holds a file handle open and regularly writes to it to pet the watchdog (there is a watchdog daemon for this purpose). In our case, we simply want to ensure that a hard reset is triggered after X seconds, so all we do is pet the watchdog once (open file handle, and close it) before calling reboot.

The timeout is specified via the heartbeat=.. (iTCO), timeout= (IPMI) or soft_margin=.. (softdog) parameters when loading the module via modprobe. The nowayout=1 parameter is necessary to ensure that the watchdog timer cannot be stopped by any means once it is started.

4. Reliably rebooting with watchdogs

In short, here’s what we do in our BCDR devices to reliably reboot them as part of the OS upgrade:

• Load watchdog kernel module with a 10-minute timeout

• Access the watchdog device to start the heartbeat timer

• Sync, sync, sync the root partition (I know we don’t have to …)

• Reboot

If any of the steps after loading the watchdog module hangs or fails, the watchdog will trigger a hard-reset after 10 minutes. The actual code we use to execute these steps is part of our image-based upgrade tool upgradectl. Here’s a shorter version in Bash that is obviously not as well-tested, but it should work:

#!/bin/bash

WATCHDOGS=(
  "iTCO_wdt nowayout=1 heartbeat=600"
  "ipmi_watchdog nowayout=1 action=reset timeout=600"
  "softdog nowayout=1 soft_margin=600"
)
BAD_BOARDS=(
  "A1SRi" "X9DRD-7LN4F" "X9DBL-3F/X9DBL-iF" "X9SRE/X9SRE-3F/X9SRi/X9SRi-3F"
  "X10SLH-F/X10SLM+F" "X10SLH-F/X10SLM+-F" "X10SLM-F" "X11SSH-F" "DCS" "0PM2CW"
)

board=$(dmidecode -s baseboard-product-name | head -n 1)

for watchdog in "${WATCHDOGS[@]}"; do
  if [[ $watchdog =~ "ipmi" && " ${BAD_BOARDS[@]} " =~ " ${board} " ]]; then
    echo "Skipping IPMI watchdog due to bad board $board"
    continue
  fi
 
  echo "Attempting to load watchdog $watchdog"
  modprobe $watchdog
  udevadm settle
  sleep 1
  if [ -e /dev/watchdog ]; then
    echo "Starting watchdog timer"   
    touch /dev/watchdog
    break
  fi
done

rootpart=$(awk -v needle="/" '$2==needle {print $1}' /proc/mounts | grep -v ^rootfs)
echo "Syncing root partition $rootpart"
for i in 1 2 3; do timeout 60s blockdev --flushbufs $rootpart; done

echo "Waiting 25 seconds"
sleep 25

echo "Reboot"
reboot

It’s not the prettiest script in the world, and yes, it’s got some magical sleeps in there that we would have rather avoided. But we have found that there are race conditions and slow background tasks that are difficult to pinpoint, so sleep is the next best thing. The sleep after loading the watchdog module is to ensure that the watchdog device has time to be put in place by the kernel. The 25 second sleep after the triple-sync is because we have found file corruption on the root disk in the field without it. It’s ugly, but it works.

You may have also noticed some special logic for the IPMI watchdog: as it turns out, quite a few mainboards have either bad or faulty IPMI watchdog implementations. We have found boards that would continuously reboot devices (devices were stuck in reboot loops, so much fun …), or watchdogs that simply didn’t behave like they were supposed to. For that reason we’ve chosen not to use the IPMI watchdog for these boards.

Another thing worth noting is that if you already have a watchdog daemon running, this script will likely need some changes: instead of loading the watchdog module, you may want to reconfigure it. For IPMI, you can do this by writing to /sys/module/ipmi_watchdog/parameters, and I’m sure the other modules have similar toggles.

5. Verdict

Rebooting reliably is more complicated than I would have ever imagined.

I don’t think the option we have chosen here is the best there is, but it has been working reliably for over 3 years. Our tens of thousands of appliances upgrade and reboot every 2-3 weeks, and the number of devices that don’t come back from a reboot is incredibly low – much lower than before we started using the watchdog-assisted reboots. Nowadays, if a device doesn’t come back from a reboot, it’s typically because of bad hardware (e.g., bad board or bad RAM), and not because it got stuck in the shutdown process somewhere.

So all in all, I’d say we’re pretty happy with this approach, and that extra special thrill that I was talking about when we reboot our devices has definitely gone away.

Fortunately we’ve always provided a remote management feature called “Remote Web” for our customers: Remote Web lets them remotely access the device’s web interface as well as other services (mainly RDP, VNC, SSH), even when the device is behind a NAT.

Internally we call this feature RLY (pronounced: “relay”, like the owl, get it?). In this post, I’d like to talk about how we implemented the feature, what challenges we faced and what lessons we learned.

Why access devices remotely?

Not having to drive to a site and physically manage a device is as important for our customers as it is for us:

When a customer calls with an issue on a device, our support team needs to be able to investigate what’s wrong. They need to be able to SSH into the device, monitor backups and change the configs. Being able to do that is vital for providing great customer support. Similarly, our customers want to be able to do the same if they get calls from their customers, or to simply set up and manage the appliance.

The primary use case for MSPs is accessing the device’s web interface. However, other use cases for this feature are not as obvious: The same technology that provides a remote web interface in the browser is also used in many small features. For instance, when a restore VM needs to be accessed from outside the LAN, we provide a way to do that with the “Relay”:

Reverse tunnels for the win!

So what is this magical technology that fuels these wonderful features? I’m sorry to disappoint, but it’s just plain old reverse tunnels using SSH. In short, it’s this: $ ssh -R

If you are now incredibly disappointed because you expected more, don’t be! We’ll get into interesting things about scaling this service to tens of thousands of devices.

In case you’re not familiar with what ssh -R does, let me briefly remind you:

ssh -R creates a reverse tunnel by connecting to a relay server, binding a port on that server and forwarding all incoming traffic on that port to the initiating client (in our case: the appliance):

It’s a port forwarding mechanism that typically breaks through firewalls and NATs because it is initiated by the client that provides the service. In the picture above, the service being provided listens on port 80/HTTP – the web interface on the appliance. Just like TLS/SSL wraps HTTP (= HTTPS), using SSH has a similar effect here: We’re encrypting the HTTP traffic over the secure SSH channel.

When instructed to forward port 80, the device creates a reverse tunnel using ssh -R as follows (step 1):

mydevice> ssh \
   -R 12345:localhost:80 \
   -o ConnectTimeout=10 \
   -o IdentityFile=... \
   -o ExitOnForwardFailure=yes \
   -o ServerAliveInterval=15 \
   -o ServerAliveCountMax=2 \
   -vvv \
   -N \
   ... \
   user@rly.company.com

Assuming a valid private key is provided, the SSH daemon on the relay server will bind the port 12345 and forward all incoming traffic on from rly.company.com:12345 to port 80 on the device. So a user could browse to http://rly.company.com:12345, and they would see this particular device’s web interface.

While that will work, there are two pretty obvious problems with this:

• First, the connection to the relay server is done over HTTP (as seen in the URL http://…), which these days is not acceptable under any circumstances.
• And second, given that there are only 65k ports, it is incredibly easy to discover valid URLs by just counting up.

Both of these can be easily fixed by putting a proxy server in between the SSH tunnel and the browser client. Equipped with a valid SSL certificate and a little bit of rewrite magic, it’s easy to turn http://rly.company.com:12345 into https://aabbccddeeff.rly.company.com. We use Apache for this (shortened excerpt):

<VirtualHost *:443>
  # Set up SSL
  SSLEngine on
  SSLCertificateFile      "/etc/apache2/ssl/rly.company.com.crt"
  SSLCertificateKeyFile   "/etc/apache2/ssl/rly.company.com.key"
  SSLCertificateChainFile "/etc/apache2/ssl/rly.company.com.chain.crt”

  # Set up rewriting & proxying
  # e.g. translates "aabbccddeeff.rly.company.com" to "localhost:12345"
  RewriteEngine on
  RewriteMap relaymap "prg:/usr/local/bin/relayrewrite"

  <Directory "/var/www">
    RewriteRule ^ ${relaymap:%{REMOTE_ADDR}:%{REQUEST_SCHEME}://%{SERVER_NAME}%{REQUEST_URI}} [L,P]
  </Directory>
</VirtualHost>

To solve our first problem (HTTP), we listen on 443 and provide the proper wildcard certificates for our domain (here: *.rly.company.com). This will terminate SSL at the relay host, meaning all requests will be handled by Apache. That takes care of problem one.

In order to solve the second problem, we have to forward the request to the correct SSH connection (in our example the SSH daemon is bound to *:12345 on the relay host). To achieve that, we can use the RewriteMap stanza in Apache’s mod_rewrite module. Using the prg: option, each request can be rewritten dynamically by an external program (here: /usr/local/bin/relayrewrite). This program looks up the hostname, maps it to the correct SSH connection and finally rewrites the request. In the example above we are effectively proxying requests originally targeted to aabbccddeeff.rly.company.com to 127.0.0.1:12345, thereby connecting the Apache request to the SSH tunnel.

That’s it. Simple right?

Horizontally scaling reverse tunnels

Believe it or not it actually is that simple. For a very long time, we’ve pretty much done this without a problem for thousands of connections. For the first few years of my company’s existence, we had exactly one server that handled all of our reverse tunnel relay traffic. Occasionally we added RAM or upgraded to a bigger box, but eventually, as it grew, we needed a more reliable balanced system without a single point of failure. And of course, as we grew internationally, our partners in Europe and Australia started complaining about abysmal speeds. No wonder since traffic was bounced around the globe twice for every packet.

On top of that, we were facing other issues with the existing system, such as slow connection times, limited connection lifecycle management (no repairs of broken connections) as well as odd outgoing port requirements for our devices.
Requirements

Faced with these growing issues, we decided to take a step back and rethink how to scale the application from the ground up. We identified the following requirements:

• Horizontal scaling & load balancing, no single point of failure
• Allow maintenance of any component without customer impact
• All outgoing connections must work on port 80/443
• Connection time of under 5 seconds
• Geographically distributed relay servers to address slow international connections
• Auditable SSH sessions for support staff
• Time-limited key-based authentication/authorization
• Automated deployment via Puppet

The most obvious (and arguably the hardest) problem to solve is the first one in the list, i.e. avoiding a single point of failure. If you’ve ever built a distributed system you know what I’m talking about. Keeping things in sync is hard if you don’t have a single point in the application to keep state. But as with many things in computing, the hardest problems are also the most interesting problems to solve.

After a little bit of weekend-tinkering followed by many weeks of hard work, we came up with version two of our remote management system which we called RLY.

Introducing RLY

RLY is a highly scalable, distributed, high availability relay server application. While that certainly sounds impressive, at its core it still uses SSH tunnels to forward traffic between systems. The things we added along the way were merely to meet our scaling and deployment requirements.

To accomplish that, RLY is split into four components that closely interact with one another:

The RLY client is a command line client used by the behind-a-NAT device (here: our SIRIS/ALTO device). It provides a daemon waiting for incoming connections and a CLI to manually open time-limited tunnels if need be (rly open|close|...).

The tracker component is the brain of the system. It currently consists of a handful of physical servers in one of our US data centers. The trackers are the central hub to manage connections. They keep track of what connections are open, what ports to forward to and which relay servers to use. They also generate per-connection keypairs which are used to access a relay server. The component has a load-balanced API that is used by our portals and the jump hosts to open new connections as well as by the client devices to check for new connections. The tracker keeps state in a tiny Cassandra cluster.

The relay component is the one doing the heavy lifting. It consists of dozens of virtual machines in different data centers around the globe, close to where our partners devices are. Its main responsibility is to forward traffic from an end user to a behind-the-NAT device (via reverse tunnels as described above). Each relay host is isolated from its peers. The tracker is the only other component that talks to its API.

The jump component provides interactive SSH sessions to our devices for our support team and partners. Using one of these jump hosts, it is possible to SSH into a behind-the-NAT device by forwarding traffic to a local SSH daemon. The jump hosts are also located close to our partners and support teams to allow for a snappy SSH session. For audit purposes, all hosts monitor all interactive sessions via script.

Opening a connection

So how do the component work together? Let’s look at a simple example: Assume a partner wants to access their device’s web interface. Here’s a brief sequence diagram that shows the communication between the systems:

• When the user clicks on the “connect” button in our partner portal, the portal backend requests a new reverse tunnel to port 80 from one of the trackers. The tracker creates a new connection ID and subdomain, generates an SSH keypair and intelligently picks the relay host. The selection is based on the GeoIP-proximity between client and relay host IP addresses and load of the relay servers. This evenly spreads the load while ensuring connection speed.
• The tracker then tells the relay host to grant access to the generated keypair. The relay host saves this connection information, assigns a random relay port and waits for the incoming SSH connection.
• The client regularly checks for changes using a heartbeat mechanism that will trigger a checkin call if there are any changes. Checkin returns the new connection information, including the relay host, port and the ephemeral private key needed for auth.
• It then uses this information to open the reverse tunnel.

Now that the connection is open, end users can access the local web interface on the client via the browser, e.g. at ttps://xhaz32dhjmndf.rly.company.com.

I’d love to talk about how the dynamic DNS is implemented (hint: it’s similar to what I described in this post, but with a Cassandra backend), but that would probably blow up this post even more. In short: The trackers serve as authoritative name servers for the rly.company.com zone and respond with a CNAME pointing to one of the relay hosts when asked. The relay host then simply applies the same rewrite magic I described above to map the hostname to the connection-specific local port

Scalability and resilience

Using this architecture, scaling the system is really easy, because it merely consists of adding tracker hosts or relay hosts and making them known in the configuration. Adding relay hosts in particular is incredibly simple, because it just means deploying a new VM and making it known to the trackers. Adding a tracker, on the other hand, is a little more manual work, because it means adding a node to Cassandra and making sure the consistency guarantees still work out as expected. Since the clients use SRV DNS records to discover trackers, they will eventually pick up the new trackers and add them to their round-robin list.

So it certainly looks like the system scales, but does it fall over if parts of it fail? Let’s discuss some scenarios to show what would happen if individual components/servers fail:

Tracker host failure: If a tracker host goes down (or is taken down for maintenance), the Cassandra backend will continue to work, because it was configured with quorum read/write consistency, meaning as long as a majority of the cluster is alive, it will remain up and functioning. The RLY clients typically round-robin through the trackers so they’ll just mark it as failed and try again a couple of minutes later. So yeyy! We can survive a tracker going down. But what if more than half of them fail? As with any quorum based system, if more than half of the hosts go down, then we’re out of luck. In our case, that’d only happen if there is a row-wide outage in the data center. That’s a choice we’ve made consciously and a risk we’ve accepted.

Relay host failure: If a relay host fails, all SSH tunnels to that host break. While that is unfortunate, it only affects a small subset of tunnels and is automatically repaired within a couple of seconds: If a client detects that a tunnel broke it tries to re-establish the connection a couple of times (to account for connection blips). If that doesn’t work, e.g. because the host is down for good, it will ask the tracker to repair the connection. The tracker will assign a new relay host and the connection is up again.

Lessons learned

Developing RLY was a learning process. We kept the things that worked for us in the past, and we tried different ways to overcome the problems we had. RLY works and we’re proud of it, but we’ve learned many things along the way. In this section, we’d like to share some of our learnings.

Please take what we learned with a grain of salt: The fact that something didn’t work for us doesn’t mean it’s wrong or bad. It merely means that we either did it wrong, or that we used the wrong tool.

Distributed datastores across continents

RLY started as a pet project of mine. It was meant to work on a single host as well as on many hosts as a distributed system. Once we started experimenting with distributed datastores as a backend for the RLY trackers, we learned quite a few things really quickly.

In the beginning I had this idea that I could split the datastore that manages connections across different data centers and different continents – you know, for maximum fault tolerance and resiliency. I started with etcd as a backend store and three trackers in Frankfurt, Sydney and on the east coast of the US. I absolutely loved etcd. It was simple and exactly what I needed. On top of that it worked beautifully in my tests. Once I deployed it in my three tracker test setup though I realized pretty quickly that it wasn’t going to work. The latency was horrendous, and even with a lot of tweaking etcd couldn’t keep up with the amount of new/closing connections of RLY. I know what you are thinking: Why would anyone try etcd across continents? Honestly I don’t know what I was thinking. It’s pretty obvious now that that’s a dumb idea. Lesson learned!

After abandoning the idea of a datastore across data centers, we experimented with GlusterFS, Redis Cluster and a few others, but ultimately settled on Cassandra. In my test setup, Gluster was not dealing well with lots of small file changes. I bet there’s someone out there that knows how to tune it, but I gave up pretty quickly. Redis Cluster support was still pretty new at the time, and while it worked nicely I wasn’t happy using a brand new technology for a vital service like this.

We ended up with Cassandra in one data center and accepted the data center downtime risk. Cassandra works very well if configured properly. The topology matters a lot though. We decided to go with six tracker nodes and split them across three racks. Due to the low amount of data, our replication factor is six (all nodes have 100% of the data), our read consistency is three (read from three nodes) and our write consistency is quorum (write to four nodes). This gives us full read consistency (all trackers will respond with the same data). For anyone interested, there is a nice Cassandra consistency calculator that helps defining what you need.

With this consistency configuration and Cassandra’s rack-awareness, we can lose one rack (two servers) without the RLY system being affected. We tested this extensively before going live, and even in production when we moved all RLY trackers (two at a time) to a different data center.

They said HTTPS is fast

People on the Internet™ say that adding TLS to HTTP (= HTTPS) doesn’t make a big difference in performance. While that is probably very true for many scenarios, it wasn’t true for us: In our case, Apache cared very much about thousands of SSL handshakes per second, so much so in fact that all processors on all trackers were busy shaking hands with all of the RLY clients all the time.

The problem here of course lies in the way we designed the system: RLY has a heartbeat mechanism that calls out to the trackers every three seconds to see if there any new connections. While that works really well, it leads to pretty bad results if combined with Apache’s HTTPS implementation: The trackers pretty much died. The load was insane. Load averages of >1000 were common. All that TLS handshaking killed the tracker servers. After switching heartbeat to HTTP (don’t freak out, it just returns a timestamp!), the load was much better.

Obviously the heartbeat/checkin architecture is a little off and we could/should have used server side events (SSE) or something similar. We’d probably explore this option in more detail if we were to design the system again. However, given that we have other products for which the design has proven to work really well for years, we stuck with it.

bcrypt with a cost of 10

After solving the HTTPS handshake problem we still had load problems on the trackers, which turned out to be authentication related: To authenticate with the RLY tracker, the client uses a per-client secret key. In our database (think: on the trackers), we store a bcrypt hash of that secret key with the cost of 10 (i.e. 2^10 = 1024 rounds). bcrypt is of course designed to be slow and consume lots of CPU on purpose. After lots of analysis, we found that that was the cause for the load: The trackers were consuming a lot of CPU because they were constantly verifying bcrypt hashes.

Of course it took a bit of time to actually determine that as the root cause, but finding a solution was fairly straight forward: Instead of constantly verifying the bcrypt-2^10-hash, we would only verify it once every few hours, and generate a new bcrypt-2^4-hash upon first auth to use for subsequent requests. We store that weaker hash only in memory of course, but verifying that is much faster.

On my laptop, for instance, verifying 100 bcrypt hashes with the cost of 10 takes ~6 seconds, whereas a cost 4 comparison of 100 hashes takes only .01 seconds.

After that change the load was much much better.

Limits, Limits, Limits!

Another thing we noticed is that high traffic servers like the trackers need a few Linux kernel parameters set to function properly. This may be obvious to some, but it’s not obvious if you’ve never done this before (like me). Luckily we have some smart people in our systems engineering team that know exactly how to do that.

The default limits on a Linux system are too restrictive for high volume servers to perform well under load. Most notably the number of open files (nofile), the number of running processes (nproc) and the maximum number of queued connections (net.core.somaxconn) need to be increased. This is mostly so nginx and Cassandra are allowed to accept more connections.

Here’s our limits.conf configuration. It increases the hard and soft limits for the number of open files:

# <domain> <type> <item> <value>
* soft nofile 1024000
* hard nofile 1024000
* soft nproc 10240
* hard nproc 10240
root soft nproc unlimited
cassandra - memlock unlimited
cassandra - nofile 100000
cassandra - nproc 32768
cassandra - as unlimited

And here’s how we increase the number of allowed queued connections in sysctl.conf:

net.core.somaxconn=10000

There are much more details on how to vertically scale a Linux server for high load. Here’s a great article explaining the details.

Versioning and logging makes things easy

Something we actually did right from the beginning is to add a version string to everything. Every API call, every serialized object, and even every running process has a the RLY version number attached to it. That means we can easily identify old clients and always be backwards compatible. So far we only had to use the RLY version number once to change the relayPort field type to an array (to support multiple ports) and ensure that old clients don’t break. It worked very nicely. Versioning things FTW!

We also log pretty aggressively in RLY, which helps when debugging. All components can enable LogLevel DEBUG, which then logs everything, even the output of ssh and sshd. Log all the things, people! It helps!

OpenSSH does not scale well

Believe it or not, OpenSSH’s sshd is not meant to handle thousands or even tens of thousands of connections/tunnels. We realized this only about 1.5 years after the RLY system had been live when we were thinking about persistent connections instead of on demand tunnels.

In my tests, on a VM with 16 GB of RAM and 4 CPUs, OpenSSH was really hammering the system when I opened 1,100 connections to it. It consumed about half of the RAM and the CPUs were pegged:

After researching alternatives to OpenSSH with the same feature set, we landed on using the Golang built-in crypto/ssh package with great success. The same VM was easily handling 16,000 connections with the CPU almost idle and 1.1 GB of RAM:

It would most likely be able to handle a lot more connections, but my laptop wasn’t able to handle more outgoing SSH tunnels so I stopped and called this “good enough”. Pretty cool right? Instead of adding dozens of new VMs to handle the traffic, we simply used a more appropriate tool.

Monitor and automated end-to-end testing pays off!

With dozens of servers around the globe, it’s important to be able to keep track of all of them in a centralized way and automatically end-to-end test the system regularly enough to detect potential problems before customers do. We put a lot of work into testing various parts of the system (opening tunnels, using the jump servers, …) and mapping the overall health of the system to a single number (0 = error, 1 = warning, 2 = ok).

We then use this number to trigger alerts for our operations group and for visualizations in our RLY dashboard. Warnings include things that can be dealt with in the morning (relay host down, tracker down, …) and errors must be dealt with as soon as possible, because they have customer impact. We have caught quite a few almost-outages like this before anyone noticed!

Final thoughts

In this post I talked about how we use SSH reverse tunnels to access behind-the-NAT devices, and how we created a horizontally scalable application to support the fleet of all our backup appliances. I introduced our RLY application and its architecture with all its components. And finally I briefly talked about a few of our challenges and the lessons we learned.

RLY has been working really well for over two years now and for the most part, all users (internal and external) have been happy. We’ve had no significant outages and we’ve even moved some of the relay hosts and all trackers to a different data center without downtime. We have added more device types over time, starting from the BMC/IPMI in our backup appliances, to our DNA devices and most recently we’ve started work on integrating our other networking appliances.

Naturally, as with most systems, there are things we’d do differently now: We probably wouldn’t do a polling based system to check for updates on the client again (heartbeat/checkin). We’d also probably not use SSH for tunneling as something more lightweight would have also done the trick.

Overall, it’s not really worth changing any of these, because the system works wonderfully, scales great, and is pretty low maintenance. Thanks to all of the smart engineers that made RLY happen. It’s a pleasure to work with you every day.

Being faced with this problem for work, we started exploring alternative options and came up with something that has worked reliably for almost two years for a fleet of now over 80,000 devices. In this blog post, I’d like to talk about how we solved this problem using images, loop devices and lots of Grub-magic. If you’d like to know more, keep reading.

1. Debian packages all the way?

Our BCDR appliance has always been Ubuntu-based, so the natural way to update our software was through Debian packages. That’s what we did for a long time: every two weeks, we’d build releases for Ubuntu 10.04/12.04 (yes, we know, read on!) and after thoroughly testing them, we’d ship those to the fleet.

That worked for a long time, but it had some significant drawbacks:

• Updating third party dependencies: Having a few Debian packages implies that you are really just taking responsibility for your own software and not for other software on the appliance. Managing updates for Apache, Samba, libc or even PHP is non-trivial when all you have available is your own something.deb. Given that my employer sells the entire appliance, we of course have to take responsibility for the entire stack, especially when you think about security patches for third party software.
• Service restarts / reboots: The dependency problem becomes particularly tricky with major updates that require service restarts or even reboots. Of course, Debian packages are supposed to handle service restarts themselves, but in reality not all of them do that gracefully. And as soon as you get into reboot-territory (for things like kernel upgrades), you need to make sure that you’re not interrupting important device tasks (backups, virtualizations, …), and you have to make sure that your device actually comes back up (not as easy as you think, see below!).
• Distribution upgrades: Now when it really becomes tricky is when the entire OS version needs to be moved forward. Upgrading from Ubuntu 10.04 to 16.04 using just apt-get dist-upgrade and the occasional reboot, for instance, will take incredibly long and many times will straight up not work (if you’ve ever used apt-get dist-upgrade, you’ll understand).
• Thousands of versions and states: With the Debian upgrade model, devices really behaved like normal computers: Right after they were freshly imaged and shipped out, everything was great and new, but the older they got the more the OS deteriorated – thereby leaving devices in vastly different states. Just to give you an idea: We had 40 different versions of VirtualBox on our devices (before we switched to KVM), 25 different ZFS versions and over 80 different Linux kernels!

That is by no means a complete list of problems, but I don’t want to bore you with even more problems. Let’s get to the fun part: How to solve it!

2. Images, not packages!

With all these problems it’s pretty obvious that we needed something better. We needed to manage the state and configuration for our devices. It was important to minimize the number of different device configurations/packages/versions in the fleet, and to ensure that with every upgrade, we’d be able to upgrade the entire stack – be it our own software, third party software, or even system libraries such as libc or the kernel.

2.1. Requirements

We started defining our requirements for a solution, and they were actually pretty short:

• All devices follow the same upgrade path. There is only one upgrade path.
• All devices can be upgraded this way (even older ones with small OS drives)
• Switching from one version to another is atomic (or as atomic as possible).
• Rolling back to a previous version is possible (if the upgrade fails).

The strongest implication of these requirements is of course that we were going to drop package based upgrades, and – if you can read between the lines – that rebooting an appliance as part of the upgrade process is acceptable.

Both of these are a pretty big deal. Big decisions were being made!

2.2. What are images anyway?

To be able to reduce the number of configurations, we decided that we wanted to stop treating our software and all its dependencies as separate pieces. Instead, we wanted to combine them all into a single deliverable – an image.

So what’s an image? An image (in our world) is an ext4 file system that contains everything needed to boot and run a BCDR appliance. That includes:

• The Ubuntu base operating system (kernel, system libraries, ..)
• Required third party tools and libraries (Apache, KVM, ZFS, …)
• The device software (dubbed IRIS by our marketing team)

Here’s a nice picture showing an example:

We were excited about this idea because with images there is exactly one number, the image version (415 in the picture above), that defines the version of every piece of software that is installed. No more testing our software with multiple ZFS versions, no more hoping that it’ll work with all KVM versions. Yey!

3. Image based upgrades (“IBU”)

After making all of these important decisions, we still needed to find a way to build, distribute and boot these images on our devices. So let’s dive into these topics.

3.1. Building an image

We typically build an image automatically every time we tag a new release (or release candidate): Every time we push a tag in Git, a CI worker starts the image build process. The build process itself is pretty interesting, but a little out of scope for this post. But I don’t want to leave you hanging so here is the short version:

We first build Debian packages of our own software and publish that to a Debian repository. We snapshot the Debian repository using aptly, and we do the same regularly with an upstream Ubuntu repository (“Upstream packages”). We then we use debootstrap to create a base Ubuntu system and install our software with all its dependencies in a chroot. Once that’s done, we tar it up and rsync it to our image server. On there, we extract the tarball and rsync it over the last image, which resides in a ZFS volume (zvol) formatted as ext4. After we zero out unused ext4 blocks, we finally snapshot the zvol containing the file system.

Here’s what that looks like on the image server:

That zvol now contains the ext4 file system of our BCDR appliance. That is the image. That’s our single deliverable. It can be tested as one unit and once it passes QA it can be distributed to our customers’ BCDR devices.

3.2. Distributing the image

Great, now we’ve successfully built an image. How do we get it from our data centers to the >80k devices? Pretty simple: We use ZFS send/recv!

Our devices all have ZFS pools anyway to store their image based backups, and we already heavily use ZFS send/recv to offsite their backups. So using this technology in reverse was a no-brainer.

So here’s what we do: When it’s time to upgrade, we instruct a device to download a ZFS sendfile diff over HTTPS (we’d use ZFS send/recv over SSH directly but that cannot be cached unfortunately):

As you can see in the screenshot, we typically don’t have to download the full image, because the devices have upgraded before and already have a version of the image in their local pool. That’s pretty sweet: Using this technique, we can do differential operating system upgrades – meaning devices only have to download the changed blocks of the image.

That’s a win-win, because it means we don’t overuse our customers’ networks and we don’t have to provide lots of bandwidth for upgrades from our data centers.

Once the image is downloaded, we import it to the local ZFS pool. This is particularly important for the next upgrade (to ensure we can download diffs):

3.3. Booting the image

Now that we have the image, how do we boot into that file system? And if every image version we build is a completely fresh OS, how do we go from one version to the next?

3.3.1. ZFS-on-root, A/B partitions and A/B folders

Not surprisingly, there is more than one answer to these questions. There are many ways to use the image to produce a bootable system, so we had to experiment a little bit to find the best one.

Given that they are quite interesting, I thought I’d briefly mention them and why we didn’t end up doing them:

• ZFS-on-root with A/B datasets: We use ZFS pretty heavily for our image based backups, so we naturally thought we could also use it as the root file system of our appliance. The idea here was to distribute the image for the BCDR appliance as a ZFS dataset (not as zvol as described above), clone it and then directly boot into the ZFS clone. Since Grub supports reading ZFS in its newer versions and there’s also a ZFS initramfs module, ZFS-on-root is absolutely possible. To upgrade from one image to the next (i.e. one ZFS dataset to the next), we’d simply update the Grub config and reboot. It worked beautifully, but since booting into ZFS is pretty new, we didn’t feel like it was mature enough for our main product. Pass.
• Simple A/B partitions: Quite a few appliances and phones simply have two partitions, one with the current system and one with the next. This idea was straight forward: Download the new image, rsync it to the inactive partition, update Grub, and then reboot. The unfortunate thing here was that not all of our devices have spare space for an extra partition (not without repartitioning at least). We experimented (and succeeded) with splitting the active root partition from inside initramfs during the first reboot (cool, right?), but again, given that this is our main product, it felt like an insane risk to take. Pass.
• Boot into A/B directories: Given the lack of a spare partition, we experimented with having two copies of the image on the root partition in two different folders (e.g. /images/412 and /images/415), and then modifying initramfs to boot into /images/415 instead of /. Believe it or not, while it sounds super crazy, it actually worked and was super easy by messing with initramfs a little: mount --bind /images/415 /root did the trick. Everything booted just fine. However, lots of Linux tools (df, mount, …) got really confused by the fact that the root is not / – so we decided to pass on that as well.

3.2.2. Loops all the way!

After trying so many different ways of booting into an image, what we ended up doing almost sounds a bit boring. But boring is good, right?

The simplest and most reliable way we found to boot into an image was to leverage Grub’s loopback booting mechanism, combined with the loop support in initramfs (see loop=... parameter):

Grub, as you most certainly know, is a boot loader. Its responsibility is to load the initial RAM disk and the kernel. For this purpose, Grub has built-in read support for many file systems and as we will see shortly also for file systems within file systems via the loopback command. The loopback command will find an image file on the root partition and loop it, so that the linux and initrd command can be used like normal to find the kernel and RAM disk. Here’s one of the menu entries we generate (via a hook in /etc/grub.d) for our device’s grub.cfg files:

menuentry 'Custom OS (v415.0)' {
  search --set=root --no-floppy --fs-uuid 8c43bf01-046c-401c-8cb8-97cb658ef698
  loopback loop /images/415.0.img
  linux (loop)/vmlinuz root=UUID=8c43bf01-046c-401c-8cb8-97cb658ef698 rw loop=/images/415.0.img ...
  initrd (loop)/initrd.img
}

In this example, Grub will first find the root partition (just like on a normal Ubuntu installation) by its UUID via search. It will then discover the image file /images/415.0.img on that root partition and finally find the kernel ((loop)/vmlinuz) and RAM disk ((loop)/initrd.img) inside the image.

It’s incredibly simple, but really also incredibly cool: The fact that a boot loader can do all that still amazes me.

Once Grub found the kernel and the initial RAM disk, it loads the RAM disk into memory (shocker!), which is then responsible for mounting the root file system before handing control over to the init process.

In Ubuntu, the initramfs-tools package provides utilities to create and modify the initial RAM disk. Luckily it already supports loopback booting, so typically there isn’t really much more to do besides passing a loop= parameter in the kernel line. If that is set, initramfs will loop-mount the root file system to the image via mount -o loop (see source code). Given that there is a pretty scary FIXME message in the code (# FIXME This has no error checking), we thought it’d be good to make that a little more resilient by adding error handling and fscking to it. In most cases, it should boot without messing with initramfs though.

That’s really it. A lot of words for such a simple solution.

Here’s what it looks like in practice. As you can see, this device’s root file system lives on /dev/loop0. This loop device was set up in initramfs and it points to the image file:

In this case, the image resides on the root partition (e.g. /dev/sda1) in /images/412.0.img. Note if an empty /host folder is present inside the image, initramfs will mount the root partition there:

3.4. Upgrading between images

So now we can build, distribute and boot images. If we combine these things, you can already see that upgrading from one image to the next is not hard:

1. Clean up old images, download new image, import into pool, export to image file
2. Migrate configuration from current image to next image
3. Update Grub to point to new image
4. Reboot

And that’s pretty much what we do. We wrote a tool called upgradectl for that:

upgradectl is typically triggered remotely by our checkin process: It prepares the upgrade in the background during normal device operation by downloading and exporting the image (step 1). And when it’s finally time to upgrade (usually at night when the device is idle), the actual upgrade process will be blazingly fast, because it only has to migrate configuration, update Grub and reboot (steps 2-4). Typically, the downtime for a device during an upgrade is about 5-10 minutes, depending on how long it takes to reboot (larger devices will take longer due to IPMI/BMC initialization).

Now of course there are tons and tons of gotchas and edge cases to consider: It sounds easy, but it’s really quite hard to get right – especially considering that our 80k existing appliances had been in production for up to 7 years.

That made for some fun challenges: We upgraded thousands of devices from Ubuntu 12.04 (and even 10.04) directly to Ubuntu 16.04. We implemented logic to deal with falling back to old images in case the upgrade failed for some reason or the next. We dealt with full OS drives, faulty hardware (disks, IPMI, RAM, …), RAIDed OS drives and Grub’s inability to write to them, ZFS pool corruption, hung Linux processes (good old D state), stuck reboots, and many many more.

But guess what: It was worth it. It felt like spring cleaning after a winter that lasted 7 years. We brought devices into the best shape of their lives. And we’re still doing it. Every 2 weeks!

4. Summary

In this post, I showed you how we changed the deployment process for our BCDR appliance from Debian packages to images. I talked about how we build images, how we distribute them and how we use Grub’s loopback mechanism to boot them.

While I touched all the pieces to perform image based upgrades, the most exciting thing about all of it is this: Using this mechanism, we can move between kernels and even major operating system versions. We’re effectively booting into a completely fresh, new operating system every time we ship an upgrade. That means that the system won’t deteriorate over time, manual changes will be wiped out and that we could technically switch Linux distros in a heartbeat if we wanted to.

And all of this is done in the background, with no interaction of our customers, completely invisible to their eyes: 80k operating system upgrades every two weeks, how cool is that?

So I started on a journey to attempt to dedup NTFS. This blog post briefly describes my journey and thoughts, but also introduces a tool called fsdup I developed as part of a 3 week proof-of-concept. Please note that while the tool works, it’s highly experimental and should not be used in production!

0. Link to the fsdup tool

If you’re impatient, here’s a link to the fsdup tool I created as part of this proof-of-concept.

1. Thoughts on NTFS deduping

1.1. The obvious approach

The obvious approach to dedup NTFS would be to simply dedup blocks (= clusters), so typically that would mean deduping with a fixed offset of 4096 bytes (4K cluster size). While that will lead to the best dedup results, it also produces a lot of metadata and the metadata-to-data ratio (metadata overhead) is pretty bad. In addition to that, the more chunks you produce the higher the chance of hash collisions (unless you increase the hash size, which in turn increases the metadata overhead …). On top of that, many object stores or file systems are not really good with small files like that, so a higher chunk size is desirable.

That said, given that we’re talking about hundreds of thousands (maybe even millions? I should really check …) of NTFS file systems, it might not matter if the objects are 4K or 128K in size.

1.2. What I did instead

Since this was a proof-of-concept, and I wanted to learn about NTFS and write some fun Go code, I chose to go a different route:

Instead of fixed offset chunking with 4K chunks, I decided to parse the NTFS filesystem using a combination of content aware chunking (for the file system itself) and fixed offset chunking (for the files within the file system).

Here’s my approach:

• Index files within NTFS: Read all entries in the master file table ($MFT) that are larger than X (e.g. 128K, 2M, …). For each entry, read all data runs (extents, fragments) sequentially (i.e. re-assmble the file). Break the re-assembled file into max. 32M sized chunks and then hash each chunk using blake2b. For each chunk checksum, compare if it’s in the global chunk index and only store new chunks.

• Index empty space: Read the NTFS bitmap ($Bitmap), find unused sections and mark them as sparse. This step is optional. If it is used, the re-assembled file system will likely not be identical as the original one, because unused space is rarely ever filled with zeros. However, the space savings due to this step are enormous and the user won’t care.

• Find gaps: All sections of the file system that are not covered by step 1 or 2 are stored in gap chunks. This typically includes all files smaller than X (see above).

The algorithm will create a manifest file for every NTFS file system which can later be used to re-create it from scratch, either by exporting it from the global chunk store, or by mapping it to a drive or mountpoint.

Here’s a (simplified) diagram showing (mostly) step one:

2. fsdup: A tool to dedup NTFS

So to prove that the method above works, I created a small tool called fsdup. It implements the algorithm above, and also provides ways to re-assemble deduped file systems or expose them as mapped drives. Here’s the current feature set:

• fsdup index: Index/dedup NTFS file systems into a chunk index (stored locally or in Ceph). This creates a manifest file to describe the chunks that make up a file system.
• fsdup export: Using a manifest and a chunk index, this command re-assembles a file system into an image file. This is the reverse of the index command.
• fsdup map: Using a manifest and a chunk index, this command exposes a block device of the block device that was indexed using the index command. This command is super useful for booting VMs directly of off a chunk store.

2.1. Indexing

First, let’s dedup the NTFS file system by breaking it into chunks using the approach described above: The fsdup index command takes two arguments, an input file system (block device or file), as well as an output file (a manifest that describes what chunks the input file consists of). In this example, we assume that the NTFS file system is in a file called Cdrive.img.

# Deduplicates file, creates chunk index, generates Cdrive.manifest.
# By default, this will index files >128k and use a local folder "index" as chunk store.
$ fsdup index Cdrive.img Cdrive.manifest
Detected MBR disk
NTFS partition of size 249.0 MB found at offset 1048576
Indexed 52.2 MB in 9 file(s) (147 skipped)                        
Indexed 195.6 MB of unused space                             
Indexed 1.2 MB of gaps (21 chunk(s))             
NTFS partition successfully indexed  
Indexed 1.0 MB of gaps (1 chunk(s))           
MBR disk fully indexed

# Alternatively, Ceph can be used as chunk store, and the default parameters can be overriden
$ fsdup index -minsize 2M -store 'ceph:?pool=fsdup2m' Cdrive.img Cdrive.manifest
...

# Prints the manifest
$ fsdup print Cdrive.manifest
diskoff 000000 - 196608 len 196608  chunk a4bfc70 chunkoff 0 - 196608
diskoff 196608 - 288832 len 9092224 sparse
diskoff 288832 - 759872 len 471040  chunk 8976c52 chunkoff 0 - 471040
diskoff 759872 - 981256 len 28282   chunk beef1da chunkoff 28282 - 61040
...

The main deduplication algorithm described above is codified in the chunker_ntfs.go file. The top of the file also describes the above mentioned approach. Feel free to poke around and leave comments.

Please be reminded that this is a proof-of-concept. The algorithm is not optimized at all. In particular, it does not do a good job reading the file system sequentially (see section “Thoughts” below), meaning that it’s particularly slow on spinning disks. It wouldn’t be hard to rewrite it to work (more) sequentially though.

2.2. Exporting and mapping

Once indexed, the file system can be exported from the chunk store, or mapped to a local mountpoint using the manifest we created with the index command. These operations can be done with the fsdup export and fsdup map commands.

# Creates an image file from chunk index using the manifest
$ fsdup export Cdrive.manifest Cdrive.img

# Alternatively, if Ceph was used as a chunk store, you have to specify the pool
$ fsdup export -store 'ceph:?pool=fsdup2m' Cdrive.manifest Cdrive.img

# Creates a block device /dev/nbdX that allows reading/writing the image file without exporting it.
# The second parameter is a copy-on-write file which is used as target for write operations.
$ fsdup map Cdrive.manifest Cdrive.cow

The fsdup map command is super useful for booting VMs directly from the deduplicated chunk index in Ceph/RADOS. That eliminates wait times for exporting/re-assembling the image. One could, for instance, boot the VM like this:

$ fsdup map Cdrive.manifest Cdrive.cow # maps to /dev/nbd0
$ kvm -drive format=raw,file=/dev/nbd0 -serial stdio -m 4G -cpu host -smp 2

Please note: The fsdup map command is really really experimental and as of now super inefficient: Regardless of the size of the a read/write request, the command reads the entire 32M chunk from the global chunk store and writes it to a cache and the copy-on-write file (here: Cdrive.cow). Don’t judge: It was really easy to implement and I was planning to optimize it later. In practice, this means that until the relevant sections have been written to the local copy-on-write file, reads and writes will be slow and your Windows OS may feel sluggish.

3. Experiments

I did some preliminary experiments with 126 production NTFS file systems, totalling 53.5 TB in size, with 16.2 TB used space on disk. The goal of the experiments was to maximize the dedup ratio and the space reduction percentage. As a reminder, here’s how they are defined (for a detailed description, check out my Master’s thesis):

Disclaimer: This is a very small sample so results might vary quite significantly with more input data. Also, please note that these experiments only measure the dedup ratio and space reduction. They do not take indexing performance or compression into account, which can have tremendous impact on the overall space savings or speed.

Experiments:

1. Dedup with fsdup index -minsize 2M

2. Dedup with fsdup index -nowrite -minsize 128k

3. Dedup with fsdup index -nowrite -minsize 64k

4. Dedup with fsdup index -nowrite -minsize 8k

-minsize X defines the minimum file size to be indexed
-nowrite means to not actually write the chunk index, i.e. only create a manifest file.

Experiments 2-4 were performed without actually writing out the chunk index, which is not actually necessary to calculate the dedup ratio and space reduction.

3.1. Experiment 1: Minimum file size 2M

$ fsdup stat *.manifest
Manifests:               126
Number of unique chunks: 3,860,409
Total image size:        53.5 TB (58853137407488 bytes)
Total used size:         16.2 TB (17782383914496 bytes)
Total sparse size:       37.4 TB (41070753492992 bytes)
Total chunk size:        11.6 TB (12751789303491 bytes)
Average chunk size:      3.2 MB (3303222 bytes)
Dedup ratio:             1.4 : 1
Space reduction ratio:   28.3 %

3.2. Experiment 2: Minimum file size 128k

$ fsdup stat *.manifest
Manifests:               126
Number of unique chunks: 8,660,507
Total image size:        53.5 TB (58853137407488 bytes)
Total used size:         16.2 TB (17782383914496 bytes)
Total sparse size:       37.4 TB (41070753492992 bytes)
Total chunk size:        10.8 TB (11887333358175 bytes)
Average chunk size:      1.3 MB (1372591 bytes)
Dedup ratio:             1.5 : 1
Space reduction ratio:   33.2 %   

3.3. Experiment 3: Minimum file size 64k

$ fsdup stat *.manifest
Manifests:               126
Number of unique chunks: 10,729,245
Total image size:        53.5 TB (58853137407488 bytes)
Total used size:         16.2 TB (17782383914496 bytes)
Total sparse size:       37.4 TB (41070753492992 bytes)
Total chunk size:        10.7 TB (11811050790919 bytes)
Average chunk size:      1.0 MB (1100828 bytes)
Dedup ratio:             1.5 : 1
Space reduction ratio:   33.6 %

3.4. Experiment 4: Minimum file size 8k

$ fsdup stat *.manifest
Manifests:               126
Number of unique chunks: 17,831,801
Total image size:        53.5 TB (58853137407488 bytes)
Total on disk size:      16.2 TB (17782383914496 bytes)
Total sparse size:       37.4 TB (41070753492992 bytes)
Total chunk size:        10.6 TB (11699232512973 bytes)
Average chunk size:      640.7 KB (656088 bytes)
Dedup ratio:             1.5 : 1
Space reduction ratio:   34.2 %

4. Thoughts

4.1. Results

While these results are pretty good, I was a little disappointed by the fact that I could only get 33% space savings across so many Windows installations. It could be the sample size, or it could be the fact that people store more user-specific data than I thought they would, or it could be that I’m not excluding temporary files or large computer-specific files such as pagefile.sys or hiberfil.sys.

In any case, there is much more to be investigated and much more to be optimized. I am quite confident that I can get to much higher dedup ratios with a higher sample and a little more investigation time.

Too bad I don’t have more time right now. So this is it for now. Or is it?

4.2. Further experiments

Since I wasn’t satisfied with the results, I informally continued the experiments in a less structured way. I picked 128k as a minimum file size, because that seems good enough, especially since 64k and 8k were so much slower than 128k and 2M (not shown in the results).

In one experiment, I continued ingesting more NTFS image files (up to 195), and while the dedup ratio for a 128k min size shrunk a little (32.1%), I experimented with compressing the chunks individually before storing them. With the same dataset, i.e. 195 image files, I got the following results:

$ fsdup stat *.manifest

Manifests:                  195
Number of unique chunks:    13001394
Total image size:           70.1 TB (77086756862464 bytes)
- Used:                     23.6 TB (25972460016504 bytes)
- Sparse/empty:             46.5 TB (51114296845960 bytes)
Total chunk size:           16.0 TB (17644694259014 bytes)
Average chunk size:         1.3 MB (1357139 bytes)
Median chunk size:          92.6 KB (94845 bytes)
Dedup ratio:                1.5 : 1
Space reduction:            32.1 %

Stats after compression (Ceph/gzip):
Total size of image files:            23.6 TB (no compression)
Total size of chunks (in Ceph/gzip):  10.0 TB (deduped in Ceph with gzipped chunks)          
Dedup ratio (raw vs Ceph/gzip):       2.36 : 1 (-57.6 %)

As you can see, compression is a real game changer (obviously). We now get 57% space savings!

Now the obvious question is: What if I simply compress the original image files with gzip? Will that not lead to similar savings? Good question, my dear reader! I have, unfortunately, not done any tests with purely compressing image files using gzip.

I do, however, have at least some numbers with regards to this: Storing these 195 image files on ZFS with lz4 compression takes up about 17 TB of space, so that’s a ratio of 1.38 : 1 (23.6 : 17), which is obviously much worse than Ceph+gzip, but it also doesn’t really say anything because it’s not an apples-to-apples kind of comparison.

Maybe I’ll eventually find the time to continue the experiments … Who knows.

4.2. Interesting things I’ve learned

Besides the results of the experiments, I’ve learned quite a few things, which I thought I’d share with you.

4.2.1. Parsing NTFS is hard

Who would have thought that a simple file system could be so complicated. When I started this, I briefly looked for Go libraries to parse NTFS and I found go-ntfs, which works wonderfully, but had a super awkward interface, and it simply didn’t fit my needs. What I needed was the boundaries for all extents (data runs) for all files greater than X bytes – not something the library provided. I then briefly looked into using ntfs-3g and use cgo as an interface to use it in Go. That turned out to be tedious (maybe because I had never used cgo before? maybe because ntfs-3g is hardly documented at all?), so I didn’t go that route either.

Long story short, I decided to write my own parser. How hard could it be?

Turns out, it’s not terribly hard to read the sector size, the cluster size and other basic information from the NTFS boot sector. Getting from the boot sector to the $MFT and then iterating over all entries was also manageable. Once I had a file entry though, NTFS has these strange data runs (extends, fragments) that are terribly hard to parse. Once you know how to read little endian ints (signed and unsigned!), it’s all super easy, but getting there was quite the challenge.

Oh, and then there is this thing called NTFS fixups (also called update sequences) which replaces bytes at the end of each MFT entry as a means to detect errors. What crazy madness is this?

The one resource I can highly recommend for anything NTFS is Richard Russon’s NTFS documentation, a great reference for all the weird things that NTFS does and doesn’t do. I particularly enjoyed his writing style. Here’s a gem from the section about $Bitmap parsing:

“The backup copy of the boot sector lies in this no-mans-land the cluster is hence marked as in use. In theory, on very small volume, this attribute could be resident. In practice, Windows crashes.”

Another resource I highly recommend is the Active Disk Editor, a wonderful colored hex editor that is really good at displaying NTFS sectors, MFT entries and data runs. It only worked properly on Windows, but it was worth it working in a VM. Without it, I would not have succeeded.

Anyway. What I learned was this: Don’t write your own NTFS parser. Use someone else’s! I should have spent more time with ntfs-3g and cgo. Or just written this part in C.

4.2.2. I got better at Go

Which brings me to this. Part of why I wrote this in Go is to get better at Go. I think I did. I know how to structure things and I’m generally more confident with the language. Yeyy!

4.2.3. Random access on HDDs is sloowww ….

As of its current iteration, fsdup doesn’t do a great job to read the disk (file system) sequentially. It does this instead: Read MFT entry, read data runs, read each data run from disk. Since the MFT is (usually) at the beginning of the disk and the data runs are spread out all over the place (the more fragmented the file system the more spread out they are), the read head has to jump back and forth a lot.

That’s slow! Yes, I know. It’s obvious. I knew random access on HDDs was slow, but I didn’t know how much of a difference it made. Well now I know.

The indexing/chunking algorithm can be optimized for (mostly) sequential reads though pretty easily. I haven’t done this yet, but here’s how I’d do it if I had more time: Read all entries in the MFT, sort by first data run offset, then read each MFT entry like above.

Assuming that most file entries fit in one data run, and that the file system is not horribly fragmented, that approach should read the disk mostly sequentially.

5. What now? So what now? Well right now, I am not working on this anymore, but I suspect that I will in the near future. I’ll probably experiment with file exclusion for temporary files and files like pagefile.sys. I’ll also try to get some file size histograms so I can get some data about what a good min size will be.

In addition to that, I might try to use the simple 4K approach, i.e. simply dedup blocks/clusters instead of all this mumbo-jumbo with the files. If I find a good data store to support this many objects without much overhead, why not, right?

I also need to investigate and find a way to fit compression into this concept. The current implementation gzips the chunks before storing them, which is great for writing, but not great for random access reading of partial chunks (think: get me bytes 1024-2048 from chunk 1234), because gzip is not really easily seekable. Not great.

So there’s lots to do and no time to do it. Isn’t that what it always feels like?

Thanks for reading. Feel free to leave comments. I love conversing about this!

This (super duper) short post shows how to boot image files using straightkvm commands.

1. Booting image files with KVM (non-EFI/BIOS)

First, make sure you have KVM installed (I’m assuming you’re running Debian/Ubuntu):

_Install KVM _

$ sudo apt install qemu-kvm

Now that that’s done, let’s boot the image: assuming you have a raw image file (e.g. disk.img) or an ISO (e.g. ubuntu-18.04.iso), you can use the following script to start a virtual machine in KVM:

Script ‘boot’ to boot from a single image file via KVM

#!/bin/bash
kvm \
  -drive format=raw,file=$1 \
  -serial stdio \
  -m 4G \
  -cpu host \
  -smp 2

The command will attach one disk (-drive format=raw,file=$1 with $1 being the disk file) and it’ll assign 4 GB of RAM to the VM (-m 4G). Since no -bios flag is specified, KVM/QEMU will use its own BIOS. To boot via UEFI, see below.

If you save the snippet as a script (and call it boot), you can use it very easily to boot any kind of image like this:

$ boot disk.img
$ boot debian-live-8.4.0-amd64-standard.iso
...

If you’ve done everything right, you should see the system booting up:

You can control the VM using the KVM command shell by hitting Ctrl-Alt-1:

What I mostly use this for is to send the Ctrl-Alt-F1 to the VM to switch to a different TTY.

2. Booting image files with KVM (EFI)

Booting a virtual machine via UEFI is pretty similar. The only real thing you have to change is to specify a UEFI firmware that KVM should use. OVMF is such a firmware. You can install it via apt:

Install KVM and the UEFI firmware

$ sudo apt install qemu-kvm ovmf

Once that’s installed, simply add the -bios option like this:

Script ‘boot-efi’ to boot from a single image file via KVM

#!/bin/bash
kvm \
  -bios /usr/share/qemu/OVMF.fd \
  -drive format=raw,file=$1 \
  -net none \
  -serial stdio \
  -m 4G \
  -cpu host \
  -smp 2

After you save this as a script, e.g. boot-efi, you can use it like this:

$ boot-efi disk.img
$ boot-efi ubuntu-16.04.1-desktop-amd64.iso
...

When booting, you’ll see that the output will have changed slightly, like this:

That’s really it. You can check out more KVM options in the man page by typing man qemu-system-x86_64.

A. About this post

I’m trying a new section for my blog. I call it Code Snippets. It’ll be very short, code-focused posts of things I recently discovered or find fascinating or helpful. I hope this helps.

But while there are many tools to automatically renew certificates for publicly available webservers (certbot, simp_le, I wrote about how to do that 3 years back), it’s hard to find any useful information about how to issue certificates for internal non Internet facing servers and/or devices with Let’s Encrypt.

This blog posts describes how to issue Let’s Encrypt certificates for internal servers. At my work, we issued a certificate for each of our 65,000 90,000+ BCDR appliances using this exact mechanism.

Hello Hacker News, first time on the HN front page! I feel honored, yeyy! I responded to all of the concerns in the comments section.

If you’re looking for an implementation of this idea, you may find localtls interesting. I have not tested it myself, but it seems to do similar things to what I am describing here.

1. How does it work?

To issue a certificate through Let’s Encrypt, you must prove that you either own the website you want to issue the certificate for, or that you own the domain it runs on. Typically, automated tools like certbot use the HTTP challenge to prove site ownership using the .well-known directory. While this works beautifully if the site is Internet-facing (and Let’s Encrypt can verify the HTTP challenge files via a simple HTTP request), it doesn’t work if your server runs on 10.1.1.4 or any other internal address.

The DNS challenge solves this problem by letting you prove domain ownership through the DNS TXT record _acme-challenge.example.com. Let’s Encrypt will verify that the record matches what it expects and issue your certificate if it all adds up.

So really the magic ingredients to issuing certificates for internal non Internet facing machines are:

• A dedicated DNS zone for all your internal devices, e.g. xi8qz.example.com, and a dynamic DNS server to manage this zone (here: example.com)
• An ACME client capable of using the Let’s Encrypt’s DNS challenge to prove domain ownership

2. Example: An internal server 10.1.1.4, aka. xi8qz.example.com

The following diagram shows how we have implemented our Let’s Encrypt integration for our backup appliances. Each appliance (read: internal server) is behind a NAT and carries its own local IP address.

The general approach is simple: The appliance regularly reaches out to our control server to ensure that it can be reached via its own subdomain. If its local IP address changes, it triggers an update of its own subdomain. In addition, it checks regularly if the certificate is still valid, and requests a renewal if it’s outdated.

Here’s a bit more detail to this process:

For this example, let’s assume we’re trying to issue a certificate for an appliance with the identifier xi8qz and the local IP address 10.1.1.4. From the perspective of this appliance, there are two requests to be made:

• Steps 1-3: First, it needs to set/update its own DNS domain (here: xi8qz.example.com). This domain will later be used as a common name (CN) in the certificate. On top of that, it needs to make sure that this record is updated every time the server’s IP address changes.
• Steps 4-14: It needs to regularly check if the local certificate needs to be renewed and request a renewal if it’s time. Obviously, if there is no certificate it needs to be “renewed”.

Let’s now examine these steps in greater detail.

2.1. Prerequisites: Assigning a domain for each machine (steps 1-3)

As mentioned above, we need to give each appliance a proper domain name in order to be able to prove ownership to Let’s Encrypt, so we need to buy a domain (here: example.com) and delegate its NS records to our DDNS server:

Our DDNS server should own the domain we’ve chosen for our machines

$ dig +short NS example.com
ddns1.mycompany.com.

On top of that, we need the ability to dynamically add and remove records from it (via an API of some sort). I’ve previously written about how to spin up your own DDNS server, if you are interested.

Once that’s all set up, we need to make sure that the machine’s A record is updated whenever its IP address changes. For our internal machine, let’s assign xi8qz.example.com as its domain. If everything’s working properly, you should be able to resolve this domain to its IP address using a normal DNS query:

The machine’s A record resolves to its local IP address

$ dig +short xi8qz.example.com
10.1.1.4

2.2. Requesting a certificate (steps 4-14)

Assuming you now control the DNS zone for example.com completely and you can quickly edit it dynamically, you’re all set for actually issuing certificates for your local device domain via Let’s Encrypt.

For our example appliance, it will regularly check if the existing certificate is still valid (step 4). If there is no certificate or the existing one is about to expire, the device will generate a keypair and a certificate signing request (CSR) using its assigned hostname (here: xi8qz.example.com) as a CN, and it’ll send that CSR to the control server (step 5).

After authorizing the request (an important step not shown in the diagram!), the control server requests a DNS challenge for the given domain from the ACME API via the Pre-Authorization/new-authz API call (step 6). The ACME API responds with a DNS challenge (step 7). If all goes well, this looks something like this:

Response from the ACME API for a new-authz request

{
  "identifier": {
    "type": "dns",
    "value": "xi8qz.example.com"
  },
  "status": "pending",
  "expires": "2018-04-15T21:26:29Z",
  "challenges": [
    {
      "type": "dns-01",
      "status": "pending",
      "uri": "https://acme-staging.api.letsencrypt.org/acme/challenge/VtjihR4X8nLAj4MDwI...",
      "token": "aLptEKAeUOajkiGrx-kkbjUX4b1MC..."
    },
    // ...
  ],
  // ...
}

Using this response, the control server must set a DNS TXT record at _acme-challenge.xi8qz.example.com (step 8) and notify the ACME API that the challenge response has been placed (step 9).

Once the challenge response has been verified by Let’s Encrypt (step 10-11), the certificate can finally be requested using the CSR (step 12-13).

After Let’s Encrypt responds with a certificate, you’ll see something like this on the wire:

-----BEGIN CERTIFICATE-----
MIIGEjCCBPqgAwIBAgISAyk2izMz7OXSqHeZhg+rUR5uMA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
...

If decoded with openssl, we can see that’s it’s the real deal:

$ openssl x509 -in www.crt -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            03:29:36:8b:33:33:ec:e5:d2:a8:77:99:86:0f:ab:51:1e:6e
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=Let's Encrypt, CN=Let's Encrypt Authority X3
        Validity
            Not Before: Jul 18 23:37:35 2018 GMT
            Not After : Oct 16 23:37:35 2018 GMT
        Subject: CN=xi8qz.example.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:be:69:df:28:04:9c:2b:e9:94:72:c3:de:a6:fd:
                    a4:38:93:be:43:a7:81:8b:dc:9a:be:19:0d:c0:d1:
...

This certificate is then returned to the machine (step 14). After the webserver of the appliance/server has been restarted, it’s web interface can be accessed via HTTPS in the browser or on the command line:

Connecting to the internal server via HTTPS

$ curl -v https://xi8qz.example.com/login
*   Trying 10.1.1.4...
* TCP_NODELAY set
* Connected to xi8qz.example.com (10.1.1.4) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=xi8qz.example.com
*  start date: Jul 18 23:37:35 2018 GMT
*  expire date: Oct 16 23:37:35 2018 GMT
*  subjectAltName: host "xi8qz.example.com" matched cert's "xi8qz.example.com"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
> GET /login HTTP/1.1
> Host: xi8qz.example.com
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Sun, 05 Aug 2018 17:38:49 GMT
< Server: Apache/2.4.18 (Ubuntu)
...

3. Deployment considerations: Let’s Encrypt rate limits

It’s important to note that if you are considering implementing this mechanism for a large number of servers that you use the Let’s Encrypt staging environments for testing and, more importantly, that you consider their rate limit restrictions.

By default, Let’s Encrypt only allows you to issue 20 certificates per week for the same domain or the same account. To increase this number, you have to either request a higher rate limit or get your domain added to the public suffix list (note: adding your domain here has other implications!).

Due to these rate limits, it is vital that you spread out the initial deployment enough to stay under the rate limit, and that you leave enough room for future servers to be added. Also consider renewals in the initial rollout plan.

4. Summary As you can see it’s not really rocket science.

We first assigned each appliance (aka. internal server) a public domain name using our own dynamic DNS server and a dedicated DNS zone. Using the server’s assigned domain (here: xi8qz.example.com), we then used Let’s Encrypt’s free certificate offering and their DNS challenge to issue a certificate for that server.

By doing that for all internal servers, we can provide secure communication in our internal IT infrastructure without having to deploy a custom CA cert or having to pay for certificates.

If you have no physical access, things are a bit more tricky: This exact thing happened to me at work the other day. Unfortunately, it didn’t happen to my computer, but to a few dozen of our customer backup appliances during their scheduled upgrade/reboot. Now, while dozens out of over 60k isn’t that much, our customers rely on these devices, so it’s not acceptable to have them not boot properly.

In this short post, I’ll demonstrate how to reproduce the blinking cursor problem, and how to “fix” the MBR to ensure the computer still boots, regardless of the boot order.

1. Diagnosing a blinking cursor

Assuming that your system is booting via BIOS (not UEFI), you’ve surely encountered the infamous blinking cursor at least once:

When we encountered it a few months ago when a few of our customer devices refused to reboot, we were a bit puzzled at first. How could one possibly diagnose something like that without having physical access to the device itself? With lots of on site customer help, we quickly narrowed things down to an attached USB flash drive, and we immediately suspected that the BIOS is trying to boot from that drive.

But how was that possible? None of the partitions in the MBR on that drive was marked bootable, and there wasn’t even a bootable OS on the USB drive. How does the BIOS decide which partition to boot from anyway? Won’t it just look at the MBR partition table and pick the first bootable partition?

No. Apparently not.

As we found out quickly, the BIOS in fact doesn’t really care about the partition table at all. All it does is to list all attached drives, and go through them one by one according to the boot order. If it finds a drive with an MBR signature0x55aa at offset 0x1fe(see image below in green), it simply begins executing whatever code resides at offset0x00 of the disk (in red):

Wow isn’t that surprising? It surely was to me. I thought the BIOS was supposed to be a bit smarter than that. What that means is that really any disk in a boot position before your OS drive can render your computer unbootable if its MBR bootstrap code is not properly programmed, or has been overwritten with garbage.

In our case, we found that certain tools are apparently under the impression that the first few bytes of a disk are “unused”, because they don’t contain the partition table and use this area to store their own metadata.

Since these first 446 bytes do in fact contain the MBR bootstrap code, they are incredibly important to the BIOS during the boot process. If whatever is stored there is not valid x86 code, the BIOS will fail to execute it and drop to the blinking cursor of doom.

2. Reproducing the problem

Now that we knew what the problem is we needed a quick way to reproduce it, so that we could be certain whatever fix we developed would work. The easiest way to do this is by using a virtual machine in KVM/QEMU, a live Linux ISO and a raw disk image with “broken” or non-existent MBR bootstrap code.

First, download a live Linux distro. Something like Tiny Core Linux is more than enough for now:

# Download Tiny Core Linux
curl http://distro.ibiblio.org/tinycorelinux/9.x/x86/release/TinyCore-current.iso \
  > TinyCore-current.iso

Then, we’ll create our “broken” USB disk: To do that, we’ll create a 1 MB sparse file using truncate -s 1M and we’ll create an MBR with a single partition using fdisk. I was a bit lazy here and shortened the fdisk command by using it non-interactively like that; but you can do all of this via the interactive console: o creates the MBR, n-p-1-1 creates the partition, and w writes the changes.

# Create a sparse image file (our broken USB disk)
truncate -s 1M brokenusb.img

# Create an MBR and a partition
echo -e "o\nn\np\n1\n1\n\nw\n" | fdisk brokenusb.img

# List partitions and MBR info
fdisk -l brokenusb.img

That’s really it to reproduce the problem. If you now attach the Tiny Core ISO as a CD-ROM and the image file as a USB device, you’ll see that KVM won’t boot the Tiny Core live Linux, even though the disk with the MBR has no bootable partitions whatsoever.

# Try to boot from it. 
# This will fail with a blinking cursor!

kvm \
  -drive media=cdrom,file=TinyCore-current.iso \
  -usb -usbdevice disk:format=raw:brokenusb.img \
  -serial stdio -m 1G -cpu host -smp 2 -net none

3. Interrupt 18h to the rescue!

At first we thought we can’t really do anything about this case. We can detect it now programatically, but what could we possibly do about it?

Well, it turns out that at the very end of the BIOS Boot Specification in Appendix D.2 (yes, I read the entire 46 pages …), it says:

If an O/S is either not present, or otherwise not able to load, execute an INT 18h instruction so that control can be returned to the BIOS. Currently, hard drive boot sectors do this, but floppy diskette boot sectors execute an INT 19h instead of INT 18h. The BIOS Boot Specification defines INT 18h as the recovery vector for failed boot attempts.

Hurray! Exactly what we need. We need to tell the BIOS to try the next disk. The equivalent of “nothing to see here, move along”.

Well, let’s try it then. Let’s write an incredibly complex assembly program containing one instruction int 18h (to call interrupt 18h) and write it to the beginning of the disk:

# Create the .asm file
echo "int 18h" > skipdisk.asm

# Compile to x86 code to "skipdisk" output file
nasm skipdisk.asm

# Note:
#   If you don't have "nasm" installed, you can simply run
#   this to produce the same file: echo -en "\xcd\x18" > skipdisk

# Write to beginning of the disk
dd if=skipdisk of=brokenusb.img conv=notrunc

That’s it already. This essentially nukes the bootability of any MBR based disk, which is exactly what we needed. If you look at the disk in a hex editor like dhex, you’ll see the interrupt call at the very beginning as 0xcd18:

Now you can try to boot the VM again, using the exact same parameters, and you’ll see that this time it skips the attached USB disk, and it’ll boot the live Linux instead:

# Aaannndd let's boot it again.
# Booom! It now skips the disk, and boots Tiny Core Linux
kvm \
  -drive media=cdrom,file=TinyCore-current.iso \
  -usb -usbdevice disk:format=raw:brokenusb.img \
  -serial stdio -m 1G -cpu host -smp 2 -net none

If everything worked as expected, you should see the Tiny Core Linux boot loader screen:

X. Bonus

If you’re like me and the whole world of MBR bootstrap code is new to you, you’ll probably start experimenting with the skipdisk.asm file I provided. If you really want to, you can replace the entire bootstrap code area and do all sorts of things in there.

There is a wonderful wikibook on x86 bootloaders in assembly that I can recommend.

I found a good starting point also Dan Luedtke’s boot loader on Github. I modified his boot loader, ran nasm, and then used dd bs=446 count=1 conv=notrunc to replace the first 446 bytes of my disk. A great learning exercise.

This should post shows you how to partition a disk with GPT and make a bootable Linux system via BIOS/Legacy and UEFI.

Requirements & Assumptions

For this post, I’ll assume that you are running everything on a Debian-based system, and all commands shown must be run as root.

1. Create a minimal Linux (optional)

For this example, we’ll use a small Linux install that we can create with the following commands. This will bootstrap (debootstrap) a minimal Ubuntu 16.04 system to the chroot/ folder which we will later use to boot our system. To create a bootable system, we need a kernel and the bootloader modules and files, so let’s install linux-image-generic and grub-pc:

Bootstrapping a new Ubuntu 16.04 system (with a kernel and grub)

# Bootstrap minimal system
debootstrap --variant=minbase xenial chroot

# Install kernel and grub
for d in dev sys proc; do 
  mount --bind /$d chroot/$d
done

DEBIAN_FRONTEND=noninteractive chroot chroot apt-get install linux-image-generic grub-pc -y --force-yes
umount chroot/{dev,proc,sys}

Alternatively, you can of course use any Linux root file system, provided that it has a kernel and grub. You could, for instance, just copy your own root file system to the chroot/ folder.

2. Creating a GPT with a BIOS boot partition and an EFI System Partition

Now that we have the Linux root file system in the chroot/ folder, we’ll create a sparse file that represents our disk. You may of course do all this with a proper HDD/SSD, i.e. with /dev/sdX, but for testing things using a raw disk file is much easier. The following snippet will create 3 partitions:

• A 1 MB BIOS boot partition (of type ef02) that Grub will use to store its core image.
• A 100 MB EFI System Partition (of type ef00) formatted as FAT32 in which we will store the EFI boot image
• And a partition for our root file system (in our example formatted with ext4)

Partitioning a disk/file with GPT to boot via UEFI and BIOS

# Create sparse file to represent our disk
truncate --size 30G test.img

# Create partition layout
sgdisk --clear \
  --new 1::+1M --typecode=1:ef02 --change-name=1:'BIOS boot partition' \
  --new 2::+100M --typecode=2:ef00 --change-name=2:'EFI System' \
  --new 3::-0 --typecode=3:8300 --change-name=3:'Linux root filesystem' \
  test.img

Once this is done, you can list the partitions with gdisk -l test.img:

Listing the partitions we just created

# Now list the partitions
gdisk -l test.img 
GPT fdisk (gdisk) version 1.0.1

# ...

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048            4095   1024.0 KiB  EF02  BIOS boot partition
   2            4096          208895   100.0 MiB   EF00  EFI System
   3          208896        62914526   29.9 GiB    8300  Linux root filesystem

After the partitions are created, the EFI and root partition need to be formatted:

Formatting the root partition and the EFI System Partition

# Loop sparse file
LOOPDEV=$(losetup --find --show test.img)
partprobe ${LOOPDEV}

# Create filesystems
mkfs.fat -F32 ${LOOPDEV}p2
mkfs.ext4 -F -L "demoroot" ${LOOPDEV}p3 # << Note the label 'demoroot'

# Get rid of loop device
losetup -d ${LOOPDEV}

Note that I named the root partition demoroot (the ext4 label). This will be important later in the Grub configuration.

3. Copying the root file system, installing grub into the BIOS partition

Now that the partition to be used for our root file system is formatted, let’s copy our chroot/ directory to it and then install grub to the disk with grub-install. Because the disk is GPT formatted, grub will use the BIOS partition to install its core image:

# Loop sparse file
LOOPDEV=$(losetup --find --show test.img)
partprobe ${LOOPDEV}

# Mount OS partition and copy from chroot
MOUNTDIR=$(mktemp -d -t demoXXXXXX)
mount ${LOOPDEV}p3 $MOUNTDIR
rsync -a chroot/ ${MOUNTDIR}/

# Install grub, create config
for d in dev sys proc; do mount --bind /$d ${MOUNTDIR}/$d; done
chroot ${MOUNTDIR}/ grub-install --modules="ext2 part_gpt" ${LOOPDEV}
chroot ${MOUNTDIR}/ update-grub

# Unmount OS partition
umount $MOUNTDIR/{dev,proc,sys,}
rmdir $MOUNTDIR

# Remove loop
losetup -d ${LOOPDEV}

After this, you should be able to boot the system via BIOS from the root GPT partition. I always do that via KVM like this:

Test booting the image via KVM

kvm -drive format=raw,file=test.img -serial stdio -m 4G -cpu host -smp 2

4. Preparing the EFI partition

After you’ve verified that you can boot via BIOS, let’s make sure that we can also boot on UEFI systems. We formatted the EFI partition earlier. All that’s remaining is to create/copy the bootx64.efi image file and a valid grub.cfg to it.

Grub can create the EFI image via the grub-mkimage command using the Grub modules in /usr/lib/grub/x86_64-efi (part of the grub-efi-amd64-bin package). In order to avoid having to load anything from the file system, we include all the modules in the image. If you are not booted via EFI, you may need to install the grub-efi-amd64-bin package.

The Grub config file in the EFI partition grub.cfg is rather simple: It uses search --label demoroot to look for the root partition and then simply includes the actual grub config file via configfile ....

Let’s mount the EFI partition again, create the image and the grub config:

# Loop sparse file
LOOPDEV=$(losetup --find --show test.img)
partprobe ${LOOPDEV}

# Mount EFI partition
MOUNTDIR=$(mktemp -d -t demoXXXXXX)
mount ${LOOPDEV}p2 $MOUNTDIR

# Create EFI boot image
apt-get install grub-efi-amd64-bin -y --force-yes

mkdir -p ${MOUNTDIR}/EFI/BOOT
grub-mkimage \
  -d /usr/lib/grub/x86_64-efi \
  -o ${MOUNTDIR}/EFI/BOOT/bootx64.efi \
  -p /efi/boot \
  -O x86_64-efi \
    fat iso9660 part_gpt part_msdos normal boot linux configfile loopback chain efifwsetup efi_gop \
    efi_uga ls search search_label search_fs_uuid search_fs_file gfxterm gfxterm_background \
    gfxterm_menu test all_video loadenv exfat ext2 ntfs btrfs hfsplus udf

# Create grub config
cat <<GRUBCFG > ${MOUNTDIR}/EFI/BOOT/grub.cfg
search --label demoroot --set prefix # << Note again, this searches by the 'demoroot' label!
configfile (\$prefix)/boot/grub/grub.cfg
GRUBCFG

# Unmount and clean up
umount ${MOUNTDIR}
losetup -d ${LOOPDEV}

After that, the EFI partition should contain the two files /EFI/BOOT/bootx64.efi and /EFI/BOOT/grub.cfg. You can verify that like this:

# Loop sparse file
LOOPDEV=$(losetup --find --show test.img)
partprobe ${LOOPDEV}
 
# Mount EFI partition
MOUNTDIR=$(mktemp -d -t demoXXXXXX)
mount ${LOOPDEV}p2 $MOUNTDIR

# List things
(cd $MOUNTDIR; find .)

# Should output this:
# ./EFI
# ./EFI/BOOT
# ./EFI/BOOT/bootx64.efi
# ./EFI/BOOT/grub.cfg

# Unmount and clean up
umount ${MOUNTDIR}
losetup -d ${LOOPDEV}
rmdir ${MOUNTDIR}

That’s it. You should now be able to boot this image via UEFI. I always test this with KVM and OVMF:

Booting the image via UEFI/EFI

apt-get install ovmf -y --force-yes
kvm --bios /usr/share/qemu/OVMF.fd -net none -drive format=raw,file=test.img -serial stdio -m 4G -cpu host -smp 2

5. One script to do it all

If you are an impatient man/woman, then you may just want to run the following script to do all of this in one go. You’ll of course have to adjust it to your needs for production use.

The script can be called in two ways:

Examples on how to use the mkbiosefi script

# Makes a BIOS/UEFI bootable test.img using "demoroot" as OS partition label
# and create a minimal chroot via debootstrap in the "chroot/" folder to do so.
./mkbiosefi test.img demoroot

# Makes a BIOS/UEFI bootable test.img using "demoroot" as OS partition label,
# but re-uses the existing "chroot/" folder (this is faster!)
./mkbiosefi test.img demoroot chroot/

# If you want to use your own image file, you may create it beforehand and loop it
truncate -s 40G my.img
myloop=$(losetup --find --show my.img)
./mkbiosefi $myloop demo chroot/

# Or use a raw disk, of course
./mkbiosefi /dev/sdX demo chroot/

Here’s the script:

mkbiosefi script that can be used to create a BIOS/UEFI bootable disk/image

#!/bin/bash 

DISK=$1
BOOTLABEL=$2
ROOTDIR=$3

if [ -z "$DISK" -o -z "$BOOTLABEL" ]; then
  echo "Syntax: $0 <image|disk> <root-label> [<chroot-dir>]"
  exit 1
fi

if [ "$UID" != "0" ]; then
  echo "Must be root."
  exit 1
fi

# Exit on errors
set -xe

# Install dependencies
apt-get install -y --force-yes \
  debootstrap \
  gdisk \
  rsync \
  grub-efi-amd64-bin \
  e2fsprogs

# Create chroot (if requested)
if [ -z "$ROOTDIR" ]; then
  ROOTDIR=chroot/
  
  # Bootstrap minimal system
  debootstrap --variant=minbase xenial chroot
 
  # Install kernel and grub
  for d in dev sys proc; do mount --bind /$d chroot/$d; done
  DEBIAN_FRONTEND=noninteractive chroot chroot apt-get install linux-image-generic grub-pc -y --force-yes
  umount chroot/{dev,proc,sys}
fi

# Create sparse file (if we're not dealing with a block device)
if [ ! -b "${DISK}" ]; then
  truncate --size 30G $DISK
fi

# Create partition layout
sgdisk --clear \
  --new 1::+1M --typecode=1:ef02 --change-name=1:'BIOS boot partition' \
  --new 2::+100M --typecode=2:ef00 --change-name=2:'EFI System' \
  --new 3::-0 --typecode=3:8300 --change-name=3:'Linux root filesystem' \
  $DISK 

# Loop sparse file
LOOPDEV=$(losetup --find --show $DISK)
partprobe ${LOOPDEV}

# Create filesystems
mkfs.fat -F32 ${LOOPDEV}p2
mkfs.ext4 -F -L "${BOOTLABEL}" ${LOOPDEV}p3

# Mount OS partition, copy chroot, install grub
MOUNTDIR=$(mktemp -d -t demoXXXXXX)
mount ${LOOPDEV}p3 ${MOUNTDIR}

rsync -a ${ROOTDIR}/ ${MOUNTDIR}/

for d in dev sys proc; do mount --bind /$d ${MOUNTDIR}/$d; done
chroot ${MOUNTDIR}/ grub-install --modules="ext2 part_gpt" ${LOOPDEV}
chroot ${MOUNTDIR}/ update-grub

umount $MOUNTDIR/{dev,proc,sys,}
rmdir $MOUNTDIR

# Mount EFI partition
MOUNTDIR=$(mktemp -d -t demoXXXXXX)
mount ${LOOPDEV}p2 $MOUNTDIR
 
mkdir -p ${MOUNTDIR}/EFI/BOOT
grub-mkimage \
  -d /usr/lib/grub/x86_64-efi \
  -o ${MOUNTDIR}/EFI/BOOT/bootx64.efi \
  -p /efi/boot \
  -O x86_64-efi \
    fat iso9660 part_gpt part_msdos normal boot linux configfile loopback chain efifwsetup efi_gop \
    efi_uga ls search search_label search_fs_uuid search_fs_file gfxterm gfxterm_background \
    gfxterm_menu test all_video loadenv exfat ext2 ntfs btrfs hfsplus udf
 
# Create grub config
cat <<GRUBCFG > ${MOUNTDIR}/EFI/BOOT/grub.cfg
search --label "${BOOTLABEL}" --set prefix
configfile (\$prefix)/boot/grub/grub.cfg
GRUBCFG

umount $MOUNTDIR
rmdir $MOUNTDIR

# Remove loop device
sync ${LOOPDEV} 
losetup -d ${LOOPDEV}

echo "Done. ${DISK} is ready to be booted via BIOS and UEFI."

The current ZFS encryption implementation is not (yet) merged into the upstream repository (as of January 2017). There is a pretty big pull request which is still being reviewed, but because the feature is so incredibly cool (and because my colleague Tom Caputi developed it), I thought a sneak preview is absolutely necessary.

0. This post

This post demonstrates a feature that has not yet been released. For the demos I will focus on ZFS on Linux on an Ubuntu 16.04 based machine. The code I run is available in the official repos (links below) or in my forks (SPL & ZFS, use branch “blogpost” for both).

1. Introduction

At-rest encryption is a new feature in ZFS (zpool set feature@encryption=enabled <pool>) that will automatically encrypt almost all data written to disk using modern authenticated ciphers (AEAD) such as AES-CCM and AES-GCM.

The CLI makes it incredibly easy to enable encryption on a per dataset/volume basis (zfs create -o encryption=on <dataset>). The keys used for encryption can be inherited or are manually set for a dataset. Keys can be loaded from different sources (prompt or file) and various input formats are available (raw, hex or passphrase). Keys and key sources can be changed after the dataset/volume creation, and without re-encrypting the data (as they are never used directly).

The encryption parameters and key status of a dataset/volume are represented in various properties (encryption=<on|aes-128-gcm|...>, keysource=<raw|hex|passphrase>,<prompt|file>, keystatus=<none|available|unavailable>, pbkdf2iters=<n>).

Many normal ZFS commands are available even if the key of a dataset is not loaded, meaning that administrators can manage the pool without having to know the keys. For instance: a pool can be scrubbed (zpool scrub <pool>) without the keys, and datasets and snapshots can be listed (zfs list -rt). In future releases, zfs send and zfs recv will also work even if the key is not available.

Having built-in support for encryption at the file system level is huge. It means that you no longer have to use dm-crypt if you want to encrypt your data on disk, and you can still manage your pools even if keys are not loaded. Many thanks toTom Caputi for bringing us this incredible feature.

1.1. What’s encrypted

All important pieces are encrypted (actual data and metadata, ACLs, permissions, directory listings, …), while some things are unencrypted to allow managing pools more easily.

Here’s a listing of what’s encrypted and what’s not:

1.2. Crypto Details

Note: This section describes the nitty gritty crypto details. You can safely skip it if you just want to use the feature.

Crypto concepts are always a bit hard to explain without confusing everyone. Tom has done an excellent job explaining the ZFS encryption crypto concept in his talk and it is visualized very nicely in his slides (PDF; or on Google Drive: original, mirror).

If you don’t have the time to watch the entire talk, let me try to summarize the concepts one of his slides:

Normal / non-dedup case: Before the plaintext block data (or metadata) is written, it is encrypted using AES (in CCM or GCM mode, depending on the -o encryption=.. property) with a 128/192/256-bit encryption key (default is AES-CCM-256). The 96-bit initialization vector (IV) used for CCM/GCM is randomly generated using the standard linux PRNG, and it is never reused. The encryption key itself is derived from the encrypted master key (see below) using the key derivation function HKDF. The 64-bit salt used for HKDF is randomly generated (using the above mentioned PRNG) and stored with the encryption key in a volatile salt cache. The encryption key is reused (for performance reasons) until it goes stale.

Dedup case: If deduplication is enabled, the algorithm behaves slightly differently, because it has to produce the same ciphertext for the same plaintext (given the same master key). To achieve that, the salt and the IV are not randomly generated, but instead a 160-bit HMAC of the plaintext is used: the first 64 bits are used as the salt, the remaining 128 bits are used as IV. The 256-bit HMAC key is randomly generated (using above mentioned PRNG), and stored alongside the master key.

Master key: The master key is randomly generated (using above mentioned PRNG) and it is never exposed to the user directly. Instead, the master key is encrypted (with the same cipher and mode with a 256-bit key) using a user provided wrapping key. This wrapping key is provided via a file (as hex or raw, see -o keysource=.. property) or via a password prompt. If a passphrase is supplied by the user, the wrapping key is derived using the password-based key derivation function PBKDF2 (using 100k iterations by default, or whatever you specify in the property -o pbkdf2iters=..).

If you want to know more, I highly suggest watching Tom Caputi’s ZFS encryption talk, or reviewing the slides (PDF; or on Google Drive: original, mirror).

https://www.youtube.com/watch?v=frnLiXclAMo

2. Using ZFS encryption

Using the encryption feature is pretty simple. All the relevant commands and properties are described in great detail in the ZFS man page (man zfs), but here’s an excerpt of what you need to know.

2.1. Enabling the feature on the pool

Assuming you have installed a version of ZFS with encryption installed (if not, follow the steps in sectioncompile and install at your own risk), you need to turn it on for your pool. I’ll create a test pool called testpool for this post:

Creating a test pool, and turning on the encryption feature

# Creating a test pool
$ truncate -s 1G block
$ zpool create testpool $(pwd)/block

# Turning on encryption (you NEED a ZFS version that supports this!)
$ zpool set feature@encryption=enabled testpool

2.2. Creating an encrypted dataset

Once you’ve enabled the feature on the pool, you can create encrypted datasets and volumes. To do that, you need to pass the two properties -o encryption=.. -o keysource=.. to the zfs create command. Depending on your preferences, you may also pass -o pbkdf2iters=..:

Creating an encrypted dataset with ‘zfs create’

$ zfs create \
    -o encryption=<off | on | aes-128-ccm | aes-192-ccm | aes-256-ccm | aes-128-gcm | aes-192-gcm | aes-256-gcm> \
    -o keysource=<raw | hex | passphrase>,<prompt | file://...> \
    -o pbkdf2iters=<n> \
    <dataset>

The -o encryption=.. property controls the ciphersuite (cipher, key length and mode). The default is aes-256-ccm, which is used if you specify -o encryption=on.

The -o keysource=.. property controls what format the encryption key will be provided as and where it should be loaded from. The key can be formatted as raw bytes, as hex representation or as a user password. It can be provided via a user prompt which will pop up when you first create it, or when you mount the dataset (zfs mount) or load the key manually (zfs key -l). Unless you want to automate things, -o keysource=passphrase,prompt seems like a good option.

The -o pbkdf2iters=.. property is only used if a passphrase is used (-o keysource=passphrase,..). It controls the iterations of PBKDF2. Higher is better as it slows down potential dictionary attacks on the password. The default is -o pbkdf2iters=100000.

Here are a few examples of how to create encrypted datasets and volumes (ZVOLs):

Creating an encrypted dataset, using the defaults:

Creating an encrypted dataset, using the defaults

$ zfs create \
    -o encryption=on \
    -o keysource=passphrase,prompt \
    testpool/enc1
  
# This will ask you to enter/confirm a password.

Creating an encrypted child dataset, which inherits all parameters and keys from its parent:

Creating an encrypted child dataset, inheriting params and keys

$ zfs create testpool/enc1/encinherit

Creating an encrypted dataset, using AES/GCM with 128-bit key loaded from a file (encoded as hex):

Creating an encrypted dataset, and loading the key from a file

  
$ echo 0000111122223333444455556666777788889999AAAABBBBCCCCDDDDEEEEFFFF > /dev/shm/enc2key
$ zfs create \
    -o encryption=aes-128-gcm \
    -o keysource=hex,file:///dev/shm/enc2key \
    testpool/enc2
# No prompts for a password!

Creating an encrypted dataset, using AES/GCM with a 256-bit key loaded from a file (not encoded):

$ head -c 32 /dev/urandom > /dev/shm/enc3key
$ zfs create \
    -o encryption=aes-256-gcm \
    -o keysource=raw,file:///dev/shm/enc3key \
    testpool/enc3

Creating an encrypted ZFS volume (ZVOL), using the defaults with one million PBKDF2 rounds:

$ zfs create \
    -V 10M \
    -o encryption=on \
    -o keysource=passphrase,prompt \
    -o pbkdf2iters=1000000 \
    testpool/enc4
    
# This will ask you to enter/confirm a password.    

2.3. Reading the encryption properties

Once you’ve created a dataset or volume, you can query its encryption properties like you normally would:

Querying the encryption parameters

$ zfs get -p encryption,keystatus,keysource,pbkdf2iters
testpool/enc1             encryption   aes-256-ccm                  local
testpool/enc1             keystatus    available                    -
testpool/enc1             keysource    passphrase,prompt            local
testpool/enc1             pbkdf2iters  100000                       local
testpool/enc1/encinherit  encryption   aes-256-ccm                  inherited from testpool/enc1
testpool/enc1/encinherit  keystatus    available                    -
testpool/enc1/encinherit  keysource    passphrase,prompt            inherited from testpool/enc1
testpool/enc1/encinherit  pbkdf2iters  100000                       inherited from testpool/enc1
testpool/enc2             encryption   aes-128-gcm                  local
testpool/enc2             keystatus    available                    -
testpool/enc2             keysource    hex,file:///dev/shm/enc2key  local
testpool/enc2             pbkdf2iters  1000000                      local
testpool/enc3             encryption   aes-256-gcm                  local
testpool/enc3             keystatus    available                    -
testpool/enc3             keysource    raw,file:///dev/shm/enc3key  local
testpool/enc3             pbkdf2iters  0                            -
testpool/enc4             encryption   aes-256-ccm                  local
testpool/enc4             keystatus    available                    -
testpool/enc4             keysource    passphrase,prompt            local
testpool/enc4             pbkdf2iters  1000000                      local

2.4. Importing a pool, mounting datasets and loading keys

If a dataset is encrypted, the read-only property keystatus represents the status of the key, and thereby also whether the dataset can be used (mounted, written to, read from …). It can be either off (unencrypted dataset), available (the key is loaded) or unavailable (the key is not loaded).

When a pool is imported using zpool import, encrypted datasets are left unmounted, because their keys are not automatically loaded. Only if the -l option is passed will encrypted datasets be loaded (if they can):

Importing a pool with -l to load the keys

$ zpool import testpool -d . -l
Enter passphrase for 'testpool/enc1': (enter password)
Key load error: Failed to open key material file  # << Key file does not exist!
Enter passphrase for 'testpool/enc4': (enter password)

Instead of using zpool import -l ..., you can manually load the keys for individual datasets and volumes using zfs key -l:

Manually loading and unloading a key using ‘zfs key -l’

$ zfs key -l testpool/enc4
Enter passphrase for 'testpool/enc4': (enter password)

$ zfs mount testpool/enc4
# Does not prompt. Just mounts!

If zfs mount is called on an encrypted dataset with unavailable key, it will prompt you:

Mounting an encrypted dataset with unavailable key will prompt

$ zfs mount testpool/enc4
Enter passphrase for 'testpool/enc4': (enter password)

Unloading a key of a mounted dataset won’t work, because it’s still in use. The dataset has to be unmounted first:

Unloading a key only works if the dataset is not mounted

# Does not work because dataset is mounted
$ zfs key -u testpool/enc4
Key unload error: Dataset is busy.

# Works like a charm
$ zfs umount testpool/enc4
$ zfs key -u testpool/enc4

That’s essentially all the magic. If you want to know more, I suggest reading the ZFS man page (man zfs).

3. Compile and install

If you want to try the current implementation (before it is released), here are a few steps to compile and install it yourself on an Ubuntu 16.04-based system. Other systems will be very similar, but not identical. You can consult the Building ZFS wiki page for details.

Warning: Please be sure to only perform these steps on a test machine or a throw-away VM, because this will replace the ZFS kernel modules.

First, install all the build dependencies:

Installing build dependencies

apt install libtool zlib1g-dev attr uuid-dev libblkid-dev libattr1-dev autoconf

Once that’s done, compile and install SPL using the steps below. All relevant ZFS encryption pulls have been merged (as of January 2017), so this should “just work”. If it doesn’t (e.g. because the code has changed; you may be reading this in the future …), you may want to use my forked version instead (see SPL and ZFS, use branch “blogpost” for both):

Compiling and installing SPL

# Compile and install SPL
#
# If this does not work anymore, use my forked version instead:
#   $ git clone https://github.com/binwiederhier/spl.git
#   $ git checkout blogpost
#
git clone https://github.com/zfsonlinux/spl.git
cd spl
./autogen.sh
./configure
make
make install
cd ..

Next, compile and install ZFS using Tom’s fork. If the ZFS encryption pull request has been merged, you may just want to use the upstream master branch:

Compile and install ZFS from Tom’s fork (with the encryption code)

# Compile and install ZFS (Tom's fork)
#
# If this does not work anymore, use my forked version instead:
#   $ git clone https://github.com/binwiederhier/zfs.git
#   $ git checkout blogpost
#
git clone https://github.com/tcaputi/zfs.git
cd zfs
./autogen.sh
./configure --prefix=/usr  # << Don't forget the --prefix
make install
cd ..

Now the ZFS modules should be built in /lib/modules/$(uname -r)/extra, so all you have to do is load them. Be sure to remove the old modules first:

Removing the old modules, inserting the new modules

# Remove existing modules (repeat until successful!)
modprobe -r splat
modprobe -r zavl
modprobe -r zcommon
modprobe -r zunicode
modprobe -r znvpair
modprobe -r icp
modprobe -r spl
modprobe -r zfs

# Insert newly compiled modules (all of these must succeed!)
cd /lib/modules/$(uname -r)/extra
insmod avl/zavl.ko 
insmod unicode/zunicode.ko 
insmod spl/spl.ko
insmod nvpair/znvpair.ko 
insmod zcommon/zcommon.ko 
insmod icp/icp.ko 
insmod zfs/zfs.ko

If that succeeded, be sure to yell “hooray” before you move on, because these steps took me a while to get right when I did it for the first time.

You can now use the feature (as described above).

4. FAQ

4.1. Is deduplication supported?

Yes, it is supported. However, there is some information that is leaked due to the nature of deduplication. See more in Tom’s talk.

4.2. Can I change the password? Will data be re-encrypted?

Yes, the password can be changed. The data does not get re-encrypted, because the password is merely used to decrypt a master key.

4.3. Does it work with TPM?

No, not yet.

5.Links

• OpenZFS Roadmap (with ZFS encryption on it)
• Hacker News discussion about ZFS encryption
• GitHub pull request for ZFS encryption (ZFS on Linux)
• GitHub pull request for ZFS encryption (OpenZFS)
• Talk at OpenZFS Developer Summit by Tom Caputi about ZFS encryption (video)
• Instructions on how to compile / build ZFS
• Slides used for the talk at the OpenZFS Developer Summit (PDF; or on Google Drive: original, mirror)

It consists of the following tools: zfsu tx (aka zfstx) maintains a mirror of a ZFS pool over the network. zfsu ret (aka zfsret) is a simple script to apply local retention (destroy snapshots) of a file system and its snapshots. zfsu res (aka zfsres) is a script to resilver a slow mirror, e.g. a HDD disk if mirrored with a SSD.

0. Code available on GitHub

All the code for this post is available in a zfsu Project on GitHub. Feel free to poke around there and/or steal bits and pieces from it.

1. Installation

I host a package of the scripts in my Debian repository, so if you are Debian/Ubuntu based, feel free to install it like this:

Installing zfsu on Debian/Ubuntu

# Add my repository
wget -qO - http://archive.philippheckel.com/apt/Release.key | sudo apt-key add -
sudo sh -c "echo deb http://archive.philippheckel.com/apt/release/ release main > /etc/apt/sources.list.d/archive.philippheckel.com.list"
sudo apt-get update

# Install zfsu
sudo apt-get install zfsu

If you’re using a different system, simply download and place these scripts in your PATH.

2. zfsu tx (aka zfstx)

zfstx lets you pull ZFS snapshots from a remote host into the local zpool, meaning that you need to install this script on the backup machine itself, not on the machine that’s being backed up. While this may seem strange at first, it’s great if you’re backing up many different machines, because you can manage the schedule and logic in one place.

The tool was originally developed by Jaako as part of his zfs_transfer tool. I merely adjusted it a little and packaged it for my needs. Many thanks to him for this great tool.

2.1. Usage

Usage of the zfstx tool

$ zfsu tx
Usage: zfstx [OPTIONS] <remote-host>:<remote-fs> <local-fs>
Pull ZFS snapshots from a remote host into the local zpool.

Arguments:
  <remote-host>            - Remote host, e.g. myhost
  <remote-fs>              - Filesystem on remote host, e.g. tank/home
  <local-fs>               - Filesystem on local host, e.g. backuppool/myhost/home

Options:
  -k, --keep <count>       - preserved history length
  -b, --mbuffer <bufsz>    - mbuffer buffer size, default 4G
  -p, --port <port>        - custom SSH port, default 22
  -P, --no-pigz            - disable pigz
  -n, --dry-run            - Don't apply changes, just print (experimental)

2.2. Examples

Example usage of zfstx

$ zfstx platop:tank/home/pheckel tank/backups/platop/home/pheckel
  # Pull all (missing) snapshots from host "platop" into the local pool "tank"
  # and don't apply any retention.

$ zfstx --keep 5 platop:tank/home/pheckel/vms tank/backups/platop/home/pheckel/vms
  # Pull all (missing) snapshots from host "platop" into the local pool "tank"
  # and only keep the 5 latest snapshots locally.

3. zfsu ret (aka zfsret)

zfsret is a simple script to apply local retention (destroy snapshots) of a file system and its snapshots. Its meant to complement the zfstx utility to only maintain a certain number of snapshots on the backup host. It can also be used to locally prune snapshots on demand, or regularly.

The retention logic is pretty simple as you can only specify how many of the last snapshots to keep. There is no sophisticated logic (yet) to keep X per week or Y per month.

3.1. Usage

Usage of the zfsret tool

$ zfsu ret
Usage: zfsret [OPTIONS] <local-fs> <keep>
Destroy local ZFS snapshots for a specific filesystem.

Arguments:
  <local-fs>               - Filesystem on local host, e.g. backuppool/myhost/home
  <keep>                   - Number of snapshots to keep

Options:
  -n, --dry-run            - Don't apply changes, just print

3.2. Examples

Example usage of zfstx

$ zfsu ret tank/home/pheckel 10
  # Destroy all but 10 snapshots of filesystem tank/home/pheckel
  # This is not recursive (no -r)!

4. zfsu res (aka zfsres)

zfsres is a tool specifically written for a ZFS mirror/RAID1 between a slow/HDD and a fast/SSD disk. zfsres will regularly resilver the slow HDD and thereby not make the array slow for normal write/read operations, but still allow for a RAID1 configuration of two disks of vastly different speeds.

This tool covers a pretty exotic use case. I wrote it because my laptop has a SSD and a HDD, and I wanted to keep a second copy on the HDD. Note that this tool only makes sense in this scenario.

4.1. Usage

Usage of the zfsret tool

$ zfsu res
Usage: zfsres <pool> <slow-mirror>
Enable slow mirror(s) and wait for them to be resilvered and exit.

Arguments:
  <pool>           - Name of the ZFS pool, e.g. tank
  <slow-mirror>    - Description of a slow mirror device, e.g. wwn-0x50004cf20c41a05b

4.2. Examples

Example usage of zfstx

$ zfsres tank wwn-0x50004cf20c41a05b
  # Will online the disk wwn-0x50004cf20c41a05b and wait for resilvering to 
  # complete, and then offline the disk again.

5. Example: My setup

I use these scripts to back up 5 different machines, some of which have ZFS file systems, and others that do not. My backup hub’s ZFS pool looks like this (shortened):

‘zfs list’ output of my backup hub ZFS pool (shortened)

tank/backups/laptop1  1.09T  1.29T  9.65G  /tank/backups/laptop1
tank/backups/box       207G  1.29T  9.51G  /tank/backups/box
tank/backups/box2     1.07G  1.29T   868M  /tank/backups/box2
tank/backups/laptop2   893G  1.29T  14.0G  /tank/backups/laptop2
tank/backups/blurp     419G  1.29T  6.60G  /tank/backups/blurp

Backups of these machines are tiggered in a cronjob by a custom wrapper script called gobackup, which uses the zfsu utilities. It pretty much looks like this (representative excerpt):

Example usage of zfstx and zfsret in my setup

#!/bin/bash

# Example 'laptop1': a system that does not use ZFS as a file system
# We just rsync from there and manage snapshots and retention locally
laptop1_root() { 
  (
    flock -n -x 9 || { echo "Already running."; return; }
    rsync \
      -e "ssh -o ConnectTimeout=15" \
      -av \
      --delete \
      --delete-excluded \
      --one-file-system \
      --compress \
      root@laptop1:/ \
      /tank/backups/laptop1/ \
    && zfs snapshot tank/backups/laptop1@`date +%y%m%d%H%M` \
    && zfsret tank/backups/laptop1 100
  ) 9> /var/lock/gobackup_laptop1_root
}

# Example 'laptop2': a system that does use ZFS. A single command
# pulls the latest snapshots to the backup hub.
laptop2_home() { 
  zfstx \
    --mbuffer 1G \
    --keep 9000 \
    laptop2:tank/home/pheckel \
    tank/backups/laptop2/home/pheckel      
}

laptop1_root
laptop2_home

It took a bit of fiddling in the beginning, but once you know how it works, it’s a piece of cake. This short post shows you how to move your existing Linux installation to ZFS on root (preferably Ubuntu 16.04+ based, may work for others).

1. Requirements and Assumptions

For this example, we’re assuming that you’re running an Ubuntu 16.04+ based operating system (Ubuntu, Kubuntu, Xubuntu, Lubuntu, Linux Mint, …), and that you have another partition (to be used for the ZFS pool) with at least the amount of disk space that your current root partition has. If that is not the case, you can follow step 3 to make room.

2. This example: Kubuntu 16.04

In this example, I’ll be using a regular Kubuntu 16.04 installation. This could be your laptop or workstation, with any derivative of Ubuntu installed.

3. Creating the ZFS pool partition (optional)

Optional (if you have no spare partition): As you can see in the screenshot above, lsblk shows us that the machine has no extra partition for a new ZFS pool (yet), so we’ll have to split the root partition to make room for the ZFS pool partition. If you already have a disk/partition that you want to use as a pool, you can skip this step.

To modify your root partition, you need to first boot an Ubuntu/Kubuntu/Mint Live CD/USB, so you can fiddle with the partition layout in gparted or the KDE Partition Manager. I have opted to boot into an Ubuntu Live CD (to make it clear that this is not your live system).

To split your root partition,

• Open gparted or KDE Partition Manager
• Select your root partition, right click and select “Resize”, enter the target size and hit “Apply”
• Then create a new (unformatted) partition in the empty space after the old root partition
• Click Apply (Warning: Back up your system before you do this. There is a chance of data loss!)

It should look something like this (if you’re using gparted):

After you hit “Apply” and after you rebooted into your actual system, your disk layout changed and you have a new partition (here: /dev/sda3). This is the partition we will use to create the ZFS pool:

4. Creating the root ZFS pool, and copying your OS

Now that we have a free partition for our new pool, let’s first install ZFS (if you don’t have it already):

Install ZFS and its initramfs module

$ apt install zfs-dkms zfs-initramfs

Then, identify the newly created partition and create the root pool:

Identifying the new partition with lsblk

$ lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda      8:0    0   20G  0 disk 
├─sda1   8:1    0  8.8G  0 part /
├─sda2   8:2    0    1K  0 part 
├─sda3   8:3    0  9.2G  0 part             # <<<<< Our new partition! It'll likely be different for you!
└─sda5   8:5    0    2G  0 part [SWAP]

Now create the root pool and the root dataset you want to use. I usually call my pools tank and put my systems in tank/os/<osname>, but this is really up to you:

Creating a pool and the ZFS datasets

$ zpool create tank /dev/sda3     # Warning, choose your partition here! ZFS does not forgive typos!
$ zfs create tank/os
$ zfs create tank/os/kubuntu1604

# Alternatively, use the by-id symlink like so:
# $ zpool create tank /dev/disk/by-id/....

After you created the new dataset, it is auto-mounted by ZFS at /tank/os/kubuntu1604. You may now start copying your running system to this location:

Copying the current system to the ZFS dataset

$ rsync -a --one-file-system / /tank/os/kubuntu1604/

This may take a while, depending on your disk I/O speed and load; and it may even throw some warnings. Usually these warnings can be safely ignored if they are about files in /proc.

5. Make system bootable

Now that we have the dataset of our new root file system (tank/os/kubuntu1604), we need to tell our bootloader (here: grub2) that it should boot into the new system. To do that, we need to chroot into the system and run update-grub and grub-install:

Chrooting into the new system, and updating grub

$ cd /tank/os/kubuntu1604
$ mount --bind /dev dev
$ mount --bind /proc proc
$ mount --bind /sys sys
$ mount --bind /run run
$ chroot .

# You are now in the new system. Your current "/" points to "/tank/os/kubuntu1604"
# All the following commands will apply only to that system.

$ update-grub
Generating grub configuration file ...
Warning: Setting GRUB_TIMEOUT to a non-zero value when GRUB_HIDDEN_TIMEOUT is set is no longer supported.
Found linux image: /boot/vmlinuz-4.4.0-31-generic
Found initrd image: /boot/initrd.img-4.4.0-31-generic
Found memtest86+ image: /os/kubuntu1604@/boot/memtest86+.elf
Found memtest86+ image: /os/kubuntu1604@/boot/memtest86+.bin
Found Ubuntu 16.04.1 LTS (16.04) on /dev/sda1
done

$ grub-install /dev/sda                # Note: This should be your root disk (not partition!)
Installing for i386-pc platform.
Installation finished. No error reported.

$ exit
$ reboot

That’s it. During the reboot, if you press ESC, you can see that the grub menu will show the new system as “Ubuntu”, and the old system as “Ubuntu 16.04.1 LTS (16.04) on /dev/sda1”. That means if you’re having trouble with your ZFS on Root based system, you can always go back to your old system:

Once you booted, you can verify that you’re indeed running on ZFS with zfs list or mount:

6. Snapshot your system regularly, accessing snapshots

You can (and should) now snapshot your root file system, and you can even roll back if you really want to. The first thing you should do is set up a cronjob to regularly snapshot your system:

Snapshotting your root file system regularly

$ crontab -e # as root!

# Then add a line line this:
0  20     * * * zfs snapshot tank/os/kubuntu1604@`date +\%y\%m\%d\%H\%M`

Once you’ve done that, you can always restore individual files by accessing the ZFS directory in /.zfs/snapshot/<snapshotname>, like this:

Restoring a file from an old snapshot

$ cp /.zfs/snapshot/1612221520/etc/hosts /etc/hosts

Or, if you really messed up, you can also completely roll back to an old snapshot (use with caution, as running programs may expect the disk contents to be different!):

This is a short tutorial describing how I did it. It’s really not rocket science, so don’t expect too much.

1. Requirements

You need:

• A publicly accessible server with root access

• A domain and control to delegate DNS zones

• About 30min of set up time

In this example/post:

• Our new DNS server will be at 99.88.77.66

• Its name will be dyndns.ns.heckel.xyz

• The new dynamic DNS zone will be named dyndns.heckel.xyz

2. Install PowerDNS & MySQL

Assuming that you’re running Ubuntu 16.04+, install MySQL (apt install mysql-server), then install PowerDNS as per instructions on their repo site. Note that the Ubuntu 16.04 upstream version is broken, as per this issue, so be sure to install at least version 4.0.1.

Then install the PowerDNS MySQL backend via apt install pdns-backend-mysql.

If everything goes as planned, this will set up the pdns database for you automatically (including user and password), so you don’t have to manually do that. If that does not happen, configure it in /etc/powerdns/pdns.d/pdns.local.gmysql.conf, so that it looks something like this:

launch+=gmysql

gmysql-host=localhost
gmysql-port=
gmysql-dbname=pdns
gmysql-user=pdns
gmysql-password=supersecretpass
gmysql-dnssec=yes

Once you’ve configured it, you may want to start the service to see if it is working (systemctl start pdns.service). Given that it is not properly configured with a zone, it may actually fail here. Check journalctl -u pdns.service for details.

3. Add your DNS zone

With the MySQL backend, you can configure your zone completely via SQL tables (duh!). For our dynamic DNS server, the only relevant tables are pdns.domains and pdns.records. If you want to know more about the other tables, a detailed documentation of the schema and plugin can be found on the PowerDNS plugin documentation page.

For this example, we’re configuring dyndns.heckel.xyz as our DNS zone, so let’s add this domain to the pdns.domains table:

Inserting the DNS zone we want to use for the dynamic DNS server

INSERT INTO pdns.domains SET id=1, name='dyndns.heckel.xyz'

Once we’ve done this, the pdns.domains table should look like this (assuming you’re looking at it via phpMyAdmin):

If you know DNS, you know that there are different types of records that can be added to a zone. Our dynamic DNS zone is pretty simple. We just need two records to get started. Let’s add a SOA record to declare that this server is authoritative for our dynamic DNS domain (here: dyndns.heckel.xyz) and our first dynamic A record (here: let’s assume our home network’s public IP address is 12.11.121.221):

Inserting the SOA record for the DNS zone and a first dynamic record

INSERT INTO pdns.records SET 
  domain_id=1, 
  name='dyndns.heckel.xyz', 
  type='SOA', 
  content='dyndns.ns.heckel.xyz noreply.heckel.xyz 1483141082 60 60 60 60',
  ttl=60, 
  change_date=unix_timestamp();

INSERT INTO pdns.records SET 
  domain_id=1, 
  name='dyndns.heckel.xyz', 
  type='A', 
  content='12.11.121.221', 
  ttl=60, 
  change_date=unix_timestamp();

If you add other A records, this may look like this:

Side note: PowerDNS looks at the change_date column (not shown in screenshot) to determine whether its cache needs to be updated. Be sure to always update that column if you change a record, and also always update the SOA record serial (third part of the record, see above ... 1483141082 ...) when anything in the zone has changed. You’ll get outdated/invalid results if you don’t do this.

That’s all you need to configure PowerDNS. After restarting the service (systemctl restart pdns), you should be able to query the DNS server for the two records that you added:

Querying the DNS server locally for the two records we added

$ dig SOA dyndns.heckel.xyz @localhost +short
dyndns.ns.heckel.xyz. noreply.heckel.xyz. 1483141082 60 60 60 60

$ dig A dyndns.heckel.xyz @localhost +short
12.11.121.221

4. Delegating the domain

So far we can only verify that our zone works locally, but we haven’t made the dynamic DNS domain known to the world. If you ask for any domain *.dyndns.heckel.xyz via DNS, the nameserver of your domain will still feel responsible. To tell it to delegate all requests for dyndns.heckel.xyz to the our server dyndns.ns.heckel.xyz (aka 99.88.77.66), we need to change some records in our domain’s DNS entries.

My domain is hosted with Hosteurope, so I had to log into their domain management system and add two entries:

The two records you see are to map an IP address to the DNS server domain (dyndns.ns.heckel.xyz A 99.88.77.66) and to actually delegate the subdomain to this DNS server (dyndns.heckel.xyz NS dyndns.ns.heckel.xyz).

Once these records are live (this may take a few hours), you should be able to resolve queries for the zone from any computer. It’s easiest to do this via the requests we ran before, but this time run it from your workstation (instead of the server itself):

Querying the DNS server from a workstation

$ dig SOA dyndns.heckel.xyz +short
dyndns.ns.heckel.xyz. noreply.heckel.xyz. 1483141082 60 60 60 60

$ dig A dyndns.heckel.xyz +short
12.11.121.221

5. Updating the DNS zone regularly

Now that the zone is live and can be queried from anywhere, all you need to do is keep it up-to-date. How you do that is of course up to you. I’ve set up a JSON-RPC API on my server (I also wrote a guide on how to do that) and I’m updating it through that.

Each of my clients/machines runs a simple script every 2 minutes in a cron job:

Updating the host IP address in a cronjob

#!/bin/bash
curl \
  -u worklaptop:supersecretkey \
  -d '{"jsonrpc":"2.0","id":1,"method":"hosts/update","params":{"host":"worklaptop"}}' \
  https://heckel.xyz/api.php

On the server, the API endpoint essentially runs these two queries for every host update request:

Updating the records table on the server side

$dbh = new PDO(...);

// Update A record for '<host>.dyndns.heckel.xyz'
$stmt = $dbh->prepare("
    update records 
    set change_date=unix_timestamp(), content=:content
    where type='A' and name=:name
");

$stmt->execute(array(
    'name' => "$host.dyndns.heckel.xyz",
    'content' => $_SERVER['REMOTE_ADDR']
));

// Update SOA record timestamp for 'dyndns.heckel.xyz'
$serial = time();
$dbh->query("
    update records 
    set change_date=unix_timestamp(), content='dyndns.ns.heckel.xyz noreply.heckel.xyz $serial 60 60 60 60'
    where type='SOA' and name='dyndns.heckel.xyz'
");

That’s essentially it. Your dynamic DNS server is ready to be used. PowerDNS really makes things incredibly easy for us. As always, leave notes and questions in the comments section.

In this post, I’d like to demonstrate how to set up a PHP based JSON-RPC API, with authentication, validation and logging.

0. Code available on GitHub

All the code for this post is available in a JSON-RPC Demo Project on GitHub. Feel free to poke around there and/or steal bits and pieces from it.

To try the examples, check out the code and run this:

git clone https://github.com/binwiederhier/json-rpc-demo
cd json-rpc-demo
composer install
php -S localhost:8888 -t web

1. Demo API: Managing devices (e.g. phones, computers)

For this blog post, we’ll create a simple Device Management API to list and add devices (e.g. phones, computers, and such) using two endpoints called devices/add and devices/listAll. The endpoints will be simple stub implementations, but we’ll still see how easy it is to create and manage endpoints.

Here’s an example JSON-RPC API request and its response to list all devices:

// 2. Request to list all devices, sorted by id
{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "devices/listAll",
    "params": {
        "sortBy": "id"
    }
}

// The corresponding response returns a list
{
    "jsonrpc": "2.0",
    "id": 1,
    "result": [
        {"name": "Philipp PC", "id": 1, "type": "pc"},
        {"name": "Phil Phone", "id": 2, "type": "phone"},
        {"name": "Phil Washing Machine", "id": 3, "type": "other"}
    ]
}

2. JSON-RPC core library (php-json-rpc)

The core JSON-RPC library php-json-rpc is written by Spencer Mortensen. It’s a great piece of software that implements the core specification of JSON-RPC 2.0.

2.1. Sample code

At the center of it, the JsonRpc\Server class provides a way to encode and decode messages according to JSON-RPC standards. All you have to do is provide your own JsonRpc\Evaluator implementation to use it:

$evaluator = new DevicesEvaluator();
$server = new JsonRpc\Server($evaluator);

header('Content-Type: application/json');
$message = file_get_contents('php://input');

echo $server->reply($message); // {"jsonrpc":"2.0","id":1,"result":3}

An Evaluator basically translates JSON-RPC methods and parameters to PHP methods/arguments and executes the method. Your implementation must only provide a method evaluate($method, $arguments). It might (but should not) look as simple as this:

class DevicesEvaluator implements JsonRpc\Evaluator
{
    // ...

    public function evaluate($method, $arguments)
    {
        if ($method === 'devices/add') {
            return $this->devices->add($arguments['id'], $arguments['name'], $arguments['type']);
        } else if ($method === 'devices/listAll') {
            return $this->devices->listAll($arguments['sortBy']);
        }else {
            throw new Exception\Method();
        }
    }
}

2.2. Usage

Once we’ve implemented the endpoints/methods, we’ll be able to add and list devices via HTTP(S). Here’s an example via curl:

# Adding a device
$ curl \
    -d '{"jsonrpc":"2.0","id":1,"method":"devices/add","params":{"name":"Philipp PC","type":"pc","id":1}}' \
    http://localhost:8888/example0/api.php
{"jsonrpc":"2.0","id":1,"result":{"id":1,"name":"Philipp PC","type":"pc"}}

Of course you can also use your own JSON-RPC client, or you can use JavaScript/jQuery (see below). I’m using curl here because it’s super easy.

2.3. More examples

Find a working demo of the php-json-rpc library in the examples folder of the library or in example 0 of the JSON-RPC demo project.

3. Method and parameter mapping (php-json-rpc-simple)

While we can certainly implement all use cases with the core library, having to provide the mapping yourself (usually with a giant switch-case) can be tedious. The php-json-rpc-simple library provides a way to do that mapping for you. It offers a simple implementation of the JsonRpc\Evaluator interface for you and maps JSON-RPC method and params to a corresponding PHP class, method and arguments.

3.1. Sample code

To use the mapping magic of the php-json-rpc-simple library, alter the API server logic to automatically map JSON-RPC methods to a PHP class/method inside the Demo\Api\Endpoint namespace (see example1/api.php):

$evaluator = new Simple\Evaluator(new Simple\Mapper('Demo\\Api\\Endpoint\\'));
$server = new JsonRpc\Server($evaluator);

header('Content-Type: application/json');
$message = file_get_contents('php://input');

echo $server->reply($message);

Since the mapping is done for you, the endpoint methods only contain the actual logic and/or library calls (see src/Api/Endpoint/Devices.php):

namespace Demo\Api\Endpoint;

class Devices
{   
    public function listAll($sortBy = 'name')
    {
        return array(..);
    }
    
    // ...
}

3.2. Usage

The usage is not very different from the example above. This time, however, the mapping is performed by the library:

# Listing all devices
$ curl \
    -d '{"jsonrpc":"2.0","id":1,"method":"devices/listAll","params":{"sortBy":"id"}}' \
    http://localhost:8888/example1/api.php
{"jsonrpc":"2.0","id":1,"result":[{"id":1,"name":"Marian-PC","type":"pc"},{"id":1,"name":"Philipp PC","type":"pc"}]}

That also means, of course, that invalid or missing parameters will be detected by the library and a JSON-RPC error is returned:

# Invalid parameters return an error
$ curl \
    -d '{"jsonrpc":"2.0","id":1,"method":"devices/add","params":{"INVALID":"1"}}' \
    http://localhost:8888/example1/api.php
{"jsonrpc":"2.0","id":1,"error":{"code":-32602,"message":"Invalid params"}}

3.3. More examples

Find a working demo of the php-json-rpc-simple library in example 1 of the JSON-RPC demo project.

4. Validation of parameters (php-json-rpc-validator)

Another tedious thing in API development is parameter validation. User input can be invalid or even malicious, an API (or any web page really) must validate the input parameters. The php-json-rpc-validator library offers annotation-based validation of parameters.

4.1. Sample code

Normally, a publicly available endpoint method must check the arguments that are passed to it. Code like this is not uncommon:

class Devices
{   
    public function listAll($sortBy = 'name')
    {
        if ($sortBy !== 'name' && $sortBy !== 'type' && $sortBy !== 'id') {
            throw new Exception('Illegal argument');
        }

        // Actual API code
        // ...
    }
    
    // ...
}

Instead of this annoying and polluting validation code, we can now define the constraints of each method argument using the @Validate annotation. This annotation is based on Symfony’s @Collection annotation and supports a variety of constraints. Examples include @Assert\NotBlank, @Assert\NotNull, @Assert\EqualTo, @Assert\Regex, @Assert\GreaterThan, and may more.

class Devices
{   
    /**
     * @Validate(fields={
     *   "sortBy" = { @Assert\Regex("/^(id|name|type)$/") }
     * })
     */
    public function listAll($sortBy = 'name')
    {        
        // Actual API code
        // ...
    }
}

To add validation support to your API server file (in our examples: api.php), all you need to do is wrap the existing JsonRpc\Evaluator in a Validator\Evaluator. After that, the API server logic looks like this (see example2/api.php):

$mapper = new Simple\Mapper('Demo\\Api\\Endpoint\\');
$evaluator = new Validator\Evaluator(new Simple\Evaluator($mapper), $mapper);
$server = new JsonRpc\Server($evaluator);

header('Content-Type: application/json');
$message = file_get_contents('php://input');

echo $server->reply($message);

4.2. Usage

If the input is valid, the request and response look no different than above. However, if the parameter input is invalid, the API now responds with a JSON-RPC error:

# Invalid parameter values are rejected by validation 
$ curl \
    -d '{"jsonrpc":"2.0","id":1,"method":"devices/listAll","params":{"sortBy":"INVALID"}}' \
    http://localhost:8888/example2/api.php
{"jsonrpc":"2.0","id":1,"error":{"code":-32602,"message":"Invalid params"}}

4.3. More examples

Find a working demo of the php-json-rpc-validator library in example 2 of the JSON-RPC demo project.

5. Authentication (php-json-rpc-auth)

What would an API be without authentication? Probably reckless, in most cases.

The php-json-rpc-auth library offers a simple framework to implement any kind of authentication and authorization for your API. Instead of implementing all the different auth mechanisms (HTTP Basic Auth, Digest, OAuth, SAML, Cookies, …), it merely provides a simplistic Auth\Authenticator class to consult a user-provided set of Auth\Handlers. The actual implementation of these handler class(es) must be provided by the developer.

5.1. Sample code

Assuming that we want to implement HTTP Basic Auth for our API, we only need to implement an Auth\Handler (see src/Api/Auth/BasicAuthHandler.php):

class BasicAuthHandler implements Auth\Handler
{
    public function canHandle($method, $arguments)
    {
        return isset($_SERVER['PHP_AUTH_USER']);
    }

    public function authenticate($method, $arguments)
    {
        // Don't use '===', as that's vulnerable to timing attacks!
        return $_SERVER['PHP_AUTH_USER'] === 'user' && $_SERVER['PHP_AUTH_PW'] === 'pass';
    }
}

The canHandle() method tells the authenticator whether the given request can be authenticated with this handler. The authenticate() method is only called if canHandle() returned true and then authenticates the actual request.

As for the API server code, we can simply wrap an Auth\Evaluator around our existing evaluator (see example3/api.php).

$authenticator = new Auth\Authenticator(array(
    new BasicAuthHandler()
));

$mapper = new Simple\Mapper('Demo\\Api\\Endpoint\\');
$evaluator = new Auth\Evaluator(new Validator\Evaluator(new Simple\Evaluator($mapper), $mapper), $authenticator);
$server = new JsonRpc\Server($evaluator);

header('Content-Type: application/json');
$message = file_get_contents('php://input');

echo $server->reply($message);

This may look a bit complicated, but it merely nests different Auth\Evaluator implementations. In the example above, an incoming request is first passed to the Auth\Evaluator, then (if authorized) to the Validator\Evaluator, and finally (if validated) to the Simple\Evaluator. This is a very classic implementation of the decorator pattern.

5.2. Usage

Trying to access API without authentication will lead to a “Missing auth” error message. In this example, please note that curl’s -u parameter (basic auth user/pass) is not being passed:

$ curl \
    -d '{"jsonrpc":"2.0","id":1,"method":"devices/listAll"}' \
    http://localhost:8888/example3/api.php
{"jsonrpc":"2.0","id":1,"error":{"code":-32651,"message":"Missing auth."}}

In this example, the -u parameter is passed, i.e. the Authorization HTTP header is sent, but with invalid credentials. We’re basically trying to access API with invalid Basic Auth credentials:

$ curl \
    -u invaliduser:pass \
    -d '{"jsonrpc":"2.0","id":1,"method":"devices/listAll"}' \
    http://localhost:8888/example3/api.php
{"jsonrpc":"2.0","id":1,"error":{"code":-32652,"message":"Invalid auth."}}

Only if we provide the correct credentials, we’re allowed to access the API via HTTP (authenticated via HTTP Basic Auth):

$ curl \
    -u user:pass \
    -d '{"jsonrpc":"2.0","id":1,"method":"devices/listAll"}' \
    http://localhost:8888/example3/api.php
{"jsonrpc":"2.0","id":1,"result":[{"name":"Phil Phone","id":2,"type":"phone"},{"name":"Phil Washing Machine","id":3,"type":"other"},{"name":"Philipp PC","id":1,"type":"pc"}]}

5.3. More examples

Find a working demo of the php-json-rpc-auth library in example 3 of the JSON-RPC demo project.

6. Logging with “php-json-rpc-log”

Logging requests and responses of an API can be a very important tool for troubleshooting or statistics. The php-json-rpc-log library offers a very simplistic way to log JSON-RPC equests and responses.

6.1. Sample code

We’re mostly using Monolog, but the php-json-rpc-log library supports any kind of PSR-3 logger. Simply use the Logged\Server class instead of the original JsonRpc\Server class, and pass a Psr\Log\LoggerInterface to it:

$evaluator = // see above
$logger = new Logger('demo.api', array(new SyslogHandler('api')));
$server = new Logged\Server($evaluator, $logger);

header('Content-Type: application/json');
$message = file_get_contents('php://input');

echo $server->reply($message);

6.2. Usage

In the case above, we passed a Monolog\Logger with a SyslogHandler to it, meaning that requests that we send will be logged to /var/log/syslog. The usage is the same as in the other examples, but we’ll see output like this when we tail syslog:

Jan 15 01:24:21 platop api[13157]: demo.api.INFO: Message received: {"jsonrpc":"2.0","id":1,"method":"devices/add"} [] []
Jan 15 01:24:21 platop api[13157]: demo.api.INFO: Sending reply: {"jsonrpc":"2.0","id":1,"error":{"code":-32602,"message":"Invalid params"}} [] []
Jan 15 01:24:46 platop api[13157]: demo.api.INFO: Message received: {"jsonrpc":"2.0","id":1,"method":"devices/listAll"} [] []
Jan 15 01:24:46 platop api[13157]: demo.api.INFO: Sending reply: {"jsonrpc":"2.0","id":1,"result":[{"id":0,"name":"Philipp PC","type":"pc"}]} [] []

6.3. More examples

Find a working demo of the php-json-rpc-log library in example 4 of the JSON-RPC demo project.

7. Advanced usage

There are a million ways to combine these components, and I can surely not list them all here. However, here’s a short list of very useful applications.

7.1. Using the API from the shell or via SSH (with ‘root’-only access)

Unlike REST-based APIs, we can easily expose a JSON-RPC based API via other protocols, such as SSH or raw TCP sockets. We can even just use it locally by piping the JSON-RPC request to a PHP script.

In example 5 of the JSON-RPC demo project, we extend the api.php file to support requests from STDIN (if the file is called from the CLI). The API can still be used via HTTP, but the script now checks via php_sapi_name() if we’re in CLI mode.

In addition to that, the demo implements another auth handler (see src/Api/Auth/CliAuthHandler.php) to check if the user is root, and disallows access if she isn’t:

$authenticator = new Auth\Authenticator(array(
    new CliAuthHandler(),
    new BasicAuthHandler()
));

// ...

$isCLI = php_sapi_name() === 'cli';
if ($isCLI) {
    $message = file_get_contents('php://stdin');
} else {
    header('Content-Type: application/json');
    $message = file_get_contents('php://input');
}
echo $server->reply($message);

With this insanely simple addition, we can now use the API locally by piping the request to the api.php script:

# Access locally; or NOT, since we're not 'root' (access denied by CliAuthHandler)
$ echo '{"jsonrpc":"2.0","id":1,"method":"devices/listAll"}' | php web/example5/api.php
{"jsonrpc":"2.0","id":1,"error":{"code":-32652,"message":"Invalid auth."}}

# Let's try again as 'root'
$ echo '{"jsonrpc":"2.0","id":1,"method":"devices/listAll"}' | sudo php web/example5/api.php
{"jsonrpc":"2.0","id":1,"result":[{"id":0,"name":"Philipp PC","type":"pc"}]}

Now, since we can simply pipe, that means using the API across machines via SSH is trivial:

# And via SSH ... !
$ echo '{"jsonrpc":"2.0","id":1,"method":"devices/listAll"}' \
   | ssh root@myserver php /opt/demo/web/example1/api.php
{"jsonrpc":"2.0","id":1,"result":[{"id":0,"name":"Philipp PC","type":"pc"}]}

7.2. Using the API with JavaScript and cookie-based auth

I’ve been using curl in all the examples, but of course the primary use case of an API is often the use inside a web application. Since JSON-RPC requests are merely POST requests with JSON payload, you can use jQuery’s $.ajax() method – for instance like this:

$.ajax({
    url: '/example6/api.php',
    type: 'POST',
    contentType: 'application/json',
    dataType: 'json',
    data: JSON.stringify({
        jsonrpc: '2.0',
        method: 'devices/listAll',
        params: {}
    }),
    // ...
});

In example 6 of the JSON-RPC demo project, we implement a small web page to display (and auto-refresh) the list of devices using the API, but only if we’re logged in via a cookie:

Without being logged in (no cookie):

When logged in (with cookie):

To achieve this, we implement yet another auth handler (CookieAuthHandler, see src/Api/Auth/CookieAuthHandler.php), and toggle a cookie when the “Login” button is pressed.

All that’s left is to call the API in regular intervals and update the UI. We implemented a small helper class called Datto.API.Client (see web/example6/js/client.js) to interact with the API from our web app:

var api = new Datto.API.Client('api.php');
api.call('devices/listAll', {}, function (data) {
    // Success function
    $devices.empty();
    $(data).each(function (idx, device) {
        $devices.append($('<li>').addClass(device.type).text(device.name));
    });
}, function (data) {
    // Failure function
    $devices
        .empty()
        .append($('<li>').addClass('error').text('ERROR: ' + data.error.message));
});

That’s it. I hope this was helpful. Please let me know what you think in the comments!

Updates

Feb 2016: The new version of simp_le requires a -f account_key.json argument. I’ve updated the post accordingly.

Feb 2017: Unfortunately, the original simp_le project has been abandoned by the original author. This fork still works fine though. I have changed all the links, and it should just work.

1. Install the simp_le client

On your web server, clone the simp_le client and install it in a sensible directory (e.g. /opt/simp_le). Then run the installation steps as provided in the README.md:

Install client and symlink it

# Clone
$ cd /opt
$ git clone https://github.com/zenhack/simp_le/
$ cd simp_le

# Install
$ ./bootstrap.sh
$ ./venv.sh
$ ln -s $(pwd)/venv/bin/simp_le /usr/local/sbin/simp_le

2. Generate a keypair and retrieve signed certificate

Once the installation is done, everything else is super easy. All you need is a location for your keys and certificates, as well as the publicly available document root for your website. I store my certificates and keys in /srv/cert, with a subfolder for each domain, e.g. /srv/cert/example.com, and my document root is at /srv/www/example.com/html.

Create certificate signed by Let’s Encrypt

$ cd /srv/cert/example.com
$ simp_le \
  -d example.com:/srv/www/example.com/html \
  -f key.pem -f cert.pem -f fullchain.pem -f account_key.json

2015-12-04 01:31:52,131:INFO:simp_le:157: Creating new account key
2015-12-04 01:31:56,529:INFO:requests.packages.urllib3.connectionpool:756: Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
...
2015-12-04 01:31:58,568:INFO:requests.packages.urllib3.connectionpool:207: Starting new HTTP connection (1): example.com
2015-12-04 01:31:58,588:INFO:simp_le:803: example.com was successfully verified by the client
...
2015-12-04 01:32:04,570:INFO:simp_le:409: Saving key.pem
2015-12-04 01:32:04,570:INFO:simp_le:370: Saving fullchain.pem

This command automatically generates a keypair, creates a certificate request and gets that signed by Let’s Encrypt. It also verifies your domain ownership by creating a file in the document root at http://example.com/.well-known/.... So let’s look at the key and certificate:

Certificate and private key file

$ ls
account_key.json  cert.pem  fullchain.pem  key.pem

3. Configure your web server

Now configure the SSL/TLS certificate in your web server. For me, that’s Apache. The configuration for the virtual host looks like this:

Configuring the Apache virtual host

$ cat /etc/apache2/sites-enabled/example.com

<VirtualHost *:443>
        SSLEngine on
        SSLCertificateFile /srv/cert/example.com/cert.pem
        SSLCertificateKeyFile /srv/cert/example.com/key.pem
        SSLCertificateChainFile /srv/cert/example.com/fullchain.pem

        DocumentRoot "/silv/www/example.com/html"
        ServerName example.com

        // ...
</VirtualHost>

And finally, restart Apache with service apache2 restart and you can access your website via HTTPS! Easy, right?

4. Automating certificate renewal via cronjob

The wonderful thing about the simp_le client is that the command for the initial creation and the renewal is the same, so you can run the exact same command as above in a script from a cronjob. I created a script called /srv/bin/cert-renew to do just that:

Script to check/renew the certificates every night

$ cat /srv/bin/cert-renew 
#!/bin/bash

cd /srv/cert/example.com
simp_le -d example.com:/srv/www/example.com/html -f key.pem -f cert.pem -f fullchain.pem -f account_key.json \
	&& service apache2 reload

cd /srv/cert/example.xyz
simp_le -d example.xyz:/srv/www/example.xyz/html -f key.pem -f cert.pem -f fullchain.pem -f account_key.json \
	&& service apache2 reload

This will renew the certificates for two domains (if necessary) and reload Apache if the certificate has been renewed. Note that Apache will not be reloaded if the certificate has not been altered.

The matching cronjob looks like this:

Cronjob to run renewal script every night

PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin
43 1 * * * /srv/bin/cert-renew || true

That’s it! Now every night at 1:43am, your certificates are checked and renewed if necessary.

A. About this post

I’m trying a new section for my blog. I call it Code Snippets. It’ll be very short, code-focused posts of things I recently discovered or find fascinating or helpful. I hope this helps.

1. Demo package ’netutils'

For the sake of this tutorial, we’ll create a package called netutils with a command called ipaddr. The purpose of the command will be to get the external IP address, just like this:

 # Target command 'ipaddr' as part of the
# to-be-built 'netutils' package
$ ipaddr
82.143.12.32

2. Create a Debian package

Let’s start by creating the empty project directory netutils/. This folder will contain both the source code and the Debian build instructions.

2.1. Create the debian/ directory

For now, we’ll use the dh_make command to create the debian/ directory:

# Create empty project folder
$ mkdir -p Debian/netutils
$ cd Debian/netutils

# Create debian/ folder with example files (.ex)
$ dh_make \
  --native \
  --single \
  --packagename netutils_1.0.0 \
  --email phil@example.com

This created the debian/ folder. Explore them! Especially the example files (*.ex) as well as most importantly the files:

• debian/control
• debian/changelog
• debian/rules

There are maaaany files, and Debian documented all of them in the Debian Maintainer’s Guide. For now, we don’t care about the rest of them, but here’s a list of all the created files:

$ find debian/ | sort
debian/
debian/changelog
debian/compat
debian/control
debian/copyright
debian/netutils.cron.d.ex
debian/netutils.default.ex
debian/netutils.doc-base.EX
debian/docs
debian/init.d.ex
debian/manpage.1.ex
debian/manpage.sgml.ex
debian/manpage.xml.ex
debian/menu.ex
debian/postinst.ex
debian/postrm.ex
debian/preinst.ex
debian/prerm.ex
debian/README
debian/README.Debian
debian/README.source
debian/rules
debian/source
debian/source/format
debian/watch.ex

2.2. Build the first (empty) package

As you can see there are a lot of files to tweak the package, but for now we’ll ignore all that and build an empty package. The dpkg-buildpackage command can be used to do that:

# Build empty package
$ dpkg-buildpackage

That’s it! That command built four files:

# Explore the files that got built
$ ls -1 ../netutils_*
../netutils_1.0.0_amd64.changes
../netutils_1.0.0_amd64.deb
../netutils_1.0.0.dsc
../netutils_1.0.0.tar.gz

• .tar.gz: Source package, contains the contents of the netutils/ folder
• .deb: Debian package, contains the installable package
• .dsc/.changes: Signature files, cryptographic signatures of all files

Obviously, the most interesting file (for now) is the .deb file.
Let’s examine the contents with the dpkg -c (aka dpkg --contents) command:

# Listing the contents with 'dpkg --contents'
$ dpkg -c ../netutils_1.0.0_amd64.deb 
drwxr-xr-x root/root         0 2015-10-18 14:52 ./
drwxr-xr-x root/root         0 2015-10-18 14:52 ./usr/
drwxr-xr-x root/root         0 2015-10-18 14:52 ./usr/share/
drwxr-xr-x root/root         0 2015-10-18 14:52 ./usr/share/doc/
drwxr-xr-x root/root         0 2015-10-18 14:52 ./usr/share/doc/netutils/
-rw-r--r-- root/root       141 2015-10-18 14:47 ./usr/share/doc/netutils/changelog.gz
-rw-r--r-- root/root       183 2015-10-18 14:47 ./usr/share/doc/netutils/README.Debian
-rw-r--r-- root/root      1401 2015-10-18 14:47 ./usr/share/doc/netutils/copyright

Nothing really in the package except the default changelog, copyright and README file. Instead of just listing the contents, we can also extract a Debian archive to a local location with dpkg -x (aka dpkg --extract):

# Extract the archive without installing it
$ dpkg -x ../netutils_1.0.0_amd64.deb ../netutils_extracted
$ find ../netutils_extracted/
../netutils_extracted/
../netutils_extracted/usr
../netutils_extracted/usr/share
../netutils_extracted/usr/share/doc
../netutils_extracted/usr/share/doc/netutils
../netutils_extracted/usr/share/doc/netutils/copyright
../netutils_extracted/usr/share/doc/netutils/changelog.gz
../netutils_extracted/usr/share/doc/netutils/README.Debian

2.3. Install the (empty) package

Enough playing around with the .deb file. Let’s install it with the dpkg -i (aka dpkg --install) command:

# Install the (empty) package
$ sudo dpkg -i ../netutils_1.0.0_amd64.deb 
Preparing to unpack ../netutils_1.0.0_amd64.deb ...
Unpacking netutils (1.0.0) ...
Setting up netutils (1.0.0) ...

Done. That installed the package. Check out that it was actually installed by listing the installed packages with dpkg -l (aka dpkg --list). The list itself will contain all installed (or half-installed/configured) packages on the system, so we’ll use grep to limit the output:

# Proof that it's actually installed
$ dpkg -l | grep netutils
  ii  netutils    1.0.0        amd64        <insert up to 60 chars description>
# ^^  ^           ^            ^            ^
# ||  |           |            |            |
# ||  |           |            |             - Description
# ||  |           |             - Architecture
# ||  |            - Version
# ||   - Package name
# | - Actual/current package state (n = not installed, i = installed, ...)
#  - Desired package state (i = install, r = remove, p = purge, ...)

The columns in the list are desired package state, actual package state, package name, package version, package architecture and a short package description. If all is well, the first column will contain ii, which means that the package is properly installed. The dpkg-query man page (man dpkg-query) contains all the possible desired/actual state values.

Now that the package is installed, you can list its contents with the dpkg -L (aka dpkg --listfiles) command. Unlike dpkg -c, it only works for installed packages:

# List of installed Debian package contents
$ dpkg -L netutils 
/.
/usr
/usr/share
/usr/share/doc
/usr/share/doc/netutils
/usr/share/doc/netutils/copyright
/usr/share/doc/netutils/changelog.gz
/usr/share/doc/netutils/README.Debian

2.4. Adding files and updating the changelog

Okay, enough with the empty package. Let’s now add actual files to our package. To do that, let’s create a folder files/ and use it to mirror the Linux filesystem structure. In that folder, let’s create our script files/usr/bin/ipaddr:

$ mkdir -p files/usr/bin
$ touch files/usr/bin/ipaddr
$ chmod +x files/usr/bin/ipaddr
$ vi files/usr/bin/ipaddr
  # Script contents see below

Obviously, you can add whatever you want to your script, but for the sake of this tutorial, we’ll go with a short script to grab the public IP address from a service called ipify. It provides an API to return the IP address in various formats. We’ll grab it with curl in JSON and then use jq to parse out the ‘ip’ field:

Create script ‘files/usr/bin/ipaddr’

#!/bin/bash
curl --silent 'https://api.ipify.org?format=json' | jq .ip --raw-output

If we were to rebuild the package now, the dpkg-buildpackage command wouldn’t know which files to include in the package. So we’ll create the debian/install file to list directories to include (e.g. vi debian/install):

Create file ‘debian/install’ to tell dpkg-buildpackage what files to include

files/usr/* usr

This basically means that everything in the files/usr/ folder will be installed at /usr/ on the target file system when the package is installed.

Once this is done, we could just rebuild and reinstall the package, but let’s go one step further and update the version of the package to 1.1.0. Versions are handled by the debian/changelog file. You can update it manually, or use the dch script (short for “Debian changelog”) to do so:

Update debian/changelog

# We changed the package, let's update the version and changelog for it to 1.1.0
$ dch -im
  # Opens the editor ...

$ cat debian/changelog 
netutils (1.1.0) unstable; urgency=medium

  * Added 'ipaddr' script

 -- Philipp Heckel <phil@example.com>  Sun, 18 Oct 2015 15:55:42 +0100

netutils (1.0.0) unstable; urgency=low

  * Initial Release.

 -- Philipp Heckel <phil@example.com>  Sun, 18 Oct 2015 15:06:29 +0100

While the changelog looks just like a dumb text file, Debian uses it to manage the version number of the package, as well as to define the distribution(s) that the package is built for (here: “unstable”). Furthermore, it defines who has to sign the package (name of changelog editor).

After that, let’s rebuild and reinstall the package:

Rebuild and reinstall the package

# Rebuild & re-install package
$ dpkg-buildpackage
$ sudo dpkg -i ../netutils_1.1.0_amd64.deb
$ dpkg -l | grep netutils
ii  netutils       1.1.0          amd64        <insert up to 60 chars description>

Looks like it installed correctly. Let’s check if the ipaddr script is where it’s supposed to be. And then let’s try to run it:

# Proof that it's where it's supposed to be
$ which ipaddr
/usr/bin/ipaddr

# Now let's run it
$ ipaddr
/usr/bin/ipaddr: line 3: jq: command not found

Oops! We forgot that not every system might have jq installed. Let’s add it as a dependency!

2.5. Updating the description and adding dependencies

Each Debian package can depend on other packages. In the case of our ipaddr script, we use the curl and jq commands, so the ’netutils’ package depends on these commands.

Since typically the command name is different from the package name, it might be necessary to find the actual package name. This can be done with the dpkg -S (aka dpkg --search) command:

# Find the full path for the 'curl' command
$ which curl
/usr/bin/curl

# Seems that the 'curl' command is contained in a package called 'curl'
$ dpkg -S /usr/bin/curl
curl: /usr/bin/curl

# The same can be done for 'jq' if it's installed.

To add the dependencies to the ’netutils’ package, edit the Depends: section in the debian/control file (e.g. via vi debian/control):

Edit sections ‘Depends’ and ‘Description’ in ‘debian/control’

Source: netutils
Section: utils
Priority: optional
Maintainer: Philipp Heckel <phil@example.com>
Build-Depends: debhelper (>= 8.0.0)
Standards-Version: 3.9.4

Package: netutils
Architecture: any
Depends: ${shlibs:Depends}, ${misc:Depends}, curl, jq
Description: Network management tools
  Includes various tools for network management.

That’s it. All that’s left (again) is to update the version to 1.2.0, rebuild the package and reinstall it:

Rebuild and reinstall the package

# Rebuild & re-install package
$ dch -im
$ dpkg-buildpackage
$ sudo dpkg -i ../netutils_1.2.0_amd64.deb 
Preparing to unpack ../netutils_1.2.0_amd64.deb ...
Unpacking netutils (1.2.0) over (1.1.0) ...
dpkg: dependency problems prevent configuration of netutils:
 netutils depends on jq; however:
  Package jq is not installed.
dpkg: error processing package netutils (--install):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 netutils

Wow. What happened here?

Well, unlike apt-get install, the dpkg -i command does not automatically resolve and install missing dependencies. It just complains about them. That is perfectly normal and expected. In fact, it gives us the perfect opportunity to check the package state (like we did above):

Invalid package state, as indicated by ‘iU’

# The package is in an invalid state:
$ dpkg -l | grep netutils
  iU  netutils  1.2.0      amd64        Network management tools
# ^^                                     ^
# ||                                     |
# ||                                      - Yeyy, the new description
# | - Actual package state is 'U' / 'Unpacked'
#  - Desired package state is 'i' / 'install'

As you can see by the output, the desired state for the package is ‘i’ (= installed), but the actual state is ‘U’ (= Unpacked). That’s not good. Luckily though, dependencies can be automatically resolved by apt-get install -f (aka apt-get install --fix-broken):

Fixing broken/invalid package state and dependencies

$ sudo apt-get install -f
The following extra packages will be installed:
  jq
The following NEW packages will be installed
  jq
0 to upgrade, 1 to newly install, 0 to remove and 11 not to upgrade.
1 not fully installed or removed.
Selecting previously unselected package jq.
Preparing to unpack .../jq_1.3-1.1ubuntu1_amd64.deb ...
Unpacking jq (1.3-1.1ubuntu1) ...
Setting up jq (1.3-1.1ubuntu1) ...
Setting up netutils (1.2.0) ...

Finally, it’s installed correctly. Now let’s test it:

Testing the ‘ipaddr’ script

# The package is installed correctly ('ii')
$ dpkg -l | grep netutils
ii  netutils  1.2.0      amd64        Network management tools

# And it seems to work!
$ ipaddr
82.143.12.32

3. Create and use a Debian repository

3.1. dpkg -i vs. apt-get install

So why did we use dpkg -i and not apt-get install? Because apt-get install looks in all the configured Debian repositories; it does not look for files. And as of right now, there is no Debian repository serving the netutils package:

No Debian repository serving the ’netutils’ package

# There is no Debian repo serving the 'netutils' package!
$ sudo apt-get install netutils
Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Unable to locate package netutils

3.2. Locally configured Debian/APT repositories (/etc/apt/sources.list.d/*.list)

So where does ‘apt-get install’ look? How does it know where to retrieve/download the packages and its dependencies from?

APT repositories are configured in the files /etc/apt/sources.list and /etc/apt/sources.list.d/*.list. On a typical Debian/Ubuntu system, there are quite a few of them:

APT repositories are configured in /etc/apt/sources.list and /etc/apt/sources.list.d/*.list

$ find /etc/apt/sources.list /etc/apt/sources.list.d/
/etc/apt/sources.list
/etc/apt/sources.list.d/
/etc/apt/sources.list.d/syncany.list
/etc/apt/sources.list.d/archive.philippheckel.com.list
/etc/apt/sources.list.d/official-package-repositories.list
...

Each of these files contains a list of Debian repositories:

$ cat /etc/apt/sources.list.d/official-package-repositories.list 
deb http://archive.ubuntu.com/ubuntu trusty main restricted universe multiverse
#   ^                                ^      ^
#   |                                |      |
#    - Main URL/source               |       - Component list (e.g. main, universe, ...)
#                                     - Distribution (e.g. precise, trusty, release, ...)

3.3. The APT cache and Packages/Packages.gz files

Whenever apt-get update is called, all of these repos are checked for new versions. The APT cache (see apt-cache command) is refreshed:

Refreshing the APT cache with ‘apt-get update’

$ apt-get update | grep Packages
Hit http://archive.ubuntu.com trusty/main amd64 Packages
...

The ‘Hit’ actually means that it downloaded/checked the Packages.gz file (in the example above from the URL http://archive.ubuntu.com/ubuntu/dists/trusty/main/binary-amd64/Packages.gz). The file looks like this:

$ curl -s http://archive.ubuntu.com/ubuntu/dists/trusty/main/binary-amd64/Packages.gz | zcat | head -n 14
Package: account-plugin-aim
Priority: optional
Section: gnome
Installed-Size: 941
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Original-Maintainer: Debian Telepathy maintainers <pkg-telepathy-maintainers@lists.alioth.debian.org>
Architecture: amd64
Source: empathy
Version: 3.8.6-0ubuntu9
Replaces: account-plugin-empathy
Depends: empathy (= 3.8.6-0ubuntu9), telepathy-haze, mcp-account-manager-uoa, unity-asset-pool (>> 0.8.24daily13.03.20.1)
Breaks: account-plugin-empathy
Filename: pool/main/e/empathy/account-plugin-aim_3.8.6-0ubuntu9_amd64.deb
Size: 8838
...

Looks familiar? Yes! A Packages.gz file looks very, very similar to the debian/control file in our package. And that’s no coincidence.

3.4. Creating a local Debian repository

So in a nutshell, all we need to create a Debian repository is a Packages.gz file and a way to expose this file – either via HTTP or locally. Easy, easy, easy!

For the sake of this tutorial, let’s create a local repository at /tmp/repo and copy all the netutils_*.deb files to it. Once that is done, we’ll create a Packages.gz file using the dpkg-scanpackages command:

$ mkdir /tmp/repo
$ cp ../netutils_*.deb /tmp/repo/
$ cd /tmp/repo
$ dpkg-scanpackages -m . | gzip --fast > Packages.gz

That’s it. Let’s check what the Packages.gz file looks like:

$ zcat /tmp/repo/Packages.gz
$ zcat Packages.gz 
Package: netutils
Version: 1.0.0
Architecture: amd64
Maintainer: Philipp Heckel <phil@example.com>
Installed-Size: 2
Filename: ./netutils_1.0.0_amd64.deb
Size: 1994
MD5sum: 1de1a751a5bc0c9892e3e7c7740c57a9
SHA1: 2ea91ec16519638fcfd371d1530fe13aaa123f49
SHA256: c40ea5184715e6d6cdf946fd66e84c19c4b0d7f516fe6842d57da385af7fc180
Section: unknown
Priority: optional
Homepage: <insert the upstream URL, if relevant>
Description: <insert up to 60 chars description>
 <insert long description, indented with spaces>

Package: netutils
Version: 1.1.0
Architecture: amd64
Maintainer: Philipp Heckel <phil@example.com>
Installed-Size: 2
Depends: curl, jq
Filename: ./netutils_1.1.0_amd64.deb
...

3.5. Adding the local Debian repository and installing via apt-get install

All that’s left is to add that repository to the local APT config by adding a *.list file, e.g. at /etc/apt/sources.list.d/local.list:

Local repository definition in file ‘/etc/apt/sources.list.d/local.list’

deb file:/tmp/repo ./

Now we can install the netutils package via apt-get install:

Installing the ’netutils’ package via apt-get

# Update the APT cache and install the package
$ apt-get update
$ sudo apt-get install netutils

# Yup, it works
$ ipaddr
82.143.12.32

3.6. Listing and installing older versions

Maybe you noticed that the Packages.gz file contained multiple versions of the package. That means that we can actually list and install specific/older versions of the package if we wanted to.

Let’s explore what versions we could install. This can be done with apt-cache policy or apt-cache madison:

# You can use 'apt-cache policy' to display all available versions
$ apt-cache policy netutils
netutils:
  Installed: 1.2.0
  Candidate: 1.2.0
  Version table:
 *** 1.2.0 0
        500 file:/tmp/repo/ ./ Packages
        100 /var/lib/dpkg/status
     1.1.0 0
        500 file:/tmp/repo/ ./ Packages
     1.0.0 0
        500 file:/tmp/repo/ ./ Packages

# The 'apt-cache madison' command does the same
$ apt-cache madison netutils
 netutils |      1.2.0 | file:/tmp/repo/ ./ Packages
 netutils |      1.1.0 | file:/tmp/repo/ ./ Packages
 netutils |      1.0.0 | file:/tmp/repo/ ./ Packages

By default apt-get install always picks the newest one, but by doing apt-get install netutils=1.0.0, we could tell it to install version 1.0.0:

Downgrading a package

$ sudo apt-get install netutils=1.0.0
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be DOWNGRADED:
  netutils
0 to upgrade, 0 to newly install, 1 to downgrade, 0 to remove and 13 not to upgrade.
Need to get 0 B/1,994 B of archives.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
dpkg: warning: downgrading netutils from 1.2.0 to 1.0.0
(Reading database ... 208809 files and directories currently installed.)
Preparing to unpack ..././netutils_1.0.0_amd64.deb ...
Unpacking netutils (1.0.0) over (1.2.0) ...
Setting up netutils (1.0.0) ...

3.7. Normal/typical repository layout

Our local repository is very basic. It does not have the same sophisticated structure of a normal repository such as the main upstream Ubuntu repository. Normal repositories looks more like this:

Layout of a standard Debian repository (excerpt)

$ find dists/trusty/ -maxdepth 3; find pool/ -maxdepth 1
dists/trusty
dists/trusty/Release.gpg
dists/trusty/Release
dists/trusty/InRelease
dists/trusty/main
dists/trusty/main/binary-amd64
dists/trusty/main/binary-amd64/Packages
dists/trusty/main/binary-amd64/Release
dists/trusty/main/binary-amd64/Packages.bz2
dists/trusty/main/binary-amd64/Packages.diff
dists/trusty/main/binary-amd64/Packages.gz
dists/trusty/main/Contents-amd64.bz2
dists/trusty/main/Contents-amd64
dists/trusty/main/Contents-amd64.gz
dists/trusty/Contents-amd64.bz2
dists/trusty/Contents-amd64
dists/trusty/Contents-amd64.gz
pool/
pool/main

If you really wanted to, you could create that structure yourself. For the fun of it, let’s do that with our repository at /tmp/repo. Remember, this is what it looks like:

Previously created simple structure

$ find /tmp/repo/ | sort
/tmp/repo/
/tmp/repo/netutils_1.0.0_amd64.deb
/tmp/repo/netutils_1.1.0_amd64.deb
/tmp/repo/netutils_1.2.0_amd64.deb
/tmp/repo/Packages.gz

Now to “transform” it:

Creating the typical repository structure (manually)

# Delete the old Packages.gz file
$ rm Packages.gz

# Create new structure
$ mkdir -p pool/main/n dists/trusty/main/binary-{amd64,i386}
$ mv *.deb pool/main/n

# Create new Packages.gz files
$ dpkg-scanpackages -m pool | gzip > dists/trusty/main/binary-amd64/Packages.gz
dpkg-scanpackages: info: Wrote 3 entries to output Packages file.

$ dpkg-scanpackages -m pool | gzip > dists/trusty/main/binary-i386/Packages.gz
dpkg-scanpackages: info: Wrote 3 entries to output Packages file.

After that, the directory structure of our almost-proper repository looks like this:

Almost-proper repository for netutils

# find /tmp/repo/ | sort
/tmp/repo/
/tmp/repo/dists
/tmp/repo/dists/trusty
/tmp/repo/dists/trusty/main
/tmp/repo/dists/trusty/main/binary-amd64
/tmp/repo/dists/trusty/main/binary-amd64/Packages.gz
/tmp/repo/dists/trusty/main/binary-i386
/tmp/repo/dists/trusty/main/binary-i386/Packages.gz
/tmp/repo/pool
/tmp/repo/pool/main
/tmp/repo/pool/main/n
/tmp/repo/pool/main/n/netutils_1.0.0_amd64.deb
/tmp/repo/pool/main/n/netutils_1.1.0_amd64.deb
/tmp/repo/pool/main/n/netutils_1.2.0_amd64.deb

Then, we obviously need to update our sources .list file at /etc/apt/sources.list.d/local.list:

New sources file with distribution and component at ‘/etc/apt/sources.list.d/local.list’

deb file:/tmp/repo trusty main

1. Scenario: Multiple optional components

Imagine you have multiple components of a project, and each of them has their own composer.json file. Maybe something like this:

project-core/ composer.json vendor/ project-ui/ composer.json vendor/ project-addon-1/ composer.json vendor/ project-addon-2/ composer.json vendor/ ...

Each of the components has its own dependencies. Normally you would just put them on your own Packagist server and then use Composer’s require statement to let them depend on each other. But what if some dependencies are optional? Like the add-ons above. Or you want to be able to have different user interfaces and you don’t want to distribute the entire project as a whole? What if your components are Debian packages?

project-core.deb project-ui.deb project-addon-1.deb project-addon-2.deb ...

You won’t have any luck with Composer. Because Composer only loads and initializes dependencies at build time. It has no ability to discover components or dependencies at runtime and then load them, unfortunately. So even if we do a composer install on all the subprojects/components, there is no easy way to plugin them together dynamically.

To achieve this, we have to be a little creative …

2. “Solution 1”: Merge at build time

Wait? Didn’t I say runtime? Yes, I said runtime. However, if you know at build time what components will be shipped, or you can pull them down from Git (as submodule or subtree), you can use a very neat little plugin called the composer-merge-plugin. This plugin can extend a composer.json file by including others. However, be aware that it creates only one vendor folder. It does not use multiple composer.json files and vendor directories at runtime!

3. Solution 2: Merge at runtime

If the option above is not really an option for you, you can write your own little autoloader instead, or reuse the composer autoloaders of the projects that you want to include. Be aware that while this works if you have no conflicting libraries, things might blow up if you have:

Loading classes from namespace of other project

<?php

/** @var Composer\Autoload\ClassLoader $loader */

$loader = require_once __DIR__ . '/../vendor/autoload.php';
$map = require_once __DIR__ . '/../../core/vendor/composer/autoload_psr4.php';

foreach ($map as $namespace => $path) {
    $loader->addPsr4($namespace, $path);
}

The snippet above loads the autoloader of the current project (../vendor/autoload.php) and adds the PSR-4 namespaced classes of the ‘core’ project to it (../../core/vendor/composer/autoload_psr4.php). The autoload_psr4.php file returns an associative array of namespace-to-folder mappings, which then will be used by the composer autoloader to discover the class file.

I realize that this is not the most elegant solution, but it worked for me. Let me know if any of you have a better solution.

A. About this post

I’m trying a new section for my blog. I call it Code Snippets. It’ll be very short, code-focused posts of things I recently discovered or find fascinating or helpful. I hope this helps.