Taking on a new platform design recently I thought it was interesting to see how things evolved in the past years and how we design and think about platform architecture.

So what do we do ?

As system developers, system administrators and system engineers, what do we do ?

• We develop software
• We design architectures
• We configure systems

But it isn’t the purpose of our jobs, for most of us, our purpose is to generate business value. From a non technical perspective we generate business value by creating a system which renders one or many functions and provides insight into its operation.

And we do this by developing, logging, configuration and maintaining software across many machines.

When I started doing this - back when knowing how to write a sendmail configuration file could get you a paycheck - it all came down to setting up a few machines, a database server a web server a mail server, each logging locally and providing its own way of reporting metrics.

When designing custom software, you would provide reports over a local AF_UNIX socket, and configure your software by writing elegant parsers with yacc (or its GNU equivalent, bison).

When I joined the OpenBSD team, I did a lot of work on configuration files, ask any members of the team, the configuration files are a big concern, and careful attention is put into clean, human readable and writable syntax, additionally, all configuration files are expected to look and feel the same, for consistency.

It seems as though the current state of large applications now demands another way to interact with operating systems, and some tools are now leading the way.

So what has changed ?

While our mission is still the same from a non technical perspective, the technical landscape has evolved and went through several phases.

The first era of repeatable architecture

We first realized that as soon as several machines performed the same task the need for repeatable, coherent environments became essential. Typical environments used a combination of cfengine, NFS and mostly perl scripts to achieve these goals.

Insight and reporting was then providing either by horrible proprietary kludges that I shall not name here, or emergent tools such as netsaint (now nagios), mrtg and the like.

The XML mistake

Around that time, we started hearing more and more about XML, then touted as the solution to almost every problem. The rationale was that XML was - somewhat - easy to parse, and would allow developers to develop configuration interfaces separately from the core functionality.

While this was a noble goal, it was mostly a huge failure. Above all, it was a victory of developers over people using their software, since they didn’t bother writing syntax parsers and let users cope with the complicated syntax.

Another example was the difference between Linux’s iptables and OpenBSD’s pf. While the former was supposed to be the backend for a firewall handling tool that never saw the light of day, the latter provided a clean syntax.

Infrastructure as code

Fast forward a couple of years, most users of cfengine were fed up with its limitations, architectures while following the same logic as before became bigger and bigger. The need for repeatable and sane environments was as important as it ever was.

At that point of time, PXE installations were added to the mix of big infrastructures and many people started looking at puppet as a viable alternative to cfengine.

puppet provided a cleaner environment, and allowed easier formalization of technology, platform and configuration. Philosophically though, puppet stays very close to cfengine by providing a way to configure large amounts of system through a central repository.

At that point, large architectures also needed command and control interfaces. As noted before, most of these were implemented as perl or shell scripts in SSH loops.

On the monitoring and graphing front, not much was happening, nagios and cacti were almost ubiquitous, while some tools such as ganglia and collectd were making a bit of progress.

Where are we now ?

At some point recently, our applications started doing more. While for a long time the canonical dynamic web application was a busy forum, more complex sites started appearing everywhere. We were not building and operating sites anymore but applications. And while with the help of haproxy, varnish and the likes, the frontend was mostly a settled affair, complex backends demanded more work.

At the same time the advent of social enabled applications demanded much more insight into the habits of users in applications and thorough analytics.

New tools emerged to help us along the way:

• In memory key value caches such as memcached and redis
• Fast elastic key value stores such as cassandra
• Distributed computing frameworks such as hadoop
• And of course on demand virtualized instances, aka: The Cloud

Some daemons only provide small functionality

The main difference in the new stack found in backend systems is that the software stacks that run are not useful on their own anymore.

Software such as zookeeper, kafka, rabbitmq serve no other purpose that to provide supporting services in applications and their functionality are almost only available as libraries to be used in distributed application code.

Infrastructure as code is not infrastructure in code !

What we missed along the way it seems is that even though our applications now span multiple machines and daemons provide a subset of functionality, most tools still reason with the machine as the top level abstraction.

puppet for instance is meant to configure nodes, not cluster and makes dependencies very hard to manage. A perfect example is the complications involved in setting up configurations dependent on other machines.

Monitoring and graphing, except for ganglia has long suffered from the same problem.

The new tools we need

We need to kill local configurations, plain and simple. With a simple enough library to interact with distant nodes, starting and stopping service, configuration can happen in a single place and instead of relying on a repository based configuration manager, configuration should happen from inside applications and not be an external process.

If this happens in a library, command & control must also be added to the mix, with centralized and tagged logging, reporting and metrics.

This is going to take some time, because it is a huge shift in the way we write software and design applications. Today, configuration management is a very complex stack of workarounds for non standardized interactions with local package management, service control and software configuration.

Today dynamically configuring bind, haproxy and nginx, installing a package on a Debian or OpenBSD, restarting a service, all these very simple tasks which we automate and operate from a central repository force us to build complex abstractions. When using puppet, chef or pallet, we write complex templates because software was meant to be configured by humans.

The same goes for checking the output of running arbitrary scripts on machines.

Where we’ll be tomorrow

With the ease PaaS solutions bring to developers, and offers such as the ones from VMWare and open initiatives such as OpenStack, it seems as though virtualized environments will very soon be found everywhere, even in private companies which will deploy such environments on their own hardware.

I would not bet on it happening but a terse input and output format for system tools and daemons would go a long way in ensuring easy and fast interaction with configuration management and command and control software.

While it was a mistake to try to push XML as a terse format replacing configuration file to interact with single machines, a terse format is needed to interact with many machines providing the same service, or to run many tasks in parallel - even though, admittedly , tools such as capistrano or mcollective do a good job at running things and providing sensible output.

The future is now !

Some projects are leading the way in this new orientation, 2011 as I’ve seen it called will be the year of the time series boom. For package management and logging, Jordan Sissel released such great tools as logstash and fpm. For easy graphing and deployment etsy released great tools, amongst which statsd.

As for bridging the gap between provisionning, configuration management, command and control and deploys I think two tools, both based on jclouds are going in the right direction:

• Whirr: Which let you start a cluster through code, providing recipes for standard deploys (zookeeper, hadoop)
• pallet: Which lets you describe your infrastructure as code and interact with it in your own code. pallet’s phase approach to cluster configuration provides a smooth dependency framework which allows easy description of dependencies between configuration across different clusters of machines.

Who’s getting left out ?

One area where things seem to move much slower is network device configuration, for people running open source based load-balancers and firewalls, things are looking a bit nicer, but the switch landscape is a mess. As tools mostly geared towards public cloud services will make their way in private corporate environments, hopefully they’ll also get some of the programmable

Protocols and mixins

I recently had to implement something in clojure i’ve done many times in ruby, which involved using protocols. I thought it would be a nice example of comparing class re-opening in ruby and protocol extension in clojure.

The problem

I use cassandra, and in many places, cassandra needs to work with UUID types a lot. When exposing results over JSON, this is often a problem since standard serializers don’t support these types.

What we want to do in ruby and clojure is simple:

1
2
3
4
5

require 'simple_uuid'
require 'json'

# This fails
{:uuid => SimpleUUID::UUID.new }.to_json

This code fails because the json module looks for a to_json method in each object. Failing to do so, it calls Object#to_s.to_json. Now this would work fine if to_s gave a good textual representation of a UUID, but it returns the byte array for that UUID.

1
2
3
4
5
6

(ns foo
 (:use clojure.data.json)
 (:import java.util.UUID))

; This fails
(println (json-str {:uuid (UUID/randomUUID)}))

In clojure we are informed that java.util.UUID doesn’t respond to write-json

Fixing the problem in ruby

How to fix this in ruby is no problem, and widely known, since the simple_uuid gem provides a to_guid method which returns the textual representation, it’s as easy as:

1
2
3
4
5
6
7
8
9
10
11
12

require 'simple_uuid'
require 'json'

module SimpleUUID
  class UUID
    def to_json *args
      "\"#{to_guid}\""
    end
  end
end

puts({:uuid => SimpleUUID::UUID.new}.to_json)

This was simple enough, reopening the module then class is allowed - and to some extent, encouraged - in ruby. We just added a to_json method which is what the JSON module looks for when walking through objects.

Fixing the problem in clojure

clojure has the ability to provide so-called protocols, similar to java interfaces. Protocols are defined with defprotocol and implemented anywhere. Here is the appropriate bit from clojure.data.json

1
2
3
4
5

;;; JSON PRINTER

(defprotocol Write-JSON
  (write-json [object out escape-unicode?]
              "Print object to PrintWriter out as JSON"))

This defines that write-json will be dispatched based on class to an appropriate writer. The clojure page on protocols has all the detailed information, but I’ll focus on the extend part here, which allows to extend a type with new protocol implementations. extend expects a type then pairs of protocol names to maps, the map containing function name to implementation mappings.

Protocol functions always have the object they need to operate on as their first argument, here the function takes two additional arguments

• out which is the output stream the representation should be pushed to
• escape-unicode? which determines whether unicode characters should be escaped.

Following that logic, the implementation can now be written like this:

1
2
3
4
5
6
7
8
9
10
11
12

(ns somewhere
  (:import java.util.UUID)
  (:use clojure.data.json))

(defn write-json-uuid [obj out escape-unicode?]
  (binding [*out* out]
    (pr (.toString obj))))

(extend UUID Write-JSON
  {:write-json write-json-uuid})

(println (json-str {:uuid (UUID/randomUUID)}))

Writing the protocol extension

The actual function write-json-uuid is quite simple, I initially wrote it as:

1
2

(defn write-json-uuid [obj out escape-unicode?]
  (.print out (pr-str (.toString obj))))

But it seems a bit overkill to go to the trouble of writing to a string, then pushing that string out to the writer object.

Dynamic bindings

A small digression is needed here, clojure has dynamic symbols, defined like so: (def ^{:dynamic true} *my-dyn-symbol*) The enclosing stars are a convention but widely used.

Dynamic symbols can be manipulated with binding, which operates like let but the bindings will follow the rest of the execution enclosed, not just the function’s context.

Clojure uses the *out* symbol everywhere to denote the current output stream, many functions operate on it, pr is among them.

By binding *out* to the stream that was given as argument to write-json, pr can simply be called on the function.

Closing words

The most common dispatching idiom in clojure is defmethod/defmulti, but protocols also provide a very fast and useful way to implement polymorphism in clojure. It’s also nice to note that the implementation wasn’t longer in clojure than ruby.

A bit of sugar

The wrap-with function described in my last post is useful, but you still end-up having to write closures which might be confusing to people who just want to write simple wrappers.

Fortunately, clojure provides the ability to enhance the language with syntactic sugar for use cases such as this one.

A word of warning

I’m obviously going to talk about macros in this article. I still think one has to postpone the writing macros as much as possible, to avoid creating code that feels too magic to the outside reader.

There are two use cases where resorting to macro is idiomatic, we’ll explore the first one here:

• macros which help define symbols, usually named def_resource_
• macros which wrap access to a resource within a closure, usually named with-resource.

The first kind of macro is used on a common basis by the clojure programmer: defn. Yep, that’s right the idiomatic way to declare functions in clojure is a macro that wraps a call to def.

The second kind’s most popular example is with-open which encloses access to a resource and ensures that it gets closed. The with-resource calls have become common idioms in clojure libraries and provide a great equivalent to the similar ruby co-block idiom. This type of macros will be described in a later post though.

Macro terminology

Macros need access to all kind of resources and reading them might be hard on the eyes at first, several people have written on the subject of macros, and books have been written that go into great detail on the subject. So I’ll just go with a cheat sheet:

The body of a macro is usually quoted

Macros insert are expanded to code, hence you must provide the s-exprs you want to be executed by quoting them otherwise they won’t be executed at the time of execution.

Beware that there are two types of quoting available in clojure:

• Standard quoting, using ’
• Backtick quoting, using ` which expands forms into the current namespace, and is generally used for macros

Accessing data from within the quoted s-exprs

There are two ways to access data from within a quoted list of expressions:

• unquote: which takes the value of a symbol and replaces it in the expanded list, ~expr
• unquote-splicing: which takes the value of a symbol pointing to a list and expands it spliced, ~@expr

The canonical unless example

Unless is the most common example macro described, let’s see how it is written

1
2
3

(defmacro unless [test & exprs]
  `(if (not ~test)
    (do ~@exprs)))

Short but dense! The code reads like this:

• Define an unless macro which takes an arbitrary number of arguments, the first one being bound to test, the rest to a list called exprs
• Test the veracity of test
• Execute the expressions in a do block

Wrapping up

Building on our previous function wrap-with, we can then help people write wrapper functions more easily:

1
2
3
4
5

(defmacro defwrapper [wrapper-name handler bindings & exprs]
  `(def ~wrapper-name
    (fn [~handler]
      (fn ~bindings
        (do ~@exprs)))))

This is somewhat inelegant since we still need to supply a symbol which is going to be bound to the handler. We can wrap it up using our previous function:

1
2
3
4
5
6
7

(defn to-be-wrapped [payload]
  (assoc payload :reply :ok))

(defwrapper wrap-add-foo handler [payload]
  (handler (assoc payload :foo :bar)))

(wrap-with to-be-wrapped [wrap-add-foo])

Room for improvement

Now let’s play a bit of magic, how about creating a macro which rebinds a symbol altogether:

1
2
3
4
5
6
7
8

(defmacro wrap-around [handler bindings & exprs]
  `(let [x#    ~handler
         meta#  (meta (var ~handler))]
     (def ~handler
       (fn ~bindings
         (let [~handler x#]
           (do ~@exprs))))
     (alter-meta! (var ~handler) merge meta#)))

Notice the last call to alter-meta1 which preserves the initial var’s metadata, such as :tag or :arglists. Now here are the macros in context:

1
2
3
4
5
6
7

(wrap-around send-command [payload]
  (send-command (assoc payload :foo :bar)))

;; store elapsed time in 
(wrap-around send-command [payload]
  (let [start (System/nanoTime)]
    (assoc (send-command payload) (- (System/nanoTime) start))))

Closing words

This is just a peak into the power of macros in clojure, and it was a fun journey getting to the bottom of the last macro. However the last form complicates reading to some extend and should thus be avoided if possible.

Functions in the wild

Functional programming rears its head in the most unusual places in software design. Take the web stack interfaces (rack, plack, WSGI), they all implement a very functional view of what a web application is: a function that takes a map as input and returns a map as output.

Middleware then build on that abstraction composing themselves down to the actual handler call.

The graylog2 extension mechanism is a good example of this, the bulk of which is really simple:

1
2
3
4
5
6
7
8
9
10
11
12

def _call(env)
  begin
    # Call the app we are monitoring
    @app.call(env)
  rescue Exception => err
    # An exception has been raised. Send to Graylog2!
    send_to_graylog2(err)

    # Raise the exception again to pass backto app.
    raise
  end
end

All the code does is wrap application call in an exception catching block and returning the original result. Providing such a composition interface enables doing two useful things for middleware applications:

• Taking action before the handler is called, for instance parsing json data as arguments.
• Taking action after the handler was called, in this case catching exceptions.

Composition in clojure

The russian doll approach taken in rack was a natural fit for clojure’s web stack, ring. What I want to show here is how easy it is to write a simple wrapping layer for any type of function, enabling building simple input and output filters for any type of logic.

The basics

Let’s say we have a simple function interacting with a library, taking a map as parameter, yielding an operation status map back:

1
2
3
4
5
6
7

(defn send-command
  "send a command"
  [payload]
  (-> payload
     serialize     ; translate into a format for on-wire
     send-sync     ; send command and wait for answer
     deserialize)) ; translate result back as map

Now let’s say we need the following filters:

• We need certain keys in the command map sent out
• We want to provide defaults for the reply map
• We want to time command execution for statistics usage

The functions are easy to write:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

(defn filter-required [payload]
  (let [required [:user :operation]]
    (when (some nil? (map payload required))
      (throw (Exception. "invalid payload"))))
  payload)

(defn filter-defaults [response]
  (let [defaults {:status :unknown, :user :guest}]
    (merge defaults response)))

(defn time-command [payload]
  (let [start-ts    (System/nanoTime)
        response    (send-command payload)
        end-ts      (System/nanoTime)]
    (merge response {:elapsed (- end-ts start-ts)})))

Now all that is required is linking those functions together. A very naive approach would be to go the imperative way, with let:

1
2
3
4
5

(defn linking-wrappers [payload]
  (let [payload  (filter-required payload)
        payload  (filter-defaults payload)
        response (time-command payload)]
    response))

Evolving towards a wrapper interface

Thinking about it in a more functional way, it becomes clear that this is just threading the payload through functions. Clojure even has a nice macro that does just that.

1
2

(defn composing-wrappers [payload]
  (-> payload filter-required filter-defaults time-command))

This is already very handy, but needs a bit of work when we want to move the filters around, or if we wanted to be able to provide the filters as a list, even though using loop and recur it seems feasible.

One of the gripes of such an approach is that you need two types of middleware functions, those that happen before and those that happen after an action, writing a generic timing filter that can be plugged in anywhere would involve writing two filter functions!

The other gripe is that there is no way to bypass the execution of the filter chain, except by throwing exceptions, what we want is to wrap around the command call to be able to interfere with the processing.

Looking back on the rack approach, we see that the call to the actual rack handler is enclosed within the middleware, doing the same in clojure would involve returning a function wrapping the original call, which is exactly what has been done for ring, by the way:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

(defn filter-required [handler]
  (fn [payload]
    (let [required [:user :operation]]
      (if (some nil? (map payload required))
        {:status :fail :message "invalid payload"}
        (handler payload)))))

(defn filter-defaults [handler]
  (fn [payload]
    (let [defaults {:status :unknown, :user :guest}]
      (handler (merge defaults payload)))))

(defn time-command [handler]
  (fn [payload]
    (let [start-ts    (System/nanoTime)
          response    (handler payload)
          end-ts      (System/nanoTime)]
      (merge response {:elapsed (- end-ts start-ts)}))))

Reusing the threading operator, building the composed handler is now dead easy:

1
2
3
4

(def composed (-> send-command
                  time-command
                  filter-defaults
                  filter-required))

Tying it all together

We have now reached the point where composition is very easy, at the expense of a bit of overhead when writing wrappers.

The last enhancement that could really help is being able to provide a list of functions to decorate a function with which would yield the composed handler.

We cannot apply to -> since it is a macro, so we call loop and recur to the rescue:

1
2
3
4
5
6

(defn wrap-with [handler all-decorators]
  (loop [cur-handler  handler
         decorators   all-decorators]
    (if decorators
      (recur ((first decorators) cur-handler) (next decorators))
      cur-handler)))

Or as scottjad noted:

1
2

(defn wrap-with [handler all-decorators]
  (reduce #(%2 %1) handler all-decorators))

Now, you see this function has no knowledge at all of the logic of handlers, making it very easy to reuse in a many places, writing composed functions is now as easy as:

1
2

(def wrapped-command
  (wrap-with send-command [time-command filter-defaults filter-required]))

I hope this little walkthrough helps you navigate more easily through projects such as ring, compojure, and the likes. You’ll see that in many places using such a mechanism allows elegant test composition.

I just uploaded a small library to github (and clojars), It’s a generalisation of what I use for recurrent tasks in my clojure programs, the fact that the recent pragprog article had a handrolled version of it too convinced me it was worth putting together in a lib.

The library provides an easy mechanism to register process for later execution, either recurrent or ponctual. It is called TRON, replacing’s CRON’s C which stands for command – as in: command run on – with a T for task.

Here is a short excerpt of what it can do:

(ns sandbox
  (:require tron))

(defn- periodic [] (println "periodic"))
(defn- ponctual [] (println "ponctual"))

;; Run the fonction 10 seconds from now
(tron/once ponctual 10000)

;; Run the periodic function every second
(tron/periodically :foo periodic 1000)

;; Cancel the periodic run 5 seconds from now
(tron/once #(tron/cancel :foo) 5000)

The lcode is hosted on github: https://github.com/pyr/tron, the full annotated source can be found here and the artifacts are already on clojars (see here). The library still needs a better way of expressing delays which will be worked on, and might benefit from macros allowing you to embed the body to be executed later. All in due time.

Lately, monitoring has been a trending topic from the devops crowd. ripienaar and jason dixon amongst other have voiced what many are thinking. They’ve done a good job describing what’s wrong and what sort of tool the industry needs. They also express clearly the need to part from a monolithic supervision solution and monolithic graphing solution.

I’ll take my shot at expressing what I feel is wrong in the current tools:

Why won’t you cover 90% of use cases

Supervision is hard, each production is different, and complex business logic must be tested, so indeed, a monitoring tool must be able to be extended easily, that’s a given and every supervision tool got this right. But why on earth should tests that every production will need be implemented as extensions ? Let’s take a look at the expected value which is the less intrusive way to check for a machine’s load average:

• The nagios core engine determines that an snmp check must be run for a machine
• Fork, execute a shell which execs the check_snmp command
• Feed the right arguments to snmpget

You think I am kidding ? I am not. Of course each machine needing a check will need to go through this steps. So for as few as 20 machines requiring supervision at each check interval 60 processes would be spawned. 60 processes spawned for what ? Sending 20 udp packets, waiting for a packet in return. Same goes for TCP, ICMP, and many more.

But it gets better ! Want to check more than one SNMP OIDs on the same machine ? The same process happens for every OID, which means that if you have

Now consider the common use case, what does a supervision and graphing engine do most of its time:

• Poll ICMP
• Poll TCP – sometimes sending or expecting a payload, say for HTTP or SMTP checks
• Poll SNMP

So for a simple setup of 20 machines, checking these simple services, you could be well into the thousands of process spawning every check interval per machine. If you have a reasonable interval, say 30 seconds or a minute.

Add to that some custom hand written scripts in perl, python, ruby – or worse, bash – to check for business logic and you end up having to sacrifice a large machine (or cloud instance) for simple checks.

That would be my number one requirement for a clean monitoring system: Cover the simple use cases ! Better yet, do it asynchronously ! Because for the common use case, all monitoring needs to do is wait on I/O. Every language has nice interfaces for handling evented I/O the core of a poller should be evented.

There are of course a few edge cases which make it hard to use that technique, ICMP coming to mind since it requires root access on UNIX boxes, but either privilege separation or a root process for ICMP checks can mitigate that difficulty.

Why is alerting supposed to be different than graphing ?

Except from some less than ideal solutions – looking at you zabbix – Supervision and Graphing are most of the time two separate tool suites, which means that in many cases, the same metrics are polled several times. The typical web shop now has a cacti and nagios installation, standard metrics such as available disk space will be polled by cacti and then by nagios (in many cases through an horrible private mechanism such as nrpe).

Functionally speaking the tasks to be completed are rather simple: * Polling a list of data-points * Ability to create compound data-points based on polled values * Alerting on data-point thresholds or conditions * Storing time-series of data-points

These four tasks are all that is needed for a complete monitoring and graphing solution. Of course this is only the core of the solution and other features are needed, but as far as data is concerned these four tasks are sufficient.

How many times will we have to reinvent SNMP

I’ll give you that, SNMP sucks, the S in the name – simple – is a blatant lie. In fact, for people running in the cloud, a collector such as collectd might be a better option. But the fact that every monitoring application “vendor” has a private non inter-operable collecting agent is distressing to say the least.

SNMP can rarely be totally avoided and when possible should be relied upon. Well thought out, easily extensible collectors are nice additions but most solutions are clearly inferior to SNMP and added stress on machines through sequential, process spawning solutions.

A broken thermometer does not mean your healthy

(LLDP, CDP, SNMP) are very useful to make sure assumptions you make on a production environment match the reality, they should never be the basis of decisions or considered exhaustive.

A simple analogy, using discovery based monitoring solutions is equivalent to saying you store your machine list in a DNS zone file. It should be true, there should be mechanisms to ensure it is true, but might get out of sync over time: it cannot be treated as a source of truth.

Does everyone need a horizontally scalable solution ?

I appreciate the fact that every one wants the next big tool to be horizontally scalable, to distribute checks geographically. The thing is, most people need this because a single machine or instance’s limits are very easily reached with today’s solutions. A single process evented check engine, with an embedded interpretor allowing simple business logic checks should be small enough to allow matching most peoples needs.

This is not to say, once the single machine limit is reached, a distributed mode should not be available for larger installations. But the current trend seems to recommend using AMQP type transports (e.g: which while still being more economic than nagios’ approach will put an unnecessary strain on singe machine setups and also raise the bar of prerequisites for a working installation.

Now as far as storage is concerned, there are enough options out there to choose from which make it easy to scale storage. Time-series and data-points are perfect candidates for non relational databases and should be leveraged in this case. For single machine setups, RRD type databases should also be usable.

Keep it decoupled

The above points can be addressed by using decoupled software. Cacti for instance is a great visualization interface but has a strongly coupled poller and storage engine, making it very cumbersome to change parts of its functionality (for instance replacing the RRD storage part).

Even though I believe in making it easy to use single machine setups, each part should be easily exported elsewhere or replaced. Production setups are complex and demanding, each having their specific prerequisites and preferences.

Some essential parts stand out as easily decoupled:

• Data-point pollers
• Data-point storage engine
• Visualization Interface
• Alerting

Current options

There are plenty of tools which even though they need a lot of work to be made to work together still provide a “good enough” feeling, amongst those I have been happy to work with:

• Nagios: The lesser of many evils
• Collectd: Nice poller which can be used from nagios for alerting
• Graphite (http://graphite.wikidot.com) : Nice grapher which is inter-operable with collectd
• opentsdb (http://opentsdb.net) : Seems like a step in the right direction but requires a complex stack to be setup.

Final Words

Now of course if all that time spent writing articles was spent coding, we might get closer to a good solution. I will do my best to unslack(); and get busy coding.

Found myself typing this in a shell:

gem list --local | egrep '.*(.*,.*)' | sed 's/^\([^ ]*\) ([^,]*,\(.*\))/\1\2/' | sed 's/,//g' | awk '{for (i = 2; i <=NF ;i++) {printf "sudo gem uninstall %s --version=%s\n", $1, $i}}'

Sometimes perl or ruby with -e is just faster

When running pf firewalls and loadbalancers with relayd, some metrics are critical. One thing that might not be obvious when looking at pf is that the maximum number of sessions a firewall can handle is fixed, as evidenced from looking at the output of pfctl -si

Status: Enabled for 10 days 00:39:47             Debug: err

State Table                          Total             Rate
 current entries                        6               
 searches                         7970148            9.2/s
 inserts                            60227            0.1/s
 removals                           60221            0.1/s
Counters
 match                              67984            0.1/s
 bad-offset                             0            0.0/s
 fragment                               1            0.0/s
 short                                167            0.0/s
 normalize                              0            0.0/s
 memory                                 0            0.0/s
 bad-timestamp                          0            0.0/s
 congestion                             0            0.0/s
 ip-option                              0            0.0/s
 proto-cksum                            0            0.0/s
 state-mismatch                        34            0.0/s
 state-insert                           0            0.0/s
 state-limit                            0            0.0/s
 src-limit                              0            0.0/s
 synproxy                               0            0.0/s

Now this can be compared to the limit values you have set which can be queried through pfctl -sm

states        hard limit   200000
src-nodes     hard limit    10000
frags         hard limit     5000
tables        hard limit     1000
table-entries hard limit   200000

In this particular example, the limit won’t be reached for a while!

By default pf has a very low state limit to cope with machines with very small amounts of RAM, OpenBSD still runs on VAX where 32m of RAM is a lot. On a default install only 10k states are allowed, for a production firewall this is likely to be too small really soon. To raise this limit use the set limit { states }. My advice would be to start at 10000 and to raise appropriately if need be.

Beware that once the state limit is reached, no new states will be allowed to keep old ones functioning leading to a difficult debugging situation.

As a bonus, here is a simple github gist which implements statistics collection for collectd, allowing you to graph state consumption and alert when the limit is nearly reached.

When you dive in a complex project, even though you try to be as good as possible with documentation, sometime you just miss things.

I’ve had a good opportunity to realize this just recently while working on puppet. A common head scratcher in puppet is finding a way to keep system, technology and platform specifics separate. I will probably write at length on this particular subject a bit later, but to give a quick explanation, just consider this quick scenario: * You host web servers for the same vhosts in two different locations * You manage them with the same puppet instance * They need a unique configuration * Some details such as their name server change

Puppet allows you to get to the point where you can just write this:

class webserver {
    include unix
    include nginx

    nginx::upstream { rails_app:
        servers => ["127.0.0.1:8000", "127.0.0.1:8001"]
    }
    nginx::upstream_vhost { "www.example.com":
        upstream    => rails_app,
        listen      => 80
    }
    nginx::static_vhost { "static.example.com":
        root    => "/srv/www/static.example.com",
        listen  => 80
    }
}

This is great, generic and allows you to deploy configurations to many machines, so it’s a bit of a hassle to have to go through the trouble of going through case statements just for DNS.

This allows me do do clever local variables like these:

---
order:  [ fqdn, operatingsystem ]
values:
    ns_server:
        www01.remote.example.com: "10.1.1.1"
        default: "10.1.2.1"

And call them from classes like this:

class unix {
    [...]
    $ns_server = yamlvar(ns_server)
    file { "/etc/resolv.conf":
        owner => root,
        group => $root_group,
        mode => 0644,
        content => template("unix/resolv.conf.erb")
    }
    [...]
}

Well this is all fine and dandy until I came upon this: http://www.devco.net/archives/2009/08/31/complex_data_and_puppet.php . That’s right, @ripienaar already did the work and guess what ? http://www.devco.net/archives/2010/09/14/puppet_261_and_extlookup.php it’s already in puppet 2.6.1 !