Like everything at a 'proper' tech startup, resources are limited at tribily. Combine that with the fact that all sysadmins are lazy, and we had a clear need to automate as much as possible. There are many factors involved in properly automating a linux system administration environment, and we have made our toolset more complete over time. I'll write on a bunch of the tools involved in the future, highlighting one every time. I'll start today by one of my personal favorites in the DevOps scene: Puppet

I can easily say that getting properly introduced to puppet is one of those before-/after- experiences: after you start using it for everything, you wonder how you did things back in the dark ages without it. 

For those of you that are not familiar with puppet: it basically allows you to write scripts (referred to as recipes) that describe what you want your servers to look like. For instance, apache servers need the package apache2 installed, and a directory /www/myvhost created that will hold the files for the site.

These recipes are then applied to your servers using the puppet agent, which will make sure the package is actually installed and the directory is created if it is missing.

Getting started with Puppet is quite a learning curve, but asking the good folks on irc (#puppet on freenode) and having some great experts in my personal network allowed us to get started fairly quickly. We made the tough decision to not do anything on the new servers manually last year when we switched providers from linode to hetzner. That made the migration quite a bit longer (I'd estimate roughly 50% extra time investment), but the benefit is massive: our new server cluster is completely puppetised. That brings along a whole slew of advantages:

1) version controlled system administration

Since all puppet recipes are defined in a flat file structure on a central machine (the puppet master), it is very easy to check them into your favorite version control system (we use git in combination with a bronze business account on github, so we have all our git repo's in one off-site place). All of a sudden, your system administration efforts become version controlled. Combine that with proper server monitoring (we obviously eat our own dog food) and all of a sudden it becomes relatively easy to find the cause of anomalies: was anything checked into puppet right before that cpu load started climbing? Roll back that change in puppet, reapply your recipes and system load will go back to normal (obviuosly depending on how reversible the change is ;-) )

2) easier collaboration

Since we work with multiple sysadmins in multiple timezones, finding good ways to collaborate is important to us. Puppet has been an unexpected help in this. By having every commit commented, it becomes fairly easy to stay up to date on what the others have been doing when you were sleeping: just pull the latest version of the git repository, read the commit logs and you should be up to date.

3) more accountability

It becomes much easier to see who did what, when and (usually) why. The so aptly named 'git blame' functionality makes it very easy to figure out who is responsible for a certain change. It's then much easier to go and ask that person why they made the change they made, and from there you can work to find an appropriate solution together.

4) set up development and staging environments

Using a feature of puppet called environments, it becomes very easy to set up different.. euhm.. environments. An environment is basically a way to create a similar deployment of servers which has only slightly altered properties. For us, that means we have a staging environment defined in our puppet repository. All we need to do is set up a few virtualboxes locally, set their puppet environment to staging and run puppet on them and it will setup these servers 99% the same as the production servers, except with different IPs and passwords and a different DNS setup. These staging environments can then be used to test new puppet modules before they go to production.

5) provide a baseline config across all servers

Another big benefit of puppet is that if implemented well, it creates a very nice baseline that you can use across all your servers. Have those pesky tools that you use so often available everywhere, have your vim settings available everywhere, and your ssh keys deployed on every server in the same way. It just saves time when you work across different servers all day.

6) easily provision new servers

this one requires you to set up your puppet repo properly, which is something that in our case gets better as we go along (The first version was pretty horrible). If you properly build your modules and services, and use proper templates, all of a sudden it becomes a breeze to set up a new server: add a node to site.pp, attach the correct classes to it, run puppet once or twice on it and voila, a fully configured server.

7) no need to reinvent the wheel

Since a lot of puppet users release their modules as open source (Just see this search for puppet on github), it becomes very easy to install new software and configure it properly on your servers. Especially when it comes to not-so-well documented software, this can save you hours of hassles setting it up.

8) easier access management for ssh users

Every system administration department changes over time, people come and go. Maybe you work with consultants who need temporary root access on your servers. When you are in need of taking away acccess, you change 'present' to 'absent' in one place, and after your next puppet run the specified user will no longer be able to log in to any of your servers. No more chances of forgetting to take away access on that one auxiliary server. Security win!

i'll be writing more on puppet and DevOps in the future, so stay tuned! In the mean time, go familiarize yourself with puppet if you haven't already done so!