Josh Samuelson

It’s my blog.

Rinse Repeat

The last few months have been a major crash course for me in configuration management and Puppet in particular. I wouldn’t call myself an expert at least considering what counts as a puppet expert at Puppet Labs, but I feel like I know enough now that I could easily do my previous job almost entirely using puppet. A big part of my job now is building VMs that we will use in the classroom. The part that has really changed my perspective is how freely I can now throw something away that doesn’t work. My attitude with computers has often been that if something is seriously screwed up, it’s often simpler and faster to just start fresh rather than figure out some arcane issue. This is espeically true with Windows desktops OS’s. It’s also why I get a little annoyed when someone asks me to fix their computer, because if it were mine, I would just wipe it out and start fresh. Doing that too often can be a big pain and with servers it can be seriously risky because it’s easy to miss something important.

That’s where puppet has really changed my perspective. I realized that I know longer think of the server (or the VM image in my case) as the “thing” I’m managing, that’s just the expression of the real thing. The puppet code is what I’m managing and maintaining. That let’s me make and undo small changes as I work toward improving things, without having to worry how undoing a change might cause issues down the line. It’s so easy to start fresh that I don’t mind trying something out, throwing away the result, and trying another variation until I find the right solution.

I think this isn’t quite what puppet has been used for, or maybe what it was initially intended for, but I think it’s really the future of how people will manage most of their systems. Puppet is a great tool for addressing drift, i.e. when something has changed on one system that should otherwise be identical to the others. In the long run, I think it’s going to become simpler to just trash a system that has drifted and redeploy from the code.

If you think of it in terms of software its more obvious. If a piece of compiled software becomes corrupted, say because of an issue with the disk, you would never go in to the file and try to repair it so that it matched a known good copy. No, the correct thing to do in that case is to simply redploy the file. Now that so many servers are virtualized or containerized, I think it makes much more sense to think of them as files.

What’s been remarkable about this for me is how much it’s shifted my thinking. Even when working on projects at home I find myself wanting to set things up in puppet just so I don’t have to worry about it when I need to start fresh. It also has me doing something I’d never bothered with before for person tinkering, documentation. I’m actually documenting my work as I go by writing the puppet code.

Contain Yourself

My most recent project has given me a chance to play with docker again. I first heard of docker when I was in the midst of trying to learn puppet. I remember thinking to myself “This is great, I don’t have to learn a whole new language, I can just do everything in shell scripts in my Dockerfile, if it gets screwed up I’ll just trash it and build a new one.”

Now that I understand how puppet works and what it’s really for, I realize how silly that way of thinking is. Using docker that way is essentially a more conventient way to manage golden images, but it still fraught with the maintenance nightmares that come from custom setup scripts and golden images. I think the idea of immutable infrastructure is appealing, but it doesn’t solve the same problems as configuration management.

What I have gotten a chance to explore is what else docker is good for. I’ve been using it for sandboxed nodes in a standalone mockup of a puppet based infrastructure, i.e. a Puppet Enterprise master that also hosts a bunch of docker containers that pretend to be nodes. As far as the master is concerned, they are totally separate machines and it offers a good way to simulate a larger environment in a lightweight way. That’s not what docker was designed to do, per se, but it’s working amazingly well.

Since I wasn’t really using it for it’s intended purpose at work, I decided to play around with it at home. I reformatted my iMac, which had been running Mint. It had been acting a little weird, it was having some package management corruption issues, the kind of thing I hate debugging, so I just wiped it out and installed Ubuntu 14.04. One of the things I lost in that was my installation of PLEX which meant I couldn’t stream shows from the mac to my TV anymore (Sara was a bit bothered because she wanted to watch MASH). Since I had a fresh install with nothing but docker and puppet installed, I decided to try a dockerized version of PLEX. I found an appropriate image on docker’s website and ran it using Gareth Rushgrove’s docker module from the puppet forge. Basically all I had to do was declare the container with a couple of folders mapped, one for config and one for data, and that was it PLEX was running again.

That’s what docker is good for, to save you from the dependency hell that so often comes with running applications. It lets you say “I want this application, here’s the config and data” instead of saying “I need to install a bunch of stuff I don’t care about, some of which might conflict with what I currently have installed and also might break because of network issues or corrupted packages, just so I can get to the application I actually want installed.” Docker is especially wonderful at handling packages that have conflicting dependencies. Because everything is inside the container, you just don’t have to worry about it.

The way I see it, docker solves a real problem that has driven SysAdmins totally nuts. Docker allows clean and easy packaging of applications, it removes the unknowns of deployment and allows apps’ dependencies to be pinned to a version. If designed correctly, they also allow package maintainers to easily update internal dependencies and application code without worrying about all of the possible platforms that users might be running. It does this well. It does it so well that people begin to think they don’t actually need configuration management. Depending on your environment, you might not, or might not need much more than a simple manifest and puppet apply.

The thing is, puppet and docker solve different problems. Puppet is often used to solve some of the problems that docker solves, that’s true. For those cases, docker is better, hands down. Puppet does make it much easier to manage applications and all the hassle of updates and patches. But that isn’t really what puppet it for. Puppet is a way of defining infractructure in a declarative way. It’s a way of defining the finished state of a system and a tool for making that happen. I’m really glad that Gareth built the docker module for puppet because it’s a great example of how puppet and docker work well together.

My PLEX server is case in point, it’s defined by puppet, and the “application” (i.e. the docker container of the PLEX app and dependencies) is managed by puppet. Docker takes care of what docker is good at, puppet takes care of what puppet is good at.

Learning With Instant Feedback

I recently watched this somewhat mind-blowing video. I know he’s making a larger point about purpose in life, and I think he’s got a point, but really the part that blew me away was the idea of instant feedback not just as a “crutch” for learning, but as he put it “writing code without a blindfold on”. It also occurs to me that part of the reason more people don’t code is because it involves different parts of the brain, you have to constantly visualize data and what’s happening inside the code while at the same time using your analytical brain to actually create the code.

I know there are people who can do this, to some extent I’m one of them. I also know that I sometimes have trouble forming complete sentences when I’m deep in hack mode working on some problem. It’s also why something like IntelliSense in VisualStudio is so popular, it gives you instant context.

I used to think of that kind of think as cheating, a crutch for those who couldn’t remember how to do things. In retrospect, that attitude is probably what kept me from doing more coding after college. It’s also an attitude that was reinforced in my CS classes.

Now that I actually have to do some programming as part of my work, I don’t really care about cheating. I’ll happily use every trick in the book to make things better or build them more quickly. This is especially true of copying or repurposing what others have done (with proper attribution obviously).

With that in mind, I’ve been trying to think of ways that I could make writing puppet manifests have a little more instant feedback. Khan Academy has a great project that they use for teaching basic programming with JavaScript. It allows a student to write their code and see the results even before they’re finished. This makes errors like missing semicolons at the end of a line almost impossible, because the feedback is so immediate.

I think some of the could be translated to puppet. For example, as a student types code into an editor, puppet-lint could be checking the syntax live. That shouldn’t be too tough and could maybe even be done by adapting Khan’s editor and just wiring it up to a different output. But the output of puppet is more tangible, in order to check the results of a manifest, it needs to be applied to an actual node and then the node’s state needs to be checked.

There are a few pieces to this puzzle:

  • A responsive editor with live feedback on code
  • A sandbox environment where puppet code can be tested
  • Serverspec (or similar) tests with a live display of environment status
  • Some way of arranging the whole thing

This isn’t really on my official project list, but I’m going to try to tackle this one. I’ll also write up as much as I can and make a blog series out of it.

Blog Reboot

Things have changed quite a bit for me in the last year and I’ve been feeling the urge to start blogging again. Since my previous blog a bit of a mess and the content was all over the map, I decided to start fresh and keep the content more focused. This time I’m going to focus on projects, mostly tech projects, but maybe I’ll throw in the occasional cooking or brewing adventure.

The other reason for the reboot is that I’m sick of dealing with Wordpress’ nonsense, most recently it was alerting me to an update and wasn’t letting me update. Rather than deal with the pain of fixing it by hand, and possibly redeploying and restoring I thought I’d just give octopress a try.

My plan is to move this into S3 eventually but for now I’m just using the built-in github pages. I like the idea of replacing a heavy slow platform like wordpress with something totally static in S3. Once that’s done, I may even transfer my domain over to Amazon too and just get rid of my old hosting provider.

I’ll probably recycle some of my old blog, but some of it I’ll leave behind, for now I’m just going to focus on new content.