Puppet really is great. Don’t ever get me wrong there. It’s saved me masses of time and money over the last few years, and allowed me to do my job quickly and efficiently. That said, it really does have issues with scalability. After about 20-30 clients using WEBBrick, everything kinda falls over a bit. We had this problem at Baseblack. We’ve now got ~60 workstations and rendernodes all using Puppet for configuration management and software deployment. It’s great. It far simplifies the process of rolling out updates and upgrades to new build machines. ...

Originally a comment here In the above article, Bhumi gives a method for storing files in the database, using MySQL and PHP. My personal distaste for PHP aside, I don’t think I could ever find a reason to store files in the database, rather than on the filesystem. I’m also primarily talking about RDBMS type databases, not NoSQL, which tend to have a mechanism for storing files a little bit more sanely than “old-fashioned” databases. ...

Firstly, welcome back. It’s now 2012, and there’s lots more to write about. Recently, Oracle withdrew the ability for Linux distributions to repackage Java and distribute their own packages. This has been widely regarded as a bad idea. I tend to agree. So, let’s re-roll an old sun-java6 deb file, with a new content to contain the latest 6u30 java release. You will need: A set of build packages (I’ve got a set for lucid, so if this goes away, I’ll find some way to host them.) from http://archive.canonical.com/ubuntu/pool/partner/s/sun-java6/ The latest Java packages: http://download.oracle.com/otn-pub/java/jdk/6u30-b12/jdk-6u30-linux-i586.bin and http://download.oracle.com/otn-pub/java/jdk/6u30-b12/jdk-6u30-linux-x64.bin dch. just install devscripts package to get this. Some idea of how packaging on debian/ubuntu works. Let’s get started. ...

Hah. So it’s New Year’s Eve. Twitter is down, and has been for about 3-4 hours. That’s because the NYE celebrations have already started. Somewhere on the other side of the world. I include Twitter on my personal website. The one you’re reading. There’s a template tag that displays my five latest tweets. About 2-3 hours ago, I got some error reports about XML parse errors on that template tag. I use api.twitter.com and pull in the XML feed for parsing. No problems there, it’s always worked. ...

Right. Let’s have a look at the massive technical implications of the Fix Puppet idea. As I mentioned in my earlier blogpost, in order to fix puppet in a sensible way, we’ll have to review all, and overhaul some of the underlying infrastructure that allows it all to run. The interlinks and dependencies between all the parts are a little tricky to visualise. So, here’s a picture. ...

I’ve decided. We need to start doing points poker here at Baseblack if we’re going to carry on this Agile DevOps thing. I’ve got to admit, the first time I came across the Agile methodology was quite late in my career. In the past, prioritisation of “operations” projects was reasonably first come first serve, or by order of priority (frequently, business need, and seldom operational requirement). For software development teams, Agile is a pretty good, native fit. The concepts embodied by stories and sprints fit a development team very cleanly. When it comes to systems administration and engineering, or what I’ve come to refer to as DevOps, Agile can be a bit more awkward initially. ...

Our new PowerDNS cluster (of 2 nodes, so far).. Is backed by Postgresql. In the past, I’ve found that Postgres performs far better for a PowerDNS backend, than MySQL, and certainly better than the BIND, LDAP or SQLite backends. Until version 9.x, Postgres replication was a pretty sorry state of affairs. There were a few options for replication. Slony was commonly used, if not very good.. You’d tend to get a horrific SPoF around the single master. In total, there were 9 or 10 different third party solutions for Postgres replication and clustering. They all had their pros and cons, and some were great, and some were downright awful. ...

This article relies on having a soon-to-expire SSL certificate on an Apple OSX Server. Ours are running Snow Leopard, and I’m yet to try the whole thing on Lion. I’ve got to admit, I went through a bit of a rigmarole to do this. To generate a new certificate, you need a key, and a CSR. To get the key, you need to export a PKCS12 file from KeychainAccess as ROOT. Yes, Root. Yes, OSX = Toy operating system. No, another admin user won’t cut it. Yes it’s a pain in the arse. ...

A little background, perhaps: Last night I attended a Winter Party at a bar called The Sterling, in the ground floor of the Gherkin. I have absolutely no problem with the organisation of the Party itself, but more complaints about the Venue. I drink in various bars quite a lot. I’ve worked in a few bars. I observe how bar staff operate, and how their tills work. There’s a few longstanding massive problems with pretty much every till/billing system I’ve ever seen in a bar. ...

It’s only a few days after Black Friday, and eBuyer have experienced their very own Black Monday. I’ll set the scene. This morning, I had an email mailshot advertising a £1 sale of clearance range items on eBuyer. Sounded like an ideal plan, especially if there were any hard disks up for grabs. I never actually got that far though. I followed the “sneak preview” instructions, duly “liked” eBuyer on facebook. 10:30 rolled by, and oh look. Error 500 from eBuyer. Refresh. Try again. Same error. Connection terminated. Session reset. Page returned no data. ...