Step By Step AWS EC2 Tutorial

Do not use any of the defaults shown below. I would no longer recommend a security group rule for port 22 to 0.0.0.0/0 This has been roughly adapted from this ServerFault question for the case when it gets removed/deleted/closed. The question was about how to configure a Flash game server on Linux, but on EC2. I had a good look around, but didn’t find any true step-by-step EC2 tutorials for proper beginners....

February 17, 2013

Building and Scaling PDFTribute

This is the brief story of how PDFTribute.net came to be, and survived being linked to from some of the top world news sites. My good friend, Patrick Socha, so moved by the outpouring of data associated with the twitter hashtag #pdftribute set up a quick and dirty twitter archive, extracting tweets containing links. I had a look at this, realised it looked awful on mobile (I was on a bus at the time!...

February 5, 2013

2012: Retrospective

2012: A year in review. So.. 2012.. What can I say.. Quite a lot, actually. When I last wrote an annual retrospective, it was 2011 going into 2012, and I’d just started at Baseblack, one of many Soho-based VFX studios. I had a good 12-odd months there, before the credit crunch hit the entire London VFX industry, and I was made redundant in October. Over my time there I wrote a lot of puppet manifests, built a render farm based on Dell Blade servers, undertook a Hitachi HNAS administrator course, learned how to use Maya, Realflow, Nuke, Shake, PFTrack, Silhouette, and a whole bunch of other VFX packages too numerous to mention....

December 31, 2012

Dennis Nedry and the Human Single Point of Failure

“John, I can’t get Jurassic Park back on line without Dennis Nedry.” Words you never want to hear uttered. Unless you work for InGen, it’s highly unlikely. Although there is the remaining problem of the Human Single Point of Failure (HSPOF). After you’ve spent the last year or two eliminating the single points of failure from your computational infrastructure, you realise that you’re the only one who knows which cronjobs run when, and on which servers....

December 25, 2012

Transferrable Skills in Higher Education

So.. it transpires that I have a friend who studied Physics at Imperial College, and as a part of that, was taught how to use C++ “As a tool to help with computational physics”. - His words, not mine. As a result, he has no explicit knowledge of some of the finer points of C++ programming, no idea how a binary search algorithm works, why you’d use a Deque and when you’d use a Vector....

December 10, 2012

Jenkins as a Job Dispatch Engine

I get easily tired of doing the same thing over and over again, and will, wherever possible, script or automate it to make life easier for myself. This could be in the form of a lightweight webapp/REST api for stuff, or in this case, I used Jenkins. So on one server, we sometimes need to reload apache. As we don’t like developers randomly executing shells on live servers, it’s better to just allow access to a few specific commands, in this case, a wrapper script on the target server’s /usr/local/bin that just wraps “/etc/init....

December 4, 2012

Interesting Thing Of The Day: Network Motifs

Interesting thing of the day: Milo, Ron, et al. “Network motifs: simple building blocks of complex networks.” Science Signalling 298.5594 (2002): 824. Fulltext available from Google Scholar: - http://bit.ly/YAstgD It occurs to me that in scalable systems engineering (the sort of thing I do for a living), you only tend to see Bi-fan networks and Bi-parallel ones. Bi-fan is rougly equivalent to a cross-connected core switch whereas Bi-parallel is a good representation of a Virtual IP with Load balancer....

December 1, 2012

Thanksgiving 2012

I think this seems like an appropriate time to say a few words in favour of the great United States of America. There’s some things they just do excellently. Customer service springs to mind as one of the best I’ve ever encountered. I’m not exactly sure why this is, but people do seem to be far more willing to be kind, courteous and helpful. There’s other things too, the weather is, generally better (and if not better, then certainly more predictable)....

November 22, 2012

GWAN: Snakeoil Beware

I’ve heard quite a bit about the “G-WAN Application Server” over the past few weeks. Initially it was a Serverfault question that left me thinking “WTF” (http://serverfault.com/questions/445835/virtual-host-on-g-wan) I took a look at their website and thought: “Those are pretty insane claims”. They’re also the kind of crap you tend to see where the intended audience is somebody who has absolutely no clue about scalability, or production-readiness. Y’know, Managers. Quite well summarised by this comment: GWAN isn’t designed to be a robust webserver, it’s designed to perform exceptionally well in contrived and outlandish benchmarks, so PHBs will demand the IT team use it and buy support… – Chris S♦ 19 hours ago Interestingly enough, the only person who answered that question was a fellow called Gil who apparently works for G-WAN....

November 12, 2012

Answered: Network Design

I’m looking to setup a site with 49 edge switches connected by fiber to a central switch. 3 VLANs will be setup to handle data, telephony, and streaming media. each edge switch should have provision for 2 SFP modules for failover, and the core switch needs to have the provision to handle this failover. i’m getting lost on the Cisco site with their specs and recommendations. if anyone could suggest a suitable model number for the core switch and the edge switch, it would be really appreciated....

November 6, 2012