RIP Schaltwerk
On Friday July 8th 2011, I said goodbye to a dear friend, one who stood by my side for far longer than was expected of her. Her passing was with some sadness, but her memory will live on.
Schaltwerk.riotnrrd.com began her life in a Magitronic assembly house on September 25th, 1994. She was a very fast machine for her day; although Pentium processors had begun to arrive in the local computer stores they were still thousands of dollars, and as a 486DX2-80 boasting a 40MHz bus she gave machines twice her price a solid run for their money. I worked at the time at the larger of the two local computer stores, and as such I was able to source a single 16M RAM SIMM for far below the retail price. I (or rather my father) paid $800 for the RAM alone!
Schaltwerk spent the next few months running OS/2 Warp, supporting an active Telegard BBS and many, many hours of Doom, Doom 2 and Heretic deathmatches over 14.4k modem - and later over a local ARCNet network, cabled through my parents house with chained 25' phone extension cords from the dollar store. When I left for college in January of 1995, she came along as my primary college computer, the task for which my father had intended her. At college she continued to support the BBS, running Renegade and then Iniquity software, but the BBS was often suspended to allow me to spend long nights mastering Autodesk 3D Studio and Photoshop. At some point I also discovered Linux, and when a friend began handing me surplus computer hardware from his work, I was able to add several more hard drives and increase her RAM to a whopping 40 megabytes. I would give a lot for photos of my workstations at this time, but as far as I know none exist.
As college came to a fruitless end - a diploma, but zero job prospects - I took a job as a graphic designer for a college web project. Schaltwerk was the main graphics workstation, putting in months of midnight-until-dawn marathon Photoshop sessions. I was also working hard on my own Linux interface design projects, working closely with the Enlightenment window manager team.
In about March of 1997 I moved back to Sussex and took a government-funded web design position. I was offered a Pentium workstation of my own, but after struggling with drivers and software installs and an unstable machine, I moved Schaltwerk into the office to be my primary workstation. This didn't last too long, as I couldn't deal with the lack of computer at home! Schaltwerk, nestled in her basement lair in my parents' house, had sprouted two more monitors - a monochrome display addressed with a second video card and a Wyse 60 dumb terminal attached to the serial port now accompanied the main SVGA monitor. One mouse, two keyboards and three displays - pretty fancy stuff for 1997!
In 1998, my friend Darren, my baby sister Jen and I packed all of our worldly possessions into Darren's car and drove across the country from New Brunswick to Calgary, Alberta. We only had about $800 between us, but with one minor speedbump we managed to get settled and employed and much to our parents' collective surprise, we made a go of it. After we all landed jobs at a major ISP, Schaltwerk became a networking powerhouse, having half a class-C subnet (128 addresses) of real internet IPs delegated to her for several months! Of course, at the time I really didn't know what to do with that kind of resource, so I occupied myself learning Linux networking and DNS, and Schaltwerk got her first live, static-IP instances of BIND, Sendmail and Apache. Thankfully by this time I had gotten over the debilitating CircleMUD addiction I picked up in college!
In early 1999, my new girlfriend and I moved into a new house with our friends Ivan and Andy, who were running what was at the time the most technologically advanced Shoutcast station in the world, BeNOW. I became their network administrator, and together we whipped eleven machines and hundreds of gigabytes of storage (a big deal at the time) into shape. Schaltwerk took over as the router and firewall, also handling DNS and mail services for the BeNOW and riotnrrd domains, as well as primary and secondary DNS for dozens of other domains.
In late 1999, I had a job offer in Vancouver, so we packed our things and moved into a geek house in East Van. Schaltwerk stayed in Calgary and went to live with my friends Jonnay and Shell, where she spent the next few years humming away under Jonnay's desk. During this time, she remained on a static IP address, becoming the primary DNS service for scores of domains, handling primary and backup mail services for dozens of others - but most importantly, she became the webserver for a number of domains. The most popular by far of any of the websites hosted on Schaltwerk was eastvan.bc.ca - a Slashcode site boasting 'News For Crackheads - Nothing That Matters' which quickly gained notoriety as a gathering place for Vancouver's dot-com underbelly. Most of the people I call close friends can trace their roots in our friend group back to eastvan.bc.ca. During this time, Schaltwerk also hosted the Black Hole Club email list, gathering a sizeable portion of Vancouver's electronic music production scene together online.
In 2001, we moved to Costa Rica, leaving Schaltwerk with Jonnay and Shell - to her credit, Schaltwerk worked almost completely without interference from her hosts, only requiring several reboots and a hard drive replacement over the four or five years that she spent in their home office. In 2003 we moved back to Vancouver, moving into a house on 10th Avenue. Schaltwerk became part of a cluster of media and internet servers driving the geek house, which we dubbed 'Pod6', a reference the Adult Swim cartoon 'Sealab 2021'. For a while Schaltwerk ran the website, but soon the site outgrew the humble 486 and we built an upgraded machine, relegating Schaltwerk to just email and DNS.
In 2005 we purchased our first home and Schaltwerk took her new position - alongside a server from Jonnay and Shell, in reciprocation for their years of hosting - in a basement closet. Too slow now to support much in the way of modern web services but still providing email and DNS services for dozens of domains, Schaltwerk also provided SSH shell endpoint access, allowing my friends and I to casually tunnel through even the fiercest of corporate firewalls.
In 2007 I began the long process of migrating all of the services off of Schaltwerk and onto a third-party host, Dreamhost. Many domain owners had to be notified, many small webpages had to be migrated and dozens of cryptic user scripts had to be decoded and disabled or ported. By 2008 almost all of this work was complete, and Schaltwerk remained online but rarely used until April of 2009, when I moved out of my basement and aboard the S/V TIE Fighter. I could not bring myself to just throw away a machine with such a history of faithful service, and so I brought her aboard, intending to find a way to celebrate her life.
Schaltwerk gave me fifteen solid years of faithful work, far more than can be reasonably expected of a PC. Her only fault was her lack of processing horsepower, and while I will admit that I entertained fantasies of one day putting her back in service as a terminal somewhere, life on a sailboat is not kind to electronics, and a slow death in a storage locker just wouldn't suit her. With a few respectful words about her life and service I sent her to her final resting place in the ocean, about a kilometer off the Sunshine Coast.
I have to admit it took a few minutes for the lump in my throat to pass.
Ok, Really Screw It!

OMG TEH BLOG CAN HAS PHOTOZ NOW!!!111!eleven
OK, FINE! I WILL SPEND THE HOURS AND HOURS TO MANUALLY MIGRATE ALL MY OLD CRAP OVER!
There. Are you happy?
Actually, I am. Wordpress is waaaay nicer than Drupal, at least for a blog. Using Drupal for a blog was kind of like driving a Jeep - I mean, sure, it's rugged and capable and even beautiful, but it uses a lot of gas, it's noisy, and while the ragtop is nice in the Summer it's just impractical in the winter. Really the only thing that prevented me from moving earlier was the complete lack of Drupal-to-Wordpress migration scripts. Yes, I did in fact have to migrate each and every post by hand.
Oh well - at least now it's done, and I have a blog I can be proud of again. Welcome to 2001, Drew - the internet now supports fancy things like "photos" and "videos" and "multimedia"! Good thing I managed to lose my camera battery charger in the move. Nice going, Drew. Seriously though - in order for Drupal to have the simple feature "add image to the blog", you had to resize the image manually, upload it to the FTP site manually, and type in the full path to the image. Now *that* is some serious 2001 stuff right there.
The downside of this whole migration fiasco is that I've basically had a mental block against doing any sort of blog updates ever since I committed to the jump. That means that it's been almost three. frackin'. months. since my last update - and honestly, this has been one of the most action-packed, adventurous, utterly insanely awesome Summers of my life. I don't even know where to start. I guess it'll be roughly at where I left off, back in June...
It’s The Future!
Some days, you just have to shake your head.
I just got off the phone with a colocation facility in Houston, Texas, trying to figure out why our servers are dropping like flies today. Four down so far, and signs of trouble on a few others, almost certainly foul play - I'm trying to keep in mind the old sysadmin truism 'Never attribute to malice that which can be adequately explained by stupidity'. Still, my lip curls a little at the thought of some fourteen-year-old kid in his mom's basement in Iowa sending his massive botnet up against our servers in response to some unrealized slight. Or maybe he's earning money somehow, or even just the admiration of his peers. Or maybe he's just being a dick. *shrug*.
I'd say "he or she", but I have yet to meet a girl with both the skills to launch a DDoS attack and the sociopathic tendencies to actually do it. I'm sure she's out there somewhere.
Just to juxtapose, by "phone" I actually mean "Skype from my tiny laptop, sitting in the sun on the roof of my sailboat". Seriously, had you told me five years ago that this would all be possible, I don't think I would have laughed outright, but I would probably have smirked. All of these little incremental upgrades in technology and lifestyle have slowly and quietly added up to the WORLD OF TEH FUTUR3!!@1 that we were promised as kids. It's finally here!
...now where's my rocket car?
Anyhow. I was actually only using Skype because my cellphone batteries are currently dead, and I left the charger in New Brunswick the last time I was there. Since it's a basic no-frills Motorola flip I can charge it up with a regular mini-USB cable I had lying around, but unfortunately my MacBook Air only has a single USB port. I get to choose - would I rather have my cellphone charged up, or would I rather be on the internet? Internet, I choose you.
Also in the realm of electricity, my diesel engine currently isn't starting. I turn the battery selector to the starter battery bank, I pull the power toggle, flip on the lights and test the alarm to verify that the panel is receiving power, hit the starter button aaaaaand... nothing. Nada. Not a grumble, not a click of the solenoid, nothing at all. I'm hoping it's just a wiring problem, as the wiring is a real mess and I may well have accidentally disconnected a wire when I was cleaning up the other day, but frankly I can't tell.
To add to that problem, I have no water. I'm drinking (and cooking, and cleaning) from a 10l jug currently, because my water tanks are empty. Filling the tanks requires motoring up to the nearest hose, and, as I said, my engine isn't starting. *sigh*.
Man. I think it's time to convert this blog over to WordPress. I've been using Drupal for the past year, and in a solid year I have yet to find a reasonable way to upload photos. For instance, I'm sitting less than four feet from the exposed wiring rats nest that is my starter panel, and my laptop has a camera - but for me to take a photo of that and post it to the blog involves using Skitch to take and resize the photo, uploading the photo to the webserver, and referencing the URL using full HTML tags. I can do that, but I don't wantto. With WordPress, I can do all that in one step, and I think that'll make a huge difference for me.
Anyhow - an email just came in requesting an RSVP for the Open Web Vancouver speakers' pre-conference social next Wednesday night - and I'm realizing that I'm just over a week away from speaking to 400-odd geeks about work that I'm currently procrastinating against. Soooo... I'm going to cut this abruptly and get back to work.
CloudCamp Aftermath
Wow, that was great! It's so nice to finally be interested in a technology again - I was honestly starting to think that I'd never enjoy another conference. Half of the fun of a conference is discussing new concepts and ideas in technology, but the other half is meeting up with folks who have similar ideas and interests, both on a professional level but also in a social (ie: beer) environment.
It was also my first "un-conference". The idea behind this is that there's a schedule, but none of the talks are booked in advance, and people come with a talk or presentation that they'd like to give and sign up to do so on the spot. In the end it's more about discussions than presentations, and in fact I was drafted (or more accurately, "tricked") into hosting a discussion group on scaling Drupal in the cloud.
Now *that* was an eye-opener - I had naïvely assumed that most people working with cloud computing were working with web applications, as that's what most of the documentation out there seems to be. Or maybe I just pitched my session wrong, and should have stressed all web applications and not just Drupal. Or perhaps I shouldn't have scheduled the discussion at the same time as Dan Kaminski's discussion on cloud security. Regardless, the room began with approximately eight people, and as it became apparent that we were really looking for a technical discussion but most of the people were in marketing and management, the room shrank to four people.
What was awesome was that the four people were:
Wild! I got more out of that half-hour session than I have reading documentation for the past month and a half. As the session started, we basically went around the room discussing backgrounds and technical histories, and it became apparent that I was really the only person in the room with any technical experience at all working with the EC2 cloud, I began to present my experiences to date, focusing on the three big problems that I had butted my head against - how to "auto-scale" web front ends using load balancers, how to scale MySQL databases elastically, and how to share storage between EC2 instances. As I discussed these, Dean Dierickx from RightScale showed up, and had some interesting notes to add from a high-level design perspective, which we talked about at length. We then started discussing database scaling, and it became apparent that nobody in the room had any practical experience with that at all.
Right about then, Jonathan Lambert from Work Habit let himself in, and after listening to us fumble about for a couple of minutes, grabbed a whiteboard marker and launched into a fifteen minute in-depth technical discussion on different methods of scaling MySQL. This was *great* stuff, though it pretty much cleared the room of the marketing and manager types. As he finished, he validated my whole trip with one statement - he said something like:
"Now, there are three major problems that everyone attempting to scale any web application in the cloud butts their heads against, and there's no simple answer to any of these: how to launch instances automatically from a load balancer, how to scale MySQL, and how to share storage between instances. We've fought these problems hard for two years now, and we still don't have a good answer, but here's how we've managed to get everything working so far..."
...and proceeded to lay out a basic, scalable platform for Drupal on the whiteboard. Most of his layout matched my work exactly!
To top that off, just last week I downloaded a script from the RightScale website that purports to set up a basic EC2 instance to be a "RightScale" machine, and poked through the script to see exactly what they're doing to prep a machine. I was shocked to see that every step of the way, their work matched what I was already doing to setup my own EC2 instances! That may not sound like much, but frankly after fifteen years of Linux administration, I have some particular ideas about how a machine should be setup for optimal networked administration; which additional packages should be installed, which default services should be turned on or off, what changes should be made to the default shell environment, etc. Seeing two large, successful operations doing exactly the same things as I do in my personal environments, and facing exactly the same challenges, was vindication to say the least!
Anyhow - even though it was a "free" conference (with the quotes of course implying bus fares and hotel fees), I think I got well more than my money's worth.





