RIP Schaltwerk

On Friday July 8th 2011, I said goodbye to a dear friend, one who stood by my side for far longer than was expected of her. Her passing was with some sadness, but her memory will live on.

delivering the eulogy
delivering the eulogy

Schaltwerk.riotnrrd.com began her life in a Magitronic assembly house on September 25th, 1994. She was a very fast machine for her day; although Pentium processors had begun to arrive in the local computer stores they were still thousands of dollars, and as a 486DX2-80 boasting a 40MHz bus she gave machines twice her price a solid run for their money. I worked at the time at the larger of the two local computer stores, and as such I was able to source a single 16M RAM SIMM for far below the retail price. I (or rather my father) paid $800 for the RAM alone!

Schaltwerk spent the next few months running OS/2 Warp, supporting an active Telegard BBS and many, many hours of Doom, Doom 2 and Heretic deathmatches over 14.4k modem – and later over a local ARCNet network, cabled through my parents house with chained 25′ phone extension cords from the dollar store. When I left for college in January of 1995, she came along as my primary college computer, the task for which my father had intended her. At college she continued to support the BBS, running Renegade and then Iniquity software, but the BBS was often suspended to allow me to spend long nights mastering Autodesk 3D Studio and Photoshop. At some point I also discovered Linux, and when a friend began handing me surplus computer hardware from his work, I was able to add several more hard drives and increase her RAM to a whopping 40 megabytes. I would give a lot for photos of my workstations at this time, but as far as I know none exist.

the login screen for iNiQUiTY BBS
the SysOp screen for iNiQUiTY BBS

As college came to a fruitless end – a diploma, but zero job prospects – I took a job as a graphic designer for a college web project. Schaltwerk was the main graphics workstation, putting in months of midnight-until-dawn marathon Photoshop sessions. I was also working hard on my own Linux interface design projects, working closely with the Enlightenment window manager team.

In about March of 1997 I moved back to Sussex and took a government-funded web design position. I was offered a Pentium workstation of my own, but after struggling with drivers and software installs and an unstable machine, I moved Schaltwerk into the office to be my primary workstation. This didn’t last too long, as I couldn’t deal with the lack of computer at home! Schaltwerk, nestled in her basement lair in my parents’ house, had sprouted two more monitors – a monochrome display addressed with a second video card and a Wyse 60 dumb terminal attached to the serial port now accompanied the main SVGA monitor. One mouse, two keyboards and three displays – pretty fancy stuff for 1997!

a screenshot of an Enlightenment theme designed on Schaltwerk
the first Enlightenment theme designed on Schaltwerk

In 1998, my friend Darren, my baby sister Jen and I packed all of our worldly possessions into Darren’s car and drove across the country from New Brunswick to Calgary, Alberta. We only had about $800 between us, but with one minor speedbump we managed to get settled and employed and much to our parents’ collective surprise, we made a go of it. After we all landed jobs at a major ISP, Schaltwerk became a networking powerhouse, having half a class-C subnet (128 addresses) of real internet IPs delegated to her for several months! Of course, at the time I really didn’t know what to do with that kind of resource, so I occupied myself learning Linux networking and DNS, and Schaltwerk got her first live, static-IP instances of BIND, Sendmail and Apache. Thankfully by this time I had gotten over the debilitating CircleMUD addiction I picked up in college!

In early 1999, my new girlfriend and I moved into a new house with our friends Ivan and Andy, who were running what was at the time the most technologically advanced Shoutcast station in the world, BeNOW. I became their network administrator, and together we whipped eleven machines and hundreds of gigabytes of storage (a big deal at the time) into shape. Schaltwerk took over as the router and firewall, also handling DNS and mail services for the BeNOW and riotnrrd domains, as well as primary and secondary DNS for dozens of other domains.

Jonnay's desk, Schaltwerk's home for years
Jonnay's desk, Schaltwerk's home for years

In late 1999, I had a job offer in Vancouver, so we packed our things and moved into a geek house in East Van. Schaltwerk stayed in Calgary and went to live with my friends Jonnay and Shell, where she spent the next few years humming away under Jonnay’s desk. During this time, she remained on a static IP address, becoming the primary DNS service for scores of domains, handling primary and backup mail services for dozens of others – but most importantly, she became the webserver for a number of domains. The most popular by far of any of the websites hosted on Schaltwerk was eastvan.bc.ca – a Slashcode site boasting ‘News For Crackheads – Nothing That Matters‘ which quickly gained notoriety as a gathering place for Vancouver’s dot-com underbelly. Most of the people I call close friends can trace their roots in our friend group back to eastvan.bc.ca. During this time, Schaltwerk also hosted the Black Hole Club email list, gathering a sizeable portion of Vancouver’s electronic music production scene together online.

In 2001, we moved to Costa Rica, leaving Schaltwerk with Jonnay and Shell – to her credit, Schaltwerk worked almost completely without interference from her hosts, only requiring several reboots and a hard drive replacement over the four or five years that she spent in their home office. In 2003 we moved back to Vancouver, moving into a house on 10th Avenue.  Schaltwerk became part of a cluster of media and internet servers driving the geek house, which we dubbed ‘Pod6’, a reference the Adult Swim cartoon ‘Sealab 2021’.  For a while Schaltwerk ran the website, but soon the site outgrew the humble 486 and we built an upgraded machine, relegating Schaltwerk to just email and DNS.

the Pod6 network
the Pod6 network operations center, Schaltwerk top left

In 2005 we purchased our first home and Schaltwerk took her new position – alongside a server from Jonnay and Shell, in reciprocation for their years of hosting – in a basement closet.  Too slow now to support much in the way of modern web services but still providing email and DNS services for dozens of domains, Schaltwerk also provided SSH shell endpoint access, allowing my friends and I to casually tunnel through even the fiercest of corporate firewalls.

In 2007 I began the long process of migrating all of the services off of Schaltwerk and onto a third-party host, Dreamhost. Many domain owners had to be notified, many small webpages had to be migrated and dozens of cryptic user scripts had to be decoded and disabled or ported. By 2008 almost all of this work was complete, and Schaltwerk remained online but rarely used until April of 2009, when I moved out of my basement and aboard the S/V TIE Fighter. I could not bring myself to just throw away a machine with such a history of faithful service, and so I brought her aboard, intending to find a way to celebrate her life.

goodbye, Schaltwerk.
goodbye, Schaltwerk. you will be remembered.

Schaltwerk gave me fifteen solid years of faithful work, far more than can be reasonably expected of a PC.  Her only fault was her lack of processing horsepower, and while I will admit that I entertained fantasies of one day putting her back in service as a terminal somewhere, life on a sailboat is not kind to electronics, and a slow death in a storage locker just wouldn’t suit her. With a few respectful words about her life and service I sent her to her final resting place in the ocean, about a kilometer off the Sunshine Coast.

I have to admit it took a few minutes for the lump in my throat to pass.

Ok, Really Screw It!

Unshaved, unshowered and happy
OMG TEH BLOG CAN HAS PHOTOZ NOW!!!111!eleven

OK, FINE!  I WILL SPEND THE HOURS AND HOURS TO MANUALLY MIGRATE ALL MY OLD CRAP OVER!

There.  Are you happy?

Actually, I am.  Wordpress is waaaay nicer than Drupal, at least for a blog.  Using Drupal for a blog was kind of like driving a Jeep – I mean, sure, it’s rugged and capable and even beautiful, but it uses a lot of gas, it’s noisy, and while the ragtop is nice in the Summer it’s just impractical in the winter.  Really the only thing that prevented me from moving earlier was the complete lack of Drupal-to-Wordpress migration scripts.  Yes, I did in fact have to migrate each and every post by hand.

Oh well – at least now it’s done, and I have a blog I can be proud of again.  Welcome to 2001, Drew – the internet now supports fancy things like “photos” and “videos” and “multimedia”!  Good thing I managed to lose my camera battery charger in the move.  Nice going, Drew.  Seriously though – in order for Drupal to have the simple feature “add image to the blog”, you had to resize the image manually, upload it to the FTP site manually, and type in the full path to the image.  Now *that* is some serious 2001 stuff right there.

The downside of this whole migration fiasco is that I’ve basically had a mental block against doing any sort of blog updates ever since I committed to the jump.  That means that it’s been almost three.  frackin’.  months.  since my last update – and honestly, this has been one of the most action-packed, adventurous, utterly insanely awesome Summers of my life.  I don’t even know where to start.  I guess it’ll be roughly at where I left off, back in June…

It’s The Future!

Some days, you just have to shake your head.

I just got off the phone with a colocation facility in Houston, Texas, trying to figure out why our servers are dropping like flies today. Four down so far, and signs of trouble on a few others, almost certainly foul play – I’m trying to keep in mind the old sysadmin truism ‘Never attribute to malice that which can be adequately explained by stupidity’. Still, my lip curls a little at the thought of some fourteen-year-old kid in his mom’s basement in Iowa sending his massive botnet up against our servers in response to some unrealized slight. Or maybe he’s earning money somehow, or even just the admiration of his peers. Or maybe he’s just being a dick. *shrug*.

I’d say “he or she”, but I have yet to meet a girl with both the skills to launch a DDoS attack and the sociopathic tendencies to actually do it. I’m sure she’s out there somewhere.

Just to juxtapose, by “phone” I actually mean “Skype from my tiny laptop, sitting in the sun on the roof of my sailboat”. Seriously, had you told me five years ago that this would all be possible, I don’t think I would have laughed outright, but I would probably have smirked. All of these little incremental upgrades in technology and lifestyle have slowly and quietly added up to the WORLD OF TEH FUTUR3!!@1 that we were promised as kids. It’s finally here!

…now where’s my rocket car?

Anyhow. I was actually only using Skype because my cellphone batteries are currently dead, and I left the charger in New Brunswick the last time I was there. Since it’s a basic no-frills Motorola flip I can charge it up with a regular mini-USB cable I had lying around, but unfortunately my MacBook Air only has a single USB port. I get to choose – would I rather have my cellphone charged up, or would I rather be on the internet? Internet, I choose you.

Also in the realm of electricity, my diesel engine currently isn’t starting. I turn the battery selector to the starter battery bank, I pull the power toggle, flip on the lights and test the alarm to verify that the panel is receiving power, hit the starter button aaaaaand… nothing. Nada. Not a grumble, not a click of the solenoid, nothing at all. I’m hoping it’s just a wiring problem, as the wiring is a real mess and I may well have accidentally disconnected a wire when I was cleaning up the other day, but frankly I can’t tell.

To add to that problem, I have no water. I’m drinking (and cooking, and cleaning) from a 10l jug currently, because my water tanks are empty. Filling the tanks requires motoring up to the nearest hose, and, as I said, my engine isn’t starting. *sigh*.

Man. I think it’s time to convert this blog over to WordPress. I’ve been using Drupal for the past year, and in a solid year I have yet to find a reasonable way to upload photos. For instance, I’m sitting less than four feet from the exposed wiring rats nest that is my starter panel, and my laptop has a camera – but for me to take a photo of that and post it to the blog involves using Skitch to take and resize the photo, uploading the photo to the webserver, and referencing the URL using full HTML tags. I can do that, but I don’t wantto. With WordPress, I can do all that in one step, and I think that’ll make a huge difference for me.

Anyhow – an email just came in requesting an RSVP for the Open Web Vancouver speakers’ pre-conference social next Wednesday night – and I’m realizing that I’m just over a week away from speaking to 400-odd geeks about work that I’m currently procrastinating against. Soooo… I’m going to cut this abruptly and get back to work.

Morning

Well, apart from the bitter cold (mitigated by a merino sweater) and strongly gusting winds, it’s a beautiful morning in False Creek.

Today is the first morning of cooking breakfast on the Coleman propane burner, and hence the first morning for oatmeal with cinnamon, raisins and craisins! Delicious. About time I can actually start making use of the stores of dried food in the pantry.

This morning when I was woken by the sun at 6:50am, I hung a dish towel over the window, which allowed me to go back to sleep for another two hours. Sweet – even the crow landing on the deck and caw-caw-cawing at me for twenty minutes didn’t phase me, and I slept. Good thing, too, I needed the extra few minutes after staying out until 1am watching the new ‘Star Trek’ movie with JT. For the record: movie 7.5/10, theatre-going experience 1/10. $12.50 tickets, $4.50 fountain pop and more than a frackin’ hour of commercials and previews prior to the movie! God. And they wonder why movie piracy is so rampant!

Today I have to launch a bunch of webserver instances in the cloud and start stress-testing the website with ‘siege’. I suspect it’s going to hold up just fine, but we’ll see. I really wish there were some way to pull information about the cloud cluster *from* the cloud, via CSV file or something, instead of having to remember long strings of identifier numbers, IP addresses and volume IDs.

Today I also have to go back to WestMarine to buy wire, and possibly a pair of rubber boots. It has become uncomfortably obvious that I do not own a single pair of waterproof footwear! I mean, seriously, I have three pairs of rainpants, four waterproof jackets, two pairs of neoprene gloves… and no shoes. Given the rain of the past two weeks, this has basically meant constantly damp feet. I’ve been trying to get a pair of boots, but everywhere I try they seem to always be sold out of my size.

Oh! One very interesting piece of news – I spoke with Rogers Wireless tech support the other day, regarding my RocketStick cellular modem thing. Specifically, I wanted to know what the charges would be if I were to go waaaaaaaay over my allotted bandwidth for the month. Currently I have a “scaling” plan, which gives me 500 megabytes for $30/month – should I go over 500 megabytes, it changes my plan to a gigabyte for $35/month. Should I go over that, it changes to 1.5G for $40/month, then 2G for $45/month, and so on until $85 for 5G.

So that’s where I was worried – currently I am using anywhere from 100 megabytes to a gigabyte per day, just in regular internet traffic, mostly from work stuff, and I would really rather not be stuck with some kind of $2000 cellphone bill. I called to ask if there were a bigger plan I could get on. They said no.

So I asked what would happen if I went over…

“Well, sir, we then bill you per-kilobyte.”

“I see. How much is it per kilobyte?”

“Ummm – actually, I don’t really know. I know we cap the bill at $100 though.”

“Pardon me?”

“We cap your bill at $100.”

“Soooooo… $100 is unlimited internet?”

“We… cap your bill at $100.”

“Ok, so $100 is unlimited internet, but you’re specifically not allowed to use those words.”

“That is correct, sir.”

Sweet. So $100/month for unlimited wireless internet on the boat. That smells like a tax-writeoff work expense to me.

Soon I will have to leave the relative safety of False Creek and head for unknown waters. My current plan is to head for Victoria and anchor in either Cadboro Bay or Esquimalt Harbour for a couple of weeks, then perhaps head north towards Nanaimo before coming back to Vancouver for another two weeks. I’m trying to keep my plans somewhat open, but I am starting to feel the itch to move.

CloudCamp Aftermath

Wow, that was great! It’s so nice to finally be interested in a technology again – I was honestly starting to think that I’d never enjoy another conference. Half of the fun of a conference is discussing new concepts and ideas in technology, but the other half is meeting up with folks who have similar ideas and interests, both on a professional level but also in a social (ie: beer) environment.

It was also my first “un-conference”. The idea behind this is that there’s a schedule, but none of the talks are booked in advance, and people come with a talk or presentation that they’d like to give and sign up to do so on the spot. In the end it’s more about discussions than presentations, and in fact I was drafted (or more accurately, “tricked”) into hosting a discussion group on scaling Drupal in the cloud.

Now *that* was an eye-opener – I had naïvely assumed that most people working with cloud computing were working with web applications, as that’s what most of the documentation out there seems to be. Or maybe I just pitched my session wrong, and should have stressed all web applications and not just Drupal. Or perhaps I shouldn’t have scheduled the discussion at the same time as Dan Kaminski’s discussion on cloud security. Regardless, the room began with approximately eight people, and as it became apparent that we were really looking for a technical discussion but most of the people were in marketing and management, the room shrank to four people.

What was awesome was that the four people were:

  • the director of business development from RightScale,
  • the CEO of Work Habit,
  • a senior technical project manager from Amazon Web Services, and
  • myself.
  • Wild! I got more out of that half-hour session than I have reading documentation for the past month and a half. As the session started, we basically went around the room discussing backgrounds and technical histories, and it became apparent that I was really the only person in the room with any technical experience at all working with the EC2 cloud, I began to present my experiences to date, focusing on the three big problems that I had butted my head against – how to “auto-scale” web front ends using load balancers, how to scale MySQL databases elastically, and how to share storage between EC2 instances. As I discussed these, Dean Dierickx from RightScale showed up, and had some interesting notes to add from a high-level design perspective, which we talked about at length. We then started discussing database scaling, and it became apparent that nobody in the room had any practical experience with that at all.

    Right about then, Jonathan Lambert from Work Habit let himself in, and after listening to us fumble about for a couple of minutes, grabbed a whiteboard marker and launched into a fifteen minute in-depth technical discussion on different methods of scaling MySQL. This was *great* stuff, though it pretty much cleared the room of the marketing and manager types. As he finished, he validated my whole trip with one statement – he said something like:

    “Now, there are three major problems that everyone attempting to scale any web application in the cloud butts their heads against, and there’s no simple answer to any of these: how to launch instances automatically from a load balancer, how to scale MySQL, and how to share storage between instances. We’ve fought these problems hard for two years now, and we still don’t have a good answer, but here’s how we’ve managed to get everything working so far…”

    …and proceeded to lay out a basic, scalable platform for Drupal on the whiteboard. Most of his layout matched my work exactly!

    To top that off, just last week I downloaded a script from the RightScale website that purports to set up a basic EC2 instance to be a “RightScale” machine, and poked through the script to see exactly what they’re doing to prep a machine. I was shocked to see that every step of the way, their work matched what I was already doing to setup my own EC2 instances! That may not sound like much, but frankly after fifteen years of Linux administration, I have some particular ideas about how a machine should be setup for optimal networked administration; which additional packages should be installed, which default services should be turned on or off, what changes should be made to the default shell environment, etc. Seeing two large, successful operations doing exactly the same things as I do in my personal environments, and facing exactly the same challenges, was vindication to say the least!

    Anyhow – even though it was a “free” conference (with the quotes of course implying bus fares and hotel fees), I think I got well more than my money’s worth.