Cloud Computing – disengage.ca

Countdowns All Around

It’s Thursday, and I’ve got three major, looming deadlines staring me in the face. I think I’ve got a handle on all of them, but it’s definitely not a relaxing time in my life right now.

<Geek>

On the work front, I’ve migrated two very large web properties into the Amazon Elastic Computing Cloud over the past eight months or so. I’ve been learning the ins and outs of the new technology as I go, and the playing field really has changed. There have been a tonne of little headaches and bugs and glitches, and I’ve been pulled out of bed at 5am more often than I care to admit. And now – just when we’re finally stable – a directive has come down from the Evil Masters to port both sites to a common backend using the latest new Drupal code. On one hand this will open a lot of doors for us, allowing us to scale a lot quicker and use some of the more modern tools, like storing all images on a Content Data Network (CDN) instead of on our current frontend webservers. Still, the deadline for launch is February 12th and that’s coming up faaaaaaaaast. I’ve barely got the preliminary test servers in place!

</Geek>

On the boat front the engine work continues, though the work seems to multiply every time I put time into it. I’m finding a great deal of satisfaction in it, actually – I mentioned to a friend yesterday that it is very much like ‘The Legend of Zelda’, in that the puzzles are difficult, but once solved there is immediate positive feedback (ie the engine works better) and you can move on to the next puzzle, often using knowledge or tools you gained from the previous level.

Yesterday’s miniboss was changing the zincs in the engine. Sacrificial zincs are bits of… well, zinc. The theory is that if you bind several types of metals together in a marine environment, the weakest metal will corrode. Because of some kind of galvanic voodoo, the other metals will not corrode until the weaker metal is completely corroded away. Zinc is a very weak metal, easy to work with and cheap, and so quite a few different parts of the boat have sacrificial lumps of zinc attached to prevent the more important bits from corroding.

The zincs in the engine should be replaced about once a year, more or less depending on use. My engine has three zincs – at $7.00 per zinc, it’s a $21.00 job to replace them all, but compared with approximately $10,000 for a new engine, the price is negligible. The zincs are attached to the end of thick bolts and screwed deep into the heart of the engine.

One of the three zincs is located right on the front of the engine, easily accessed. The other two zincs are located far down the right side, between the engine block and the wall. Once I stepped back and surveyed the engine, I found that I could just barely get a socket wrench in a gap, which allowed me to remove zinc #2 with little difficulty – but zinc #3 was a real hassle. To get at the third zinc I had to remove the fuel lift pump (skills and items gained from previous level!) and the exhaust manifold – and even then the bolt holding the zinc into the engine was seized pretty solidly. I ended up having to extend the socket handle and actually step on it to get the bolt free; never send a hand to do a boot’s job. I swear I heard victory music when that bolt finally gave way.

So far in the past month I have rerouted the fuel lines, replaced the fuel filter, installed and plumbed a second fuel filter, replaced the damaged exhaust water trap ($300, ouch), replaced the impeller in the raw water pump, and replaced the zincs. Remaining, I have to have the alternator tested and serviced, pick up new oil, drain and change the current oil, drain and change the transmission oil, take the heat exchangers to the radiator shop to have them boiled out, pick up antifreeze, install the secondary cooling pump, drain the engine cooling system and replace with antifreeze, reroute the raw water intake through the heat exchangers, rewire the instrument panel, and then get the fuel tank polished. Whew! Someday soon, I will have an engine that runs reliably; ideally one that I do not have to climb into the engine compartment with a screwdriver to start. There’s almost no chance I’ll have all this done by Monday, so I really have to pick and choose what tasks are actually important.

…and then I get to start on the electrical system! For some reason, since returning from Vegas the house batteries aren’t holding a charge anymore. I have no idea why; I need to replace the batteries and purchase and install a modern charge manager. I don’t expect to get that one sorted out for under $1000.

Lastly, I have Sequential Circus coming up on Saturday. This is a huge show, with six live-pa acts performing 45-minute sets at a local show venue slash warehouse space. Everything is coming together smoothly, mostly because it’s our sixth time running this show and we’re all getting really good at it. It’s really starting to look like we’re going to have a solid crowd too, which takes a lot of the financial stress off of my back – if everything works out well, I might just come out of it a hundred bucks richer!

I still haven’t figured out where to go on Monday, and the False Creek / Olympic Village security lockdown continues… more on that soon.

Long Overdue Update!

Wow. Three of the craziest, busiest, happiest months of my life. How to compress them into one post? WHY compress them into one post? This seems silly, but I think the best way to re-jumpstart my blogging is to get this all out of the way in one post, and then go back to more regular updates. *sigh*.

At my last major post, I was about to speak at the Open Web Vancouver conference at the Vancouver Conference Center. My talk went pretty well, I guess – I mean, I definitely didn’t win any awards, but nobody walked out either. I met some great new folks and had a good experience overall. I know now that speaking at tech conferences is almost exactly like doing live-pa techno in front of a big audience – the more prepared you are, the easier it is to let go and just be yourself.

Since then, there’s been… God. Seriously, where to start?!

I’ve had repeated, profound musical experiences on the boat, jamming with friends. Picture if you will a mirror-smooth False Creek, with the boat anchored about fifty feet offshore. Dan Ross playing guitar and singing, Chad Taylor playing muted trumpet and providing some percussive backup and myself on mandolin and backup vocals – folks walking past, double-taking and sitting down on the seawall to listen, applauding between songs. Making music on the boat with friends has given me far more joy than I ever imagined it could. Actually, making music on the boat at all – I’ve been spending on average about eight to ten hours per week sitting on my deck, playing my guitar and singing. If there is a greater peace than playing music on the water, I haven’t found it yet.

I’ve gone on three epic sailing adventures, the third of which is still ongoing – as of this writing I am anchored in this lovely little bay, surrounded by million-dollar waterfront houses and a beautiful cliff infested with rock climbers. More on that in future posts – but suffice to say this ongoing solo-sailing adventure is not without its trials and tribulations.

The first of the three epic sailing adventures was with a beautiful woman named Miya who I met at Burning Man in 2008, and who had come to visit me several times over the past year. Her confidence in my sailing ability was appreciated, though perhaps unwarranted, as we left Vancouver and immediately ran into eight-foot breaking swells just off Point Atkinson, enroute to the Sunshine Coast. The sailing got a lot better after the first day, but we still had to spend a few days on Bowen Island with engine trouble – mostly waiting around for a mechanic, until we tackled the problem head-on with the manual and some elbow grease, finally solving it ourselves and getting the engine back up and running. We then cruised up the coast to Secret Cove and Smuggler Cove, where we spent a night before returning to Vancouver. It was an amazing trip; the ocean opened my eyes and put a good fear into me, and the company was exquisite. The parting of ways at the end was wistful to say the least.

Drew and Laurel spinning fire on Tie Fighter — Drew and Laurel spinning fire on the boat at Diversity

The second sailing adventure was with yet another beautiful woman, Carrie, who joined me on a trip to the Diversity Festival on Texada Island. Technically we were supposed to sail with a crew of six, but Vancouver being the city of flailers that it is, the crew slowly called in to cancel until it was just the two of us. The winds were against us the whole way there and back, forcing us to motor around 90% of the tip, so it’s debatable whether or not we actually saved any money travelling by “sailboat”. We did get the sails up once or twice, but not nearly as much as I would have liked. The festival itself was excellent, with us arriving in full pirate regalia to great fanfare, spending a weekend surrounded by beautiful people and great music, and rolling out again on Monday with a grand exit. Sunday was a bit crazy, as the wind suddenly went from 5kn up to 25-30kn, and Tie Fighter danced in four-foot swells for the night – I now have a lot more faith in my anchor than before. Another boat nearby actually did slip their anchor, and came within a few feet of hitting us, but we held steady and Monday was much calmer. Another thing learned: rowing a dinghy in calm waters is one thing, rowing through four-foot waves as they break on the beach is another thing entirely! I made very good use of the drybags my sister gave me for my birthday.

The next weekend after Diversity was the Emrg-N-See Festival just outside of Salem, Oregon. I went to this festival with Trent last year, and it was probably the best festival I’d been to to date – it was as though someone had sent a personal invitation to every single gorgeous, blonde, dreadlocked, dubstep-loving yoga instructor on the west coast. I cannot express how many times I had to stop and shake my head at the sheer beauty surrounding me. This year was similar, though somewhat diluted, as though every guy who went last year went home and explained the situation to every guy he knew. I know I did, which is why I was surprised that the crew going down fron Vancouver was much smaller this year. Regardless, I definitely got my fill of amazing dubstep and bassline music, on very excellent soundsystems. I also got to take a tablespoon of dancefloor dirt out of my nose every morning, which I am choosing to look at as preparation for this year’s Burning Man expedition.

The weekend after Emrg-N-See was Sequential Circus 5, an electronic music event that I guess I’m sort of in charge of. I say that with some reservation, because the show couldn’t happen without every one of the seriously talented and driven people involved – we’ve got the whole thing pretty much down to a science now, and even with six live acts on a small stage, we continue to be efficient and competent, and we still have a good time doing it. This SeqCirc was probably the best music to date, though we were up against some very stiff competition. The capacity of the venue is about 180 people, and we had about 100 people, so while it was never packed, it never felt empty, and nearly everyone who was there at midnight was still there at 3am when we turned the lights on, so I count that as a win. The next Sequential Circus, SeqCircSix, will be in January.

After recovering from SeqCirc, having a few sailing missions out and around English Bay, and basically settling down and focusing on dayjob work for a while, I took off on my first big solo-sailing trip, headed for Victoria…

Fake Monday

Today is Tuesday, but it’s technically the start of the week since yesterday was Victoria Day, a statutory holiday here in Canada.

The holiday long weekend was gorgeous. I technically took Friday off of work also, so it was a four-day weekend full of sunny boat repair work. I got so much done! Where to begin…

Thursday night I went out to the Anza Club to catch a show – Tarran the Tailor was playing upstairs. Excellent time, great music as usual – afterwards I biked back to my rowboat, but as I approached I noticed a couple of bicycles abandoned at the top of the dock ramp. As I arrived, I found three 20-something folks sitting in my boat, drinking bourbon – they mistook me for a fellow Midnight Mass rider and invited me to join them. As I explained that I was actually there to kick them out of my boat and go home, they were shocked and extremely apologetic, but really, it’s not like they were hurting anything. The boat is always locked up with a padlock, and there’s nothing left in it to steal. I mean, if they’d thrown my oars in the water or tried to damage the boat in some way, it would have gone much more sourly – but as it was, they were nice enough folks, geeks even. We exchanged names and URLs, so Adam, Andrew and Rebecca, if you’re reading this, feel free to drop me a line. 🙂

Friday and Saturday I got up at 8am and worked hard on the boat – I finished a bunch of epoxy work, got hinges onto all the storage hatches finally, and made progress in getting the hinges and hasps onto the cabin hatches – that project still needs more work, of course, but the end is finally in sight. I spent a bunch of time in the engine compartment working to get the kill switch in place, and finally succeeded – but when I went to test it, it didn’t work, and in fact I’ve apparently damaged the cable to the point that I need to go and find a new one. Boo – at least now, after two visits to the marine store and one visit to Canadian Tire, I know that the cable is called a ‘utility cable’ and that I should be able to get a new one from LloydCo Auto Parts.

I also removed the traveler on Friday – ie the seven-foot-long pulleys-on-rails thing that the boom attaches to – so that I could fix a few leaks in the bedding hardware. The leaks were directly over the stove, which meant that every time it rained I’d have to use steel wool on the cast iron stove grill again to get rid of the big patch of resulting rust. The leak had, over time, caused some of the roof to rot; this led to the first cutting of a large hole in the boat roof, and the bulk of the 1/4″ of sawdust that covers everything in the galley at the moment. The hole is patched, the surface is fiberglassed, sanded, faired perfectly with epoxy and fairing compound, and the traveler is now ready to be rebedded – perhaps this afternoon, if the weather clears up for a while.

Sunday I had a few guests over helping me work on the boat – it seems unfortunately that adding more people to a project doesn’t necessarily make the project go any quicker. Still, it was nice to have the company, and a few projects got nailed down properly – though when I removed the trampolines to fix a few small cosmetic problems on the center bow of the boat, we discovered a few patches of rot that quickly grew into a huge seven-foot hole in the boat. The rot wasn’t structural, which was a relief, but all that wood still needed to be replaced. I got a bunch of the wood in, but then Sunday was mostly rainy, so I had to cover the work site with tarps and pray for the best, spending the day curled up, drinking rum and watching movies with a friend. Sunday night was more rain and a lot more wind, which picked up the tarps and blew a cold wind through the boat, though as far as I can tell not much rain got in. It’s supposed to be rainy today and tomorrow, but then it’s supposed to be calm and sunny for another five or so days in a row, so this coming weekend I should be able to completely nail down the problems in the bow, and be done with it for the foreseeable future.

Today, however, I’m back to the day job. I’m working to figure out why the bottleneck in our EC2 migration appears to be network traffic – the frontend webservers seem to handle my load testing without a hiccup, but the database server spikes to a load of over 50, even though it’s an “extra large” EC2 instance. It doesn’t appear to be file I/O wait, nor a lack of CPU time, so I’m stuck. I’m not sure what I can do about that – I’ve always been under the assumption that network bandwidth between EC2 instances would be incredible, seeing as they’re virtual instances on more or less the same physical hardware. This week I have to solve the problem, but I’m not sure how just yet.

There’s still a few holes in the boat. I still don’t have clean water, though that’s just a matter of time – a reasonable amount of time actually, because filling the tanks takes a good fifteen minutes, then the bleach should be left in for an hour or so, then fifteen minutes to empty the tanks, then fifteen minutes to refill, fifteen minutes to empty, fifteen minutes to refill, fifteen minutes to empty, and finally a final refill. The traveler is still sitting a few feet away from where it should be mounted, and I still have more research to do on epoxy compounds before I can put the hatch doors properly back on the boat. The work is tiring, but very fulfilling, and a few long days of working in the sun have topped up my stores of vitamin D and left me with a positive outlook and a fantastic tan.

Constant Hustle

God. Seriously.

It’s Tuesday, and I have two more days to get everything out of my house and have it ready for the new folks to move in. There’s still so much to do!

This week has been very expensive – I thought last week was pricy, what with the purchase of the $1700 generator (which, by the way, seems to be somewhat overkill for my needs! I probably could have gotten away just fine with the 1000w version instead of the 2000w – oh well, I guess it doesn’t hurt to have too much power), but this week has had four >$100 trips to Canadian Tire and two >$100 trips to the boating store so far! Worse yet, there’s no end in sight, as I’ve still got to completely overhaul the electrical system, finish repairing all the fiberglass problems on the deck, and then hopefully I’ll have time to start making the interior look a little more like a home and less like a hunting camp.

One nice thing though – I realized yesterday as I was loading in some groceries that my nomadship (heh) is nearly complete. I’ve got canned and dried food to last a month, two months if I really had to stretch it. All the fuel tanks are full, and there’s another hundred liters of diesel in jerry cans in the amas. The water tanks are full too, giving me over a hundred liters of potable water – all I’m really lacking for an extended absence is a source of fresh protein (ie fish or meat). Not that I really *need* to be ready for any sort of extended absence… but given the econopocalypse, impending west coast earthquake, swine flu, etc, it’s nice to know.

I don’t know if I mentioned, but the second time we had the boat out, the binnacle (thingy that sticks out of the deck like a podium that the steering wheel attaches to) pulled out the deck, exposing poor workmanship – don’t attach important things with short woodscrews! – and a small patch of rot that I’ll eventually have to tend to. I got around to fixing that yesterday, pulling the binnacle completely off and re-attaching it securely to the deck with long bolts and wide washers. When I went to reconnect the steering, however, I noticed a bunch of slack in the lines. I called Bill, the guy who sold me the boat, and asked what he thought of that.

Well, turns out one of the steering lines had slipped from a turning block. That turning block happened to be deep in the stern of the boat, reachable through an access panel, but still at the end of arm’s reach. I got to spend the next hour and a half up to my shoulder in the wall of the salon, trying to free a thick steel cable from the pulley it had fouled. Waaaay fun.

Seriously, every day something breaks on the boat, and I have to learn how to fix it. Sooner or later I’m going to know every square inch of her. In some ways I growl about this, as I look at weekend sailors with their gorgeous, perfectly-functioning boats, but in a much stronger way I know that this is the universe’s way of ensuring that the boat becomes truly mine. It is a series of tests, and as I complete each one I feel stronger for having done so.

Anyhow. Just to add to the stress, I’ve been accepted to speak at Open Web Vancouver, a big web conference in June. I’ve never actually spoken at a conference before, so we’ll see how that works out…

CloudCamp Aftermath

Wow, that was great! It’s so nice to finally be interested in a technology again – I was honestly starting to think that I’d never enjoy another conference. Half of the fun of a conference is discussing new concepts and ideas in technology, but the other half is meeting up with folks who have similar ideas and interests, both on a professional level but also in a social (ie: beer) environment.

It was also my first “un-conference”. The idea behind this is that there’s a schedule, but none of the talks are booked in advance, and people come with a talk or presentation that they’d like to give and sign up to do so on the spot. In the end it’s more about discussions than presentations, and in fact I was drafted (or more accurately, “tricked”) into hosting a discussion group on scaling Drupal in the cloud.

Now *that* was an eye-opener – I had naïvely assumed that most people working with cloud computing were working with web applications, as that’s what most of the documentation out there seems to be. Or maybe I just pitched my session wrong, and should have stressed all web applications and not just Drupal. Or perhaps I shouldn’t have scheduled the discussion at the same time as Dan Kaminski’s discussion on cloud security. Regardless, the room began with approximately eight people, and as it became apparent that we were really looking for a technical discussion but most of the people were in marketing and management, the room shrank to four people.

What was awesome was that the four people were:

the director of business development from RightScale,

the CEO of Work Habit,

a senior technical project manager from Amazon Web Services, and

myself.

Wild! I got more out of that half-hour session than I have reading documentation for the past month and a half. As the session started, we basically went around the room discussing backgrounds and technical histories, and it became apparent that I was really the only person in the room with any technical experience at all working with the EC2 cloud, I began to present my experiences to date, focusing on the three big problems that I had butted my head against – how to “auto-scale” web front ends using load balancers, how to scale MySQL databases elastically, and how to share storage between EC2 instances. As I discussed these, Dean Dierickx from RightScale showed up, and had some interesting notes to add from a high-level design perspective, which we talked about at length. We then started discussing database scaling, and it became apparent that nobody in the room had any practical experience with that at all.

Right about then, Jonathan Lambert from Work Habit let himself in, and after listening to us fumble about for a couple of minutes, grabbed a whiteboard marker and launched into a fifteen minute in-depth technical discussion on different methods of scaling MySQL. This was *great* stuff, though it pretty much cleared the room of the marketing and manager types. As he finished, he validated my whole trip with one statement – he said something like:

“Now, there are three major problems that everyone attempting to scale any web application in the cloud butts their heads against, and there’s no simple answer to any of these: how to launch instances automatically from a load balancer, how to scale MySQL, and how to share storage between instances. We’ve fought these problems hard for two years now, and we still don’t have a good answer, but here’s how we’ve managed to get everything working so far…”

…and proceeded to lay out a basic, scalable platform for Drupal on the whiteboard. Most of his layout matched my work exactly!

To top that off, just last week I downloaded a script from the RightScale website that purports to set up a basic EC2 instance to be a “RightScale” machine, and poked through the script to see exactly what they’re doing to prep a machine. I was shocked to see that every step of the way, their work matched what I was already doing to setup my own EC2 instances! That may not sound like much, but frankly after fifteen years of Linux administration, I have some particular ideas about how a machine should be setup for optimal networked administration; which additional packages should be installed, which default services should be turned on or off, what changes should be made to the default shell environment, etc. Seeing two large, successful operations doing exactly the same things as I do in my personal environments, and facing exactly the same challenges, was vindication to say the least!

Anyhow – even though it was a “free” conference (with the quotes of course implying bus fares and hotel fees), I think I got well more than my money’s worth.