A Cloud by any Other Name…is Inconceivable!

Posted on 17 April, 2011 by Arras

“You keep using that word. I do not think it means what you think it means.”

– Inigo Montoya, The Princess Bride

Over the past few years the tech industry has been quietly conspiring to slip the word “cloud” into our cultural vocabulary, but the marketing departments behind this effort are having a hard time establishing any sort of consensus about what this magical new word should mean. With a word like “cloud”, though, a nebulous definition is somehow fitting, if unsatisfying.

Cisco took the novel approach of asking the general public what they thought cloud computing was, if only to make a point. Amid all this confusion, Google felt the need to explain what they mean when they use the term. None of this stopped Dell from trying to trademark the term.

Even the definitions on Wikipedia and Webopedia aren’t very satisfying, largely because they’ve been generalized to include all the vendor-specific interpretations of the term.

If you haven’t got a firm idea of what “cloud computing” is, rest assured you’re not alone–even Larry Ellison, the founder and CEO of Oracle was flummoxed by the term (BusinessInsider.com, 2008):

“The interesting thing about cloud computing is that we’ve redefined cloud computing to include everything that we already do. I can’t think of anything that isn’t cloud computing with all of these announcements. The computer industry is the only industry that is more fashion-driven than women’s fashion. Maybe I’m an idiot, but I have no idea what anyone is talking about. What is it? It’s complete gibberish. It’s insane. When is this idiocy going to stop?

“We’ll make cloud computing announcements. I’m not going to fight this thing. But I don’t understand what we would do differently in the light of cloud computing other than change the wording of some of our ads.”

Then there’s Microsoft CEO Steve Ballmer, who in 2008 was announcing his company’s new cloud computing project, codenamed “Red Dog” (now “Azure”). He declined to provide any details, possibly because he was still unclear on the concept himself (CNET, 2008):

“I would have thought I knew what the word ‘cloud computing’ meant, until I sat with Ann [Winblad] and a bunch of venture capitalists this morning who used the word completely differently than I would have used it.”

“I think when people talk about cloud computing they’re talking about taking some stuff, putting it outside the firewall, and perhaps putting it on servers that are also shared–or storage systems–that are also shared, perhaps with other companies that they know nothing about.”

By 2010, though, Ballmer wanted CEOs and CIOs to accept a bit of hand-waving and take his word for it that cloud computing was the next big thing:

“So, what is this cloud thing we talk about? In some senses I think that the best way to think about the cloud is it’s a place where we will all work, we in the tech industry and all of the IT people who work for companies like yours. We’ll work to fuse the best of what we think of as the PC today, the phone, the TV, the Internet, and the corporate data center. And it’s a land of opportunity.”

If that wasn’t vague enough for you, then consider how Ballmer adapted that message for the average person at his next event:

“I’m not sure my goal for today is going to be to actually explain [cloud computing] to you, but I do want to make sure that people understand that I think everybody in our industry accepts it’s the next major transition point in terms of how IT gets done.”

In other words, it’s all right that you don’t know what cloud computing is. All you need to know about cloud computing is that it’s better than sliced bread, the bee’s knees, and a basket full of puppies. If all of that annoying “To the Cloud” advertising primes you to run out and purchase anything with the word “cloud” on the label, the tech industry as a whole will be richer for it. Literally.

The beauty, from a marketing standpoint, is that without a formal, accepted definition, the word “cloud” can be applied to virtually any Internet application. That should come as no surprise, really: we’ve been using the cloud symbol on networking diagrams for ages to represent…the Internet.

“Pay no attention to that man behind the curtain.”

– The Wizard of Oz

We use cloud symbols to represent the vague, the unknowable, the uncountable, the hazy bits in the middle of a troublesome math proof in which we write, “a miracle occurs”, and the second-last step in a list of instructions that ends with “Profit!” It’s the intellectual spackle we use to cover holes in our logic, downplay our inability to grasp the complexity of the problem at hand, and sometimes to be deceitful. Has computing really reached such a point that we’ve no better option than to throw up our hands and accept it all as magic?

There’s an appeal to that line of thinking, the notion of submitting to the Technology Gods and letting them have their way with our data, trusting that they’re the experts, and that we’d be better off outsourcing more of our computing to professionals, the same way we trust utilities to handle essential services. The US government seems to agree, as their CIO, Vivek Kundra explained his vision of cloud computing (CIO.gov, 2010):

“There was a time when every household, town, farm or village had its own water well. Today, shared public utilities give us access to clean water by simply turning on the tap; cloud computing works in a similar fashion. Just like water from the tap in your kitchen, cloud computing services can be turned on or off quickly as needed. Like at the water company, there is a team of dedicated professionals making sure the service provided is safe, secure and available on a 24/7 basis. When the tap isn’t on, not only are you saving water, but you aren’t paying for resources you don’t currently need. ”

Whether that vision warms your heart or gives you a case of the heebie-jeebies likely depends on where you sit on the techie scale:

If you consider yourself a techie, or at least someone who knows a thing or two about computers, you probably shudder at the thought of giving up control of your computing environment; you prefer to host your own servers at home, perform your own backups, and you buy your computers based on their speed and power. For you, this is a hobby, a passion, a point of personal pride, but most of all it’s a domain that you feel confident you can manage responsibly. Why would you want to trust that to anyone else?

If you live at the other end of the scale, however, owning a computer can be an expensive source of frustration, and it comes with nightmares about viruses that will eat your computer, Trojans that will steal your passwords and empty your bank account, disk crashes that cause you to lose all your data, etc. The need to keep antimalware software updated, the need to perform regular backups and periodic maintenance all seem like chores, and like chores they get neglected. To people in this camp, the idea of a very simple computer (e.g. a thin client) that connects to an online service that hosts the actual applications, performs all the security checks and does all the backups is going to be very appealing.

Gmail is a perfect example. When it was first announced, Gmail made a lot of the techies I know shudder, mainly because of the privacy implications of letting Google process your mail, even if it was supposedly only to help Google deliver more targeted advertisements to you while you used their web-based client. Turning mail processing over to Google seemed like a surrendering of control, particularly since Google offered no information about its spam or malware filters–they could be doing anything to your mail behind the scenes, both before and after delivery to your mailbox. At the same time, millions of people appear to have decided that trading a bit of control and privacy for pre-filtered mail is a good deal.

Those sorts of privacy and control issues have almost by themselves come to define cloud computing as it exists today. The general model seems to involve outsourcing an application to remote servers somewhere, keeping as little as possible on the client machine. Rather than installing Microsoft Word on your PC, for example, you would simply connect to one of Microsoft’s servers and use Word remotely. With a fast enough Internet connection, the (buffered) back-and-forth exchange of data as you type should be unnoticeable, so ideally this would have the same look and feel as a locally-installed version of Word. It could end up costing you less, too, particularly if you only use Word occasionally, since Microsoft could charge some sort of metered fee or renewable subscription fee rather than force you to pay the full retail price.

Privacy and trust will always be issues with such services, though. When you’re composing a document with Word installed on your own computer, your words exist only on your computer, but when you use Word in its cloud-based incarnation those words are stored at a remote location, entrusted to Microsoft for protection. We can at least rule out the possibility of your data being intercepted by a third party somewhere on the Internet, since any well-designed cloud app would provide end-to-end encryption. The real risk is that if someone were to hack into Microsoft’s storage network, they could gain access to every document users have entrusted to them. Encrypting the stored files wouldn’t help very much in this case, since the decryption would need to take place on the server rather than the client, so the keys would be available to such an attacker as well, or could be sniffed as clients connect to perform the key exchange.

“Who will guard the guards themselves?”

– Juvenal, Satires VI

Ultimately it comes down to how much you trust the company providing the cloud-based service. Do you trust them to maintain a secure network and secure storage for your data? Do you trust them not to peek at your data, or use it for purposes you did not intend or authorize? Do you trust them not to hold your data hostage if/when you fail to pay the bill? Do you trust them not to sell your data to third parties? How easily will they hand over your data to the authorities if you’re under investigation? These are non-trivial concerns when your data lives in a storage facility beyond your control.

As the industry moves further toward a cloud computing paradigm, it seems that our personal computers will become more like netbooks–thin devices with little in the way of onboard storage or horsepower–while all of our applications and our data will live on remote servers somewhere, being tended by IT professionals. It’s a realization of the thin-client concept that failed to gain much traction in the 1990′s, only now applied to the Internet rather than a LAN. Make the client machines simple and inexpensive, and put as much as possible on the server side, just like the old mainframe model that featured one big computer serving many “dumb” terminals. That, my friends, is where “the cloud” will eventually lead us, if taken to its logical extreme.

Is that a good thing? Well, in a world with fast, ubiquitous Internet connectivity it would certainly have some appeal, particularly since it could put great computing power in the hands of anyone on Earth very cost-effectively. That One Laptop per Child project, for instance, wouldn’t need to cut corners, since thin-client netbooks would be the norm worldwide, likely available for well under their $100 target. There would probably still be room for manufacturers to distinguish their products–bigger screens, better speakers, more attractive form-factors, etc.–but for the most part we’d all end up using essentially the same client hardware under the hood. There’s little benefit, after all, to having a processor twice as fast as the next guy’s, when you’re both just connecting to the same cloud service to run apps on their hardware.

Of course “fast, ubiquitous Internet” is still a ways off at this point, and without it, cloud apps are going to feel somewhat sluggish. We might be willing to put up with that for things like mail and backups–things that aren’t so time-critical that we care about the seconds–but when milliseconds matter, things break down. Games that generate their graphics on the server and send the rendered frames over the Internet to a client won’t feel very responsive or reactive to user input until there’s enough bandwidth available to make this practical.

If we get to that point, however, it will mark the end of the personal computer era, a transition akin to scrapping all of our cars and adopting mass transit–an analogy that hints, perhaps, at how unlikely that outcome is. More likely we’ll see a compromise that makes cloud computing an attractive option for non-techies, while those of us who actually enjoy working with computers will become more of a niche segment of the market, serviced by vendors who still practice the art of turning silicon into gold.

For software techies, though, cloud computing has the potential to take their work to a whole new level. Imagine being able to write software that runs on a machine that never becomes obsolete, and whose computing power can grow and shrink dynamically as needed–from the equivalent of a basic desktop computer to a supercomputer at the flip of a switch. That’s Amazon’s Elastic Compute Cloud (EC2), and that sort of “virtual computer” may very well be the catalyst for tremendous software innovation in the coming years as developers begin to design truly scalable apps.

I’m not too concerned, personally; the vagueness of the terminology irks me, but in the end it might prove to be a useful label for abstracting a certain subset of the Internet, much as the “world-wide web” did in the mid-90′s. The web gave us a new way to conceptualize the Internet, which before that had been an informal collection of servers providing SMTP, POP, NNTP, FTP, Gopher, WAIS, finger, and so on. Visualizing the Internet as a web and providing a tool for browsing that web made a huge difference in terms of making the Internet accessible to the masses. I remember being annoyed at the time, too, that “the web” was being used casually as a synonym for the Internet–the purist in me always felt the need to point out that things like email were not part of the web, but it was a lost cause.

Perhaps the cloud is just the evolution of the web–a further abstraction of the Internet that may be necessary to describe the increasingly distributed and server-side nature of this new generation of Internet applications. If so, it won’t be long before we start hearing “the cloud” replace “the web” in casual conversation and in the media. And the purists in us all will cringe, just a little.

Posted in Computing | Tagged , | 1 Comment

Zen and the Art of Knowledge Maintenance, Part IV: No Bit Left Behind

Posted on 4 December, 2010 by Arras

Table of contents for Zen and the Art of Knowledge Maintenance

  1. Zen and the Art of Knowledge Maintenance, Part I: Ashes to Ashes, Zeroes to Zeroes
  2. Zen and the Art of Knowledge Maintenance, Part II: Ideas Worth Spreading
  3. Zen and the Art of Knowledge Maintenance, Part III: Our Digital DNA
  4. Zen and the Art of Knowledge Maintenance, Part IV: No Bit Left Behind

Throughout history there have always been individuals in key positions making decisions about what bits of knowledge survive and what gets left behind. They were librarians, archivists, teachers, writers, publishers, producers, and sometimes government officials. Sometimes the censorship has been deliberate, with history at times literally written by the victors. In other cases these decisions have been made to enforce an ideological, religious, or moral standard, in the hope of expunging from the human record any trace of That Which Offends Us. Personal and social biases have always played an important role, in other words, in deciding which ideas are brought forward into a new age and which ones are stamped out.

Inca Quipu

Cultures that relied on oral histories to pass on their knowledge from each generation to the next were the first to be lost from the human record. Our understanding of such cultures is vague and speculative, whereas we know much more about the cultures that developed writing systems. Even so, writing with a chisel or a stylus took time, training, and an expensively-prepared medium on which to commit the message, so the writers of the time clearly had to be deliberate about which ideas they carved in stone, engraved in clay, inked on parchment, or tied into knots. Official records like censuses were essential, along with things like property deeds, marriage records, and so forth, but there was little “frivolous” writing–that was a luxury of the idle rich for much of our history.

Gutenberg Bible

A similar dilemma followed the invention of the printing press, when those who owned one had the power to decide which handwritten works were deserving of typesetting and mass production. It took time, money, and labour to do so, and some materials were expected to be more popular than others–the Gutenberg Bible was the first to be mass-produced. More obscure titles with a smaller audience, though, would not sell enough copies to make it worth a publisher’s investment to print, and as such a lot of handwritten works never made the transition to print, and were lost forever.

We face a similar situation today with the shift to digital formats. There are vast collections of books, films, and audio recordings that will not survive the digital transition simply because they aren’t popular enough to be worth anyone’s time and effort to digitize. Not every movie that Hollywood produced made it to VHS, and many of the movies released on VHS never made it to DVD, and not every movie available on DVD will be remastered for Blu-Ray. Books published just a few years ago, like the one I mentioned earlier in this series, aren’t being offered in digital form. The economics of popularity determines in large part what survives to become digital and what gets left behind.

This economic effect is even shaping decisions about what new material gets produced, as publishers and studios try to avoid making costly investments in products that their market research tells them will be unprofitable. An author pitching a ground-breaking and completely original concept to a publisher these days is far less likely to get hired than one who proposes an unimaginative sequel to something that has already demonstrated its profitability. Such “risky” ideas are harder to sell these days, when book publishers and film studios are looking for “safe” bets. For every Inception, then, we see a dozen versions of Transformers.

InceptionTransformers 3: Dark Side of the Moon

Susan Boyle

We see the same thing in the world of music, where only “marketably attractive” musicians are selected by the record labels these days. In effect, we rarely hear the music of ugly people–a factor that really shouldn’t matter to our ears. The record publishers, though, want the reassurance that audiences will be drawn to an attractive face on an album cover, on posters, T-shirts, and in videos, all of which are marketing vectors. Susan Boyle was a stark example of the very opposite–certainly no fashion model, no teenager, not someone whose albums you’d select on the basis of her photo, and yet the beauty of her voice made it clear that she very much deserved to be heard. We should be wondering how many other singers and musicians we aren’t hearing, simply because they lack visual appeal. How many young music students are dissuaded from pursuing their careers despite enormous musical talent, because the music industry isn’t willing to bet on them for superficial reasons like age, gender, race, and appearance?

Twilight

Literature, too, suffers from risk aversion on the part of publishers, who simply look at a wealth of economic statistics about the market for their products, and decide to invest in stories that have the broadest appeal, leaving “niche markets” to smaller publishers with smaller print volumes. If writing at an 8th-grade reading level will double the size of the target market versus writing it at a 10th-grade level, publishers are motivated to solicit manuscripts to match. If reader intellect is assumed to follow a standard bell curve, there’s little profit in writing for the extremes–the prime demographic lies in the middle. If the current meme in that demographic happens to be stories about vampires, for example, then we’ll see a glut of books in that vein (pardon the pun).

What this economic factor does, unfortunately, is pre-filter our digital legacy. Rather than looking back and surveying a wealth of creative effort and selecting the most worthy items to preserve, we are instead deciding on the basis of projected profit to produce only the small fraction of works that will please the current market. Books and films are not produced with much thought about their worth to future generations, as we’ve come to assume that tastes and interests will change quickly enough that our children will want something new anyway–any story worth re-telling deserves a new remake, right?

Enrico Caruso

We do this to ourselves already, particularly when it comes to film and music. Very few of us bother to watch old black-and-white movies these days, in part because they are products of an era that predates our own, and partly because they look so awful on our brand-new 1080p wide-screen HDTVs. We miss the CGI effects that can make the impossible look real, and the old-style cinematography that was inherited from the days of stage plays can feel stifling and claustrophobic today. Music recordings from the first half of the 20th Century are typically monophonic, and while they may preserve the performance of a legendary singer like Enrico Caruso, a modern audiophile is more likely to focus on the sonic imperfections that can be heard more clearly than ever with today’s equipment. Few of us read any of the “classics” these days, either, outside of the requirements of a literature class, because it’s so much easier to parse prose that was written by contemporary authors who share our cultural background.

What we do to preserve these cultural legacies, though, is what we’ve done since the dawn of human speech–we become editors and select the best of the last generation’s ideas and bring them forward in forms that are relevant to the current one. This is the essence of the “remake”, the “remastering”, the “adaptation”.

Akira Kurosawa adapts Shakespeare’s King Lear in the context of feudal Japan and engages a new generation with his movie Ran…

King LearRan

…while The Taming of the Shrew becomes 10 Things I Hate About You for a modern American audience…

The Taming of the Shrew10 Things I Hate About You

…and an old black-and-white film like King Kong, which most contemporary audiences have heard of but few have actually seen, was remade for the current generation by Peter Jackson…

King Kong (1933)King Kong (2005)

Rachmaninoff Plays Rachmaninoff

…and modern sound engineers perform minor miracles to restore old recordings of celebrated musicians, including Zenph’s Re-Performance Series, which lets us hear what Sergei Rachmaninoff might sound like at the piano today, thanks to a bit of artificial intelligence.

Adaptation is the key, and it’s the lesson to take away from this series: information is not to be mistaken for its container–it may move from one container to another over time, and it may have to in order to survive. Whether it’s an oral history that gets transcribed to avoid cultural extinction, a stage play that gets adapted as a film to reach a larger audience that no longer goes to the theatre, an old film that gets remade with modern technologies, or a word processor document that gets updated to a format that’s more likely to be readable a few years down the road, our ideas need to remain fluid enough to be accessible to the next generation. Gasping in horror at a remake of an old film, a new cover of an old song, or a new visualization tool for old data is the wrong reaction–one that clings to the container without regard for the information within. Your novel is not a Microsoft Word document, it’s not a sequence of bits on a magnetic or optical storage device, it’s not a cluster of words in ink on a printed page, it’s an idea. If you want an idea to have the best chance of survival you need to be willing to adapt it to the dominant media of the time, planting that seed in the most fertile soil available, whether that means updating it to a more prevalent digital medium, or incorporating it into a form that people will want to preserve and pass on, long after you’re gone.

Our children and their children are growing up in a digital world, and if we want our ideas, our memories, and the benefit of our hard-won experiences to be accessible to them, the onus is on us to bring forward the evidence of our analog lives in a form they’ll understand. Scan those yellowing photos in that shoebox and they’ll have digital images to remember you by. Digitize those old cassettes, those reel-to-reel tapes, even those old 8-track tapes and 78-rpm records–chances are, some of that stuff never made it to CD. Where past generations have had to watch the physical evidence of their lives decay steadily, we have an opportunity to deliver to our children a preserved record that requires very little space and little maintenance to keep it from ever fading. Our great-great-great-great-grandchildren will be able to hear our voices, see our images, watch our videos, and understand us in ways we struggle to do with ancestors just a couple of generations before us. Let us pass the analog torch, and leave no bits behind.

Previous in series
Posted in Archiving, Music, Photography, Writing | Tagged , , , , , | Leave a comment

Zen and the Art of Knowledge Maintenance, Part III: Our Digital DNA

Posted on 4 December, 2010 by Arras

Table of contents for Zen and the Art of Knowledge Maintenance

  1. Zen and the Art of Knowledge Maintenance, Part I: Ashes to Ashes, Zeroes to Zeroes
  2. Zen and the Art of Knowledge Maintenance, Part II: Ideas Worth Spreading
  3. Zen and the Art of Knowledge Maintenance, Part III: Our Digital DNA
  4. Zen and the Art of Knowledge Maintenance, Part IV: No Bit Left Behind

As I mentioned earlier, one of the best ways to preserve an idea is to share it as widely as possible. This sort of broadcasting used to cost a lot of money, back when that meant printing materials to distribute or buying airtime on radio or television, but the Internet has given us a way to exchange ideas with people all over the world for a relative pittance. Alas, this is precisely the reason we have spam, since it costs virtually nothing to send an email, but with the bad comes the good: the Internet has become mankind’s de facto library, albeit with a few gated wings here and there.

This in turn is a major motivation for storing knowledge in digital formats, since this makes it possible to send them to others over a network. The fact that we can download books, photos, music, movies, television shows, software, and data of every imaginable sort is rapidly making the older delivery methods–printed materials, over-the-air broadcasting, CDs, DVDs, even Blu-Ray discs–obsolete. The video rental store as we know it is going away, and it’s hard to find a dedicated music store these days. Can it be long before bookstores face that fate? Between the Kindle, Nook, Kobo, and iBooks it looks like new publications are being offered in a digital format, downloadable from an online store.

There’s a space-saving advantage that can’t be overlooked. Words on pages, bound together in books, clustered on shelves in a library take up a lot of real estate, but you can store billions of them on a mobile phone. Any endeavour to archive the collected works of humankind has to take that sort of advantage very seriously–all the books and other text items in the world’s largest library, the US Library of Congress, would fit in just 20 terabytes of digital storage, which at today’s prices would set you back a mere $1,000 and fit in a box the size of a typical desktop computer.

It’s the ability to index and search a digital document that really appeals to me, though. I’m a techie, and many of the books on my shelves these days are reference books or manuals of one sort or another, and they’re quite often 600+ pages in length, causing my bookshelves to sag and groan under their weight. The table of contents and the index are sometimes helpful to find something I need to know, but truth be told, it’s faster and easier for me to craft a search with Google than it is for me to get up from my desk, find the book on the shelf behind me, consult the index and try to locate the reference I’m looking for. If I had that book in a digital format that I could search, I’d recycle the print copy in a heartbeat.

Most modern filesystems also support tagging, which lets you associate some metadata with any file of any type, not just text documents, to make it easier to search your filesystem for relevant files. Some file types incorporate their own searchable metadata, like information about the musician or album in an MP3 file, or the name of the TV episode in an AVI file, or the make and model of the camera that took the photo in a JPEG. Tools like the Mac’s built-in Spotlight index all of this metadata to turn your filesystem into a searchable database, rather than just a filing cabinet.

The more fundamental benefit that comes with putting knowledge into a digital form is that it tends to make it more “fluid”, making the content more malleable, and causing us to think about the content before the form. Consider that documents were at one time quite literally chiseled in stone or baked in fired clay, making them very difficult to edit. As paper offered a somewhat more disposable medium the writing process became a bit easier–botched pages could be replaced without scrapping the entire document. But it took the advent of the word processor to free us, ultimately, from the need to get it right the first time. The ability to write a rough draft, or flesh out an outline, or revisit a document later to insert or remove a bit of text–even correct a single typo–has given writers more creative control over their work.

The same sort of benefit can be seen in fields like photography, where the shift to digital image processing has given photographers a level of control over the finished product unlike anything that’s come before. It’s not just that the photos are stored in a digital format, it’s that consequently a photographer can see the result immediately on the camera’s display. Combine that with the fact that no film is wasted and you’ve got a risk-free environment for photographers to experiment with different settings of aperture, shutter speed, etc. Photographers these days also know they can edit their photos later, which removes some of the pressure to get the perfect shot (perfectionists aside ;) ). The so-called “digital darkroom” is to photographers what the word processor is to writers.

Having materials like photos in digital form lets you both preserve them against the ravages of time and repair some of the damage that may already have occurred. Those slides from the 1960’s that have been sitting in a box in the attic all this time have been gradually darkening as the chemicals break down, and some may even be completely black by now, but if there’s any chance of recovering the original image it will be with the help of a good scanner and capable photo editing software. Photographic prints aren’t much safer, particularly if they’ve been sitting in a box that wasn’t designed for storing photos–the acid in the paper of a typical cardboard box will be released over the years through outgassing, and any photos in that box will suffer for it; professional archival boxes use acid-free paper instead.

In the days before digital photography you had to live with your mistakes, as well. You were stuck with the composition you chose at the time you took the picture, and there was no way to crop the photo or remove that inconvenient thumb that somehow ended up in the frame. Then there were the times you’d take the film in for processing and get back a set of photos that just didn’t look as vivid as you remembered, possibly because the developer botched the job or used inferior paper, or because the film you bought was nearing its expiry date. Going through old photos today, I find myself doing a lot of that sort of correction, restoring blue skies and turquoise oceans that look dull and grey in the prints. I repaired an old baby photo that was badly torn in several places and creased with wrinkles–something that just wasn’t possible in the pre-digital world.

Photos aren’t the only things you can preserve this way, of course. Look at all the comic book collectors who diligently seal their comics in mylar bags to keep the pages from yellowing and fading. Scan a comic book and you preserve those pages for your digital collection. There’s a newspaper clipping my wife tacked up on her wall about five years ago–a “History of Christmas Traditions” article from the Vancouver Sun–that I’ve watched age over time in its unprotected state, and it drove home just how fragile things like newsprint really are. Scan them, though, and the digital images won’t age a day.

A look through the garage a little while ago turned up a bunch of old audio cassettes and VHS tapes, too, which will soon be at risk. It’s not just that the magnetic tape is drying up and becoming more brittle, it’s that it’s becoming harder to find equipment to play these tapes in the first place. I don’t care so much about tapes of material that I’ve long since replaced with versions on CD or DVD, but homemade recordings in particular deserve to live on. Think about those camcorder videos from the 90’s and their odd 8mm analog format–get those tapes digitized while your old camcorder still works, or else you’ll be hunting on eBay for something to play those tapes. Vinyl records are in a slightly better position because of the fact that they’re favoured by audiophiles, and a new generation likes them for their larger cover art. This ensures that there will still be turntables available to play them on for some time to come.

It’s the portability of digital media, though, that most people appreciate. The fact that a digital file can be copied from one device to another has been the bane of groups like the MPAA and RIAA, but it’s precisely what makes them popular with Average Joe–you can copy your music, your videos, your documents, and so on from your desktop to your laptop to your mobile device. For the purpose of perpetuating knowledge, the ready ability to copy data to many locations around the world is a tremendous reproductive advantage. The current controversy over WikiLeaks and its release of sensitive documents is an instructive example of how difficult it can be to quash the flow of information that can be so readily communicated in digital form. Indeed, protocols like BitTorrent were designed explicitly with the goal of decentralizing the distribution of data, so no single computer needs to store the data. In effect, such torrents form a network of offsite backups for the data they mirror.

Finally, there’s the integration benefit–the ability to create “mashups” that allow us to visualize or interact with information in new ways. We used to have photo albums for our family photos, record collections for our music, bookshelves for our reading materials and another set of shelves for our videos, but there was no easy way to mash these things up until we converted them all to digital formats. These days the notion of a “family album” is more than just an assortment of photos, it’s more like a digital scrapbook with comments as metadata for each picture. Home movies can be linked to these entries, along with music to establish the mood of a period. Modern genealogy software, for instance, lets you associate media files with people, places, and events, so that it effectively becomes the family album, a way to explore your roots in rich, sensory detail, held together in context.

The binary world of ones and zeroes has rapidly evolved to become the DNA of modern civilization, a universal language for communicating our thoughts, our ideas, our knowledge, and our experiences to others. We can argue about the best file format for video, or the best word processor to use for writing screenplays, but underneath it all we’re just quibbling over bits and bytes. The move to digital form is do or die for much of our knowledge, our history, and our culture. Up next: the stuff that won’t survive.

Previous in series
Next in series
Posted in Archiving, Genealogy, Music, Photography, Writing | Tagged , , , , | Leave a comment

Zen and the Art of Knowledge Maintenance, Part II: Ideas Worth Spreading

Posted on 4 December, 2010 by Arras

Table of contents for Zen and the Art of Knowledge Maintenance

  1. Zen and the Art of Knowledge Maintenance, Part I: Ashes to Ashes, Zeroes to Zeroes
  2. Zen and the Art of Knowledge Maintenance, Part II: Ideas Worth Spreading
  3. Zen and the Art of Knowledge Maintenance, Part III: Our Digital DNA
  4. Zen and the Art of Knowledge Maintenance, Part IV: No Bit Left Behind

It’s a simple but vital distinction: knowledge and its container are two different things. Preserving knowledge doesn’t necessarily mean keeping an old book on life support in a climate-controlled vault like a museum piece–that’s no good at all, in fact, if the contents of that book never get read. Shakespeare himself wouldn’t care about the rotting paper ‘pon which he jotted his verse, he’d want the words to live on. The works of William Shakespeare survive today in large part because they have been copied so widely and maintained from one generation to the next in whatever media that generation used to tell its stories. His plays, first delivered on the stage, were adapted for radio, and later for film and television. The fact that the works of Shakespeare are not encumbered by modern conventions about copyright has meant that any publisher can make money selling copies of his work without paying anyone any royalties, and as a result you can find the works of Shakespeare in virtually any bookstore and any public library.

Along the lines of Darwinian evolution, Shakespeare’s ideas have endured not because they were necessarily the best or the “fittest”, but because they had an excellent reproductive system. His works are “backed up” in the libraries of every major city on Earth, and in the collections of countless ordinary citizens. They are freely available in their entirety online, and used in classrooms throughout the English-speaking world to educate the next generation, ensuring their familiarity with at least some of his material.

To get to this point, though, Shakespeare’s work had to be valued for its own sake by those who managed the transitions from one medium to another. Publishers had to decide to print copies of his works. Schools had to decide to use his works in support of their curricula. Radio, film, and television producers had to decide to adapt his work to their non-literary media. Archivists had to decide to scan his works into digital form to make available online. If his works had not been sufficiently appealing at any of these points, they might not have survived that key transition.

That old Rivenhelm novel of mine might have survived, had I deemed it worth spending a few thousand dollars to have a data recovery firm do some forensic work on those old disks to read the extremely faint residual magnetic field. I could have taken the time, then, to print a hard copy of the book, or been diligent about maintaining the disks’ magnetic field, had I valued the material enough. It was my decision, ultimately, that caused that novel to perish, and behind every story and every piece of human knowledge that survives there’s a similar decision on someone’s part to keep it alive.

Make no mistake, in other words–knowledge survives only because we choose to maintain it. We have to choose the things we want to archive, and we can’t forget to maintain the archives themselves, because in addition to damage, loss, and deterioration from natural, physical causes we need to ensure that our archived materials can be read with contemporary tools. We can make our lives a bit simpler by choosing forward-thinking standards for the storage of information, so that we don’t have to update the formats too frequently, and by avoiding obscure, proprietary formats that are quickest to become obsolete.

Don’t forget about the reproductive system, either–ideas that stay tightly locked up with intellectual chastity belts will have a hard time changing the world. As a writer I’m keenly aware of the need to be compensated for my work, and I do see some value in the copyright system to that end, but let’s be honest: how many of you would be reading this if you had to pay me a few dollars to read past the subject line? At this stage of the game the onus is on me to convince you that what I have to say is worth hearing; if, down the road, I develop a big enough following, I might consider selling a bit of ad space (hey, I need to eat, after all), but for my ideas to survive they need to be valued by people other than myself. You need to choose to save these ideas, to share them with others, to be influenced by them yourselves, and it’s my job to convince you to do so.

I do this with my software development projects as well, incidentally. Maia Mailguard is open source, free for anyone to use, because I believe that some technologies should be available to everyone, and that if everyone had an effective tool for managing spam and virus filters the Internet as a whole would benefit. More filters out there would mean fewer machines becoming infected with viruses, fewer machines becoming unwitting participants in spam-spewing botnets, fewer people falling victim to phishing scams, and less frustration with unwanted mail in our inboxes. To parrot the TED motto, I believe this is an idea worth spreading, so it’s in my interest to remove the greatest barrier to adoption–cost, in this case.

Once we can separate form from content–grasp the fact that Hamlet is Hamlet whether it’s printed on vellum, on paper, or encoded as ASCII text, in a formatted PDF file, or recorded on audio tape, or filmed for DVD–we can begin to focus on preserving the content without regard for the form, particularly when that form is keeping good and important ideas from reproducing.

Previous in series
Next in series
Posted in Archiving, Writing | Tagged , | 5 Comments

Zen and the Art of Knowledge Maintenance, Part I: Ashes to Ashes, Zeroes to Zeroes

Posted on 3 December, 2010 by Arras

Table of contents for Zen and the Art of Knowledge Maintenance

  1. Zen and the Art of Knowledge Maintenance, Part I: Ashes to Ashes, Zeroes to Zeroes
  2. Zen and the Art of Knowledge Maintenance, Part II: Ideas Worth Spreading
  3. Zen and the Art of Knowledge Maintenance, Part III: Our Digital DNA
  4. Zen and the Art of Knowledge Maintenance, Part IV: No Bit Left Behind

On my bookshelf there sits a book called Dark Ages II: When the Digital Data Die, by Bryan Bergeron. It was published in 2001, and it’s now out of print, so finding a copy of this treatise on the dangers of entrusting human knowledge to digital formats is increasingly difficult, left to remainder bins and second-hand bookstores. Its publisher hasn’t deemed the book popular enough to warrant issuing it in digital format, either–no Kindle version, no ePub version, no PDF version. In a few short years this book will be forgotten, lost to the ages in a feat of sublime irony.

Bergeron’s book will likely survive in a good number of public libraries, of course, but without an index of its contents that can be searched electronically, its secrets will remain behind closed covers, revealed only to those few hardy souls with the curiosity, the physical access, and the knowledge of the book’s existence. Some will stumble across it by chance on a library shelf, others will discover it while poring over the references in the endnotes of scholarly papers, but the vast remainder will never learn of it, much less read it. Bergeron’s contribution to the sum of human knowledge is destined for oblivion, in other words, because it has no currency in the digital realm, a good candidate for the Warehouse of Unwanted Books.

It’s been pointed out, of course, that a Shakespeare manuscript from the 17th Century remains largely readable today, whereas a WordStar document from 1985 on a 5.25” floppy disk might as well be random gibberish. These are both instructive examples in their own way.

That Shakespeare manuscript, for example, persists today only because of environmental control systems that carefully regulate air temperature, humidity, and oxygen levels, in order to slow the decay of the paper and ink. Without the continuous availability of electricity to power those environmental control systems, books and papers that old would perish quickly. The expense involved in maintaining such a controlled environment also forces the librarian to be choosy about what materials are deemed worthy of preservation. The works of a famous writer like Shakespeare clearly merit special treatment, but how many works by lesser-known authors over the millennia have been lost simply because the monks and scribes and library curators lacked the time, budget, or shelf space to preserve them?

The WordStar example hits closer to home for me. Back in the mid-80’s I wrote a number of short stories–even a novel, though like any fledgling effort by a teenager it was nothing to brag about. I had an Atari 400 computer in those days, and the word processor I used was called AtariWriter. Every night, for a few hours after dinner, I’d sit down and continue the exciting adventure saga of Rivenhelm, until to my amazement a few months later I’d finished a 200-page opus. I knew it wasn’t in any way ready for publication, but nevertheless it was an intensely personal work and I figured I’d eventually go back to it when I’d aged a little and become a better writer. I saved the novel on a set of four single-sided, single-density floppy disks, put them in a diskette case for safe keeping, and moved on to other projects.

It was about ten years later that I decided to revisit Rivenhelm’s adventures as part of a wave of nostalgia. By then I was using a Pentium-based Windows 95 computer, and while it still had an old-style floppy disk drive, it was expecting double-density, double-sided disks, and had no understanding of how to read an Atari-formatted disk. I still had my old Atari computer, though, so I brought it out of storage and booted it up, figuring that would solve my problem. I put the first disk in the drive, and it was unreadable. The second disk was apparently unformatted. The third and fourth disks were marginally readable, but with badly corrupted files that AtariWriter couldn’t parse. I was gobsmacked, to put it mildly. What had happened to my data in just ten years?

The harsh reality is that magnetic media only holds its state for as long as its induced magnetic field remains stronger than the force of gravity acting on the ferrous filings–something called “bit rot”. When a filing is lying flat (in the plane of the disk), it represents a 0; when it’s coaxed into a perpendicular orientation it represents a 1. Over time, the magnetic field holding those filings in place weakens, while the force of gravity relentlessly tries to flatten them, so that in the end all the 1’s become 0’s–ashes to ashes, zeroes to zeroes. You can refresh the magnetic field simply by putting the disk in a drive and letting it spin around in there for a few read operations every once in a while, but if you keep those disks in a box for years, gravity will first corrupt, then ultimately erase your data.

But even if I’d been able to read the data from those old disks somehow, I’d have run into trouble trying to get a modern word processor like Microsoft Word to read the AtariWriter files. File formats undergo evolutionary and sometimes revolutionary changes as technology lets us do more interesting and complex things with our data. There was no choice of font with AtariWriter, for example–font was determined by the printer. There was no way to change the font size, either, and no way to incorporate images into a document. Modern word processors can embed a lot more information about a document’s layout, appearance, and content in its data format, and while Word can import some older word processor formats the list is relatively short. Eventually the developers decide that the demand for support of ancient formats is so low as to be negligible, and it gets dropped altogether.

NASA ran into this problem with their Moon landing tapes, which were recorded using a type of slow-scan rendering that only a handful of purpose-built machines could read. Over the decades those machines were mothballed, dismantled, and cannibalized for parts until only one semi-functional tape reader remained. Today the Lunar Orbiter Image Recovery Project is refurbishing two of these old AMPEX FR-900 tape drives in order to recover the full-resolution video of the recently rediscovered tapes. The obscure data format required custom hardware and custom media on which to record it, all of which served to limit the accessibility of the data and fuelled Moon landing hoax conspiracy theories, since there was no way for third parties to analyze the data first-hand. Never mind the fact that the tapes themselves sat in warehouses for decades, and that NASA lost track of them for 20 years, or that there were no backups made. That’s just stupid.

Next in series
Posted in Archiving, Writing | Tagged , | 2 Comments

Connecting the Dots of History

Posted on 1 December, 2010 by Arras

Over the past week I’ve been scanning genealogy records for one side of my family and entering the details into a GEDCOM database. I’ve now documented 873 persons from 265 families going back as far as 1475 in an unbroken genetic chain, which was as much work as it sounds. Even so, this was easy compared to the research a distant cousin of mine did back in 1985, before the Internet put these sorts of records at people’s fingertips. Paul Leblanc had to do it the hard way, visiting halls of records and cemeteries, talking to parish priests, and making an effort to get in touch with the far-flung members of an enormous clan. I tip my hat to him.

That said, if Paul’s house had burned down at any point during the past 25 years all of that research would have been lost. When I contacted him recently to get ahold of these records he had some difficulty finding them, and was initially hesitant to part with his only copy long enough for me to scan it. The situation drove home a point that I’ve been passionate about for some time: the vital need to keep important information backed up, and in a modern format that can be shared with the next generation.

He kept all of his notes on file cards, for instance, which made sharing that information awkward. That in turn meant that the information would not be shared very widely, and in its unwieldy form it wasn’t very appealing to work with. These days modern genealogy software all uses GEDCOM to structure and store this sort of data, and that makes it easy to share family trees with others. I don’t want to understate the power of good genealogy software to make the process engaging, however (and yes, I’ll gladly shill for MacFamilyTree, which has been an absolute pleasure to use). A dusty shoebox full of file cards doesn’t tell you anything until you take the time to go through all of those records and organize them. When you assemble them with genealogy software you can visualize that data in many different ways, and it’s an eye-opening experience to see the “big picture” and zoom in on interesting sections.

In the course of assembling the LeBlanc branch, for instance, I was struck by some of the broader patterns in the data, like the fact that the further you traveled from the place of your birth, the shorter your average lifespan became. Indeed, many of my ancestors lived well into their 90’s, but the ones who ventured further afield tended to die in their 20’s or 30’s. It’s tempting to attribute that to soldiers going off to war, but war was everywhere in New France in those days with the French and the British fighting over Acadia, so there should have been just as many 20- and 30-somethings dying at home as abroad. It’s more a testament to the dangers involved in traveling, I suspect, between accidents of nature, war, and “misadventure”.

Then there are the heart-wrenching stories that leap out at you from the analysis. In 1717, Etienne LeBlanc and his wife Marie had a daughter they named Marie Anne, only to lose her a few months later. They tried for another child, and in 1718 they were blessed with another daughter, and they named her Marie Anne as well, perhaps in honour of her late sister. Marie Anne II lived for just 13 months. In 1723 the couple tried once more to start their childless family, and another girl was born–Marie Anne III, who perished seven months later. No other children are recorded for Etienne and Marie, struck three times by tragedy. It’s then that you start to realize that these are not just numbers and lists and charts, but the lives of real people told in a handful of snapshots–births, deaths, weddings–just waiting for someone to connect the dots and bring them back to life.

Posted in Genealogy | Tagged | 2 Comments

Legacies in a Digital World

Posted on 24 November, 2010 by Arras

I spend a lot of time these days thinking about my legacy. It’s not particularly because my days seem more countably numbered, or that I’m feeling gloomy or morbid. I think it’s because without children to pick up where I leave off, I’m facing the prospect of having left the world no better than I found it, and that’s bothersome. I have some time left, and I’d like to more deliberately share some of my life’s lessons with the world in the hope that they can help someone, somewhere, sometime and make a positive difference. This blog is a fledgling effort in that direction.

This issue of digital legacies was brought home to me yesterday with the news that a longtime Maia Mailguard supporter–and hoster of the project’s website–had passed away in March of this year. Eight months after Jacob Leaver’s death, the server that he had set up for us at his ISP was still dutifully carrying out its mission, as oblivious to his demise as I was. It wasn’t until something finally went wrong with the server and needed an administrator’s attention that we discovered what had happened. That server had been effectively forgotten, not documented anywhere, and it’s doubtful the ISP even knew what it was hosting until our inquiries brought it to light.

It got me thinking about the little undocumented bits of knowledge I carry around in my head that really should be recorded someplace, so that if I get hit by a bus or something others can carry on without me. A lot of it is simple data–accounts, usernames, passwords, PIN codes, etc. I keep all that sort of thing locked up with 1Password, encrypted and backed up both onsite and offsite, but obviously I still need to let key people know where to find it and what the master password is. A letter or a USB stick in a safe deposit box at a bank and noted in a will would probably be a good idea.

I’m somewhat less concerned about the digital legacies of my work-related projects, but that’s only because of the open, shared nature of open source software development. The Maia Mailguard code is in an offsite repository, and other developers can and do access it as they please, so if I vanished overnight the project’s work could continue without too much interruption. That said, it’s a reminder that I need to commit to committing more of my private codebase to those repositories, even if it’s still in an experimental state; I have a bad habit of not wanting to commit code until it’s ready for prime time, and perfectionism will be my downfall there.

Then there are the more “procedural” things, less about the “what” than the “how”. The best way to uncover these is to go away for a week and see how your family fares in your absence. Without fail, some electronic device chooses to malfunction whenever I’m away, and my wife either works around the problem or calls me to find out how to fix it. This is supposed to make me feel needed, I suppose, but it tells me that I need to do a better job of documenting how our somewhat elaborate network and home theatre is configured.

Some of my personal data is already being deliberately collected for posterity. I’ve got thousands of family photos to scan, spanning several generations, which I’ll be assembling into a digital album with notations, and I’m in the process of accumulating genealogy records going back four centuries, which I’m scanning and piecing together into a modern GEDCOM file with MacFamilyTree. As an only child and a leaf-node in the family tree however, I’m doing this more for the sake of parallel branches–the children of my cousins, mainly, who may one day want to know where they came from, and pick up where I leave off. Family recipes are being recorded digitally with MacGourmet.

The rest is harder to sort through–a look through my documents and projects folders reveals a lot of half-formed ideas and experiments, few of which are really documented very well. Hobbies and research passions involved the gathering of a lot of useful information on a range of topics, typically in the hope of forming the foundations of a novel or a non-fiction treatise one day. Are these of use to anyone, and worth preserving? If so, to whom? I’m beginning to organize some of these things with DEVONthink, if only to group and classify them a bit better.

The issue of file formats for archival purposes is something else I’ve been meaning to write about, but I’ll save that for another entry. To nutshell it here, though, just imagine inheriting a bunch of 5 1/4” floppy disks with documents written in WordStar to appreciate the challenge of preserving digital information across the ages. For the moment I’m just focusing on making a more deliberate effort to dump the contents of my brain into digital formats, period, since that’s typically the hardest part of the process. Converting it from one digital format to another is comparatively simple.

If there’s a moral to this story, it’s that the human brain is a volatile storage device, and unless you perform regular backups you risk permanent data loss in the event of a system crash or catastrophic failure.

Posted in Uncategorized | 1 Comment

Hello (Cruel) World!

Posted on 21 November, 2010 by Arras

“Life’s but a walking shadow, a poor player
That struts and frets his hour upon the stage
And then is heard no more: it is a tale
Told by an idiot, full of sound and fury,
Signifying nothing.”
– William Shakespeare’s Macbeth

It’s been more than a decade since I’ve bothered to maintain a personal website, and despite having configured blogs like this for other people I’ve never actually had a blog of my own. Bear with me, then, as I make all the rookie mistakes ;)

It’s tempting to think that something like Facebook could replace blogs altogether, but I think that only works for people whose updates are relatively small and frequent. Twitter is at the extreme end of that scale with its 140-character limit, and Facebook is useful for a handful of paragraphs, but if you’re the sort of person who likes to write lengthier rants or make deeper observations about life, the universe, and everything you need a blog.

I realize, of course, that the attention span of the average Netizen has been shrinking over the past decade, and that “walls of text” are daunting even to the most interested and sympathetic readers. So why bother cultivating an “old school” medium like this that (almost) no one wants to read? Shouldn’t I be adapting to the new paradigm of information chopped into bite-sized pieces the reader can swallow without chewing? No, I think there’s plenty of content out there for people who don’t want to think; I’m aiming this content at the remainder (that would be you, I hope ;) ).

Besides, there’s a therapeutic benefit to putting certain thoughts and ideas in writing, even if no one ultimately reads them–that’s the basis of the personal diary, after all. That’s not to say that everything I write will be a tirade about the many things I think are wrong with the world, but there will certainly be a few of those, now that I’ve become an old curmudgeon. The rest of the time you can expect posts about computing, dogs, aviation, music, photography, all things Apple and Linux, and whatever else happens to be distracting me from what I should be doing.

Welcome, then, to the maze of twisty passages between my ears. I hope you find something interesting in there under all the dust! :)

Posted in Writing | Tagged | Leave a comment