a history of transparency

transparency is just a concept
© DREASAN

Two projects in IBM have recently decloaked, both which I've had the pleasure of working on: Jazz and Project Zero. Both projects have come under some attack by folks as being "not open source". For instance, see Mark Pilgrim's typically humorous response to Project Zero.

I'm not interested in talking about that specific aspect of the projects. I'm an open source commie from way back, but the fact of the matter is that IBM contributes a lot of open source to various communities, and at the same time produces commercial, proprietary software.

What I'm quite happy about with both products is the transparency they provide to IBM's product development process. The term "transparent" I've borrowed from Stephen O'Grady, which he used to describe the Jazz development process. I thought I'd give a quick rundown on the transparency I've experienced in my 20+ years of software development at IBM.

From 1985 till about 1995 I worked on projects internal to IBM, and didn't really have a need to deal with customer feedback on the products I was working on, because they were internal to IBM. We did have a 'newsgroup'-like system called "IBMPC Conferencing" that was a fantastic resource for IBMers. Not a whole lot of people used it (relatively), which was unfortunate, but also kept the wheat/chaff percentage quite high. It did serve as a way to communicate with our internal customers though.

The IBMPC conferencing system expanded sometime in the early 1990's to allow customers access to a restricted set of 'newsgroups' called CFORUMs. This was quite convenient for IBM developers, since it used the same news system we already knew how to use, and customers also had access to it somehow. Sometime later, IBM created a real NNTP server named news.software.ibm.com, which served the same purpose as CFORUMs, but used the more 'standard' NNTP protocol.

By this time, I was working on VisualAge Smalltalk, and we created at least one newsgroup for it on the NNTP server. For other projects I was involved with later, we also created newsgroups on the server.

So, we've provided some kind of newsgroup-y access to product groups for a while. However, we were limited in what we could actually discuss on the newsgroups. While we would frequently accept bug reports from users posted to the newsgroups, we couldn't really provide bug tracking numbers, since there was no way for users to access our bug tracking systems. We did it anyway, to at least give our users a shorter handle by which to reference the bugs. For new feature requests, or requests for when the next release would be made available, you'd typically see a response of "We cannot provide information regarding future versions of the product". Which was insanely frustrating for us developers. But we bit our tongues and pasted that response into posts, a lot.

So now, roll forward to the mid-late 2000's, with Jazz and Project Zero. We have 'newsgroup'-y access like we have had for a while. But we also have access to the bug tracking systems. And source code management systems. And a general notion of talking more openly about future release content and dates (I hope!)

It certainly seems to me, that over time, we've become more transparent with our development processes for our commercial products. I think this is great for customers, who will have more direct access to the product development teams and the product development process itself. It's also great for the development groups within IBM, as these open processes are a great equalizer: both new hires and CTOs have the opportunity to converse equally with customers.

I'm a firm believer in this sort of transparency being a benefit to everyone, and I'm looking forward to this becoming the rule rather than the exception for more IBM products.

WADL waffling

Waddle Waddle Waddle
©

Joe Gregorio answered some questions about WADL in his post "Do we need WADL?". Also note that Leonard Richardson has chimed in recently on the WADL issue. And I of course have some different thoughts. :-)

Quotes from Joe in bold italic, mine in plain.

If I describe an Atom Syndication Feed in WADL, how close will the generated code be to a feed reader like Bloglines? Or even to a library like Abdera? If I write a really good WADL for (X)HTML how close will the generated code be to a web browser? The point is that generated code stubs are so far from a completed consumer of a web service that the utility seems questionable.

I don't think anyone is expecting a magic machine to appear that eats WADL and generates applications out the other side. At best you're going to get some code stubs. Which is what WSDL does today. And functionally works. I'm not saying it's nice, or pretty; just that it functionally works. It is easier than writing all the SOAP glorp yourself, so I would say such a scheme has a lot of value.

Code generation of data bindings from XML Schema, though, seems fraught with problems. You can either design a nice document, in which case the resulting code will be 'ugly', or you can design nice objects and your XML schema will be be 'ugly'. That's why I'm interested in JSON; perhaps we can have nice objects AND documents serialized formats!

Yes, people want to describe interfaces, and those descriptions are brittle. If I download a WADL and compile my client today it will break tomorrow when you change the service. If, instead, you use hypertext, link following and request construction based on the hypertext and client state, then the interface won't break when you change servers, or URI structures.

I think of interfaces described here in the same way as Java interfaces. Namely, it's an external description of the system that people interact with. The guts can change, the interface can remain the same. One of the nice features of Java, if you're in the 'binding contracts' business (ie, you use Java). So, no, just because the service changes, does not imply that the client breaks.

But even beyond that, there's no reason someone can't deploy a new interface for a service and leave older interfaces still working. Some people create new versions of their service interfaces every few weeks and support each version for about a year.

Code gen is brittle, and I generally dislike it. But some languages don't require any code-gen, like PHP's SOAP support. Just give it a WSDL, it provides an interface to make calls against, as long as you can figure out what the methods and data are. Even for Java, there a minimizations that can be made; for instance, using dynamic Proxy's against generated Java Interfaces could leave you with a story of just having to generate Java Interface 'stubs', the rest being all handled dynamically.

And of course it would be useful to mention non code-gen uses; even if WADL were totally useless as a code gen device, it might still be handy to have as a documentation format for someone's services. It could also be used to provide validation for the client and server.

... you can't expect me to believe that if you had a carefully crafted WADL you could hand it to WADL2Java and out would pop flickrfs.

Again, of course we're not expecting fully formed apps or filesystems to pop out of a schema-2-language grinder (though I'm curious about what it would mean to plug APP into FUSE). But perhaps something like the "API Kits" listed here? Absolutely! That's what I'm talking about!

Q: You don't expect everything to be built with APP, do you?

Paraphrasing Joe's answer: Not everything, but a lot.

I'm leery of the blog-based legacy of APP. For instance, the second-class nature of binary resources. Also, attributes on collections and entries such as categories, title, etc. A lot of human readable, textual attributes. The kind of stuff you see in ... what is that word again ... oh yeah ... hypermedia.

I'm leery of APP, but I'm hopeful. It's especially nice to think of someone creating all the infrastructure for the CRUD-like interfaces, especially being able to handle HTTP cache validators (I hope). In the end though, you still need to describe the non-Atom data you are transferring over APP.

DCampSouth 2007

DCampSouth discussion group
© Patrick Mueller

On Saturday, I attended DCampSouth at the rather bizarrely shaped, but very cool School of Communication Arts (aka "The Digital Circus") outside Raleigh.

Lots of people will tell you that the best part of conferences in general are the informal, impromptu conversations that happen in the hallways. Unconferences like DCampSouth are all day, just slightly more structured, hallway conversations.

Big thank you to Jackson Fox and the rest of the crew who put this together.

I'm looking forward to attending the Ruby Hoedown 2007 later this summer, and the next BarCamp RDU this summer or fall.

browser scripting

surfing competition
© heydrienne

Finally we have a programmable persistence engine for our browsers. Thank you, Googleplex.

Here's what I want next: more scripting languages. Obvious choices being python and ruby. Whatever happened to this? Slingshot would still have a slight advantage over such a contraption, as Slingshot has some additional desktop integration features that browsers don't currently have. It also has the advantage that it's not running in an application shell designed for browsing the entire web; there's no time machine (back button).

There's also Adobe Flex/Apollo to consider, since they will also have an embedded database available. On the language front with Flex, Adobe recently made an ActionScript Virtual Machine 2 (AVM2) Overview available. How long before someone ports some languages to that VM? Especially since dynamic languages like python and ruby are a fairly natural fit to the AVM2 engine (compared to the JVM anyway), and the AVM2 engine will likely be the most widely deployed VM in the near future (it's included in Flash 9).

The one thing I've been most excited about given the rash of new client products available, is that we've finally got a new "browser war" on our hands. Competition is fantastic; it's going to be a wild next couple of years.

not doing REST

Rest Stop
© Grant Lindsay

From Mark Baker:

So if you're writing (or generating) contract/interface-level code which can't late-bind to all resources, everywhere, you're not doing REST ...

Is this "We don't need no stinkin contracts!" meme a reaction to the non-web-friendly WS-* world, what with it's overly complex and verbose schemas? Because I think there's plenty of room for some people to apply contracts to parts of the web. I certainly don't believe the entire web can be fully described using some all-encompassing schema language; but small pieces? Sure.

I guess what I don't understand, is how you are supposed to describe your services to someone without some kind of meta-data describing the service. Every 'web api' I've ever seen has human readable text describing the URIs, the data expected as input, and the data expected as output. (Admittedly, most of these 'web api's violate best practice HTTP principles somehow, but I think that's not an issue here; they could all be refactored to be be better HTTP citizens.) That human readable text is a contract; an interface. In English. Which is terrible. I'd rather have a machine readble version of that contract, so I can generate something human readable from it. And perhaps a validator. And some client stubs. Maybe some some test cases. Diagnostic tools. Etc.

What is the the alternative to describing your services? How is anyone going to write code to use these services, if they don't know where to send requests, what verbs to use, what data to send, and what kind of data to expect? Instead of Flickr producing a description of their web services like this, they're simply supposed to say "Flickr is now fully REST-enabled. Start here, and have fun!" ??

As with data modelling, I don't feel like there is a single answer to what schema or contract language be used. I'm not initially sold on WADL (seems too verbose), and certainly wouldn't use it if there was something else, better, for whatever project I was working on. The shape of the schema language isn't important, as long as it works for you.

So I guess contract-driven HTTP interfaces aren't REST. But this is an area I'm interested in; what name should I use, so I can avoid be labelled as "not doing REST" while I'm optimizing my use of the web by being a good HTTP citizen?

typing rest

The One Thing We Can't Live Without
© Ray

Count me as someone who wants some typing in the REST world, based on the arguments made in the post by Aristotle Pagaltzis last week.

We're talking about contracts here. Contracts need to be formalized, somehow. English is not the best language to use, especially since we have much more precise languages available to us.

My thoughts here are really just an extension to my thoughts on data serialization. Services are just the next level of thing that need to be meta-described.

Several folks have pointed out WADL (Web Application Description Language) as a potential answer, but it has at least one hole: it doesn't have a way of describing non-XML data used as input or output. For example, JSON. It certainly is simpler and more direct than WSDL, so it does have that going for it.

All in all, good thoughts all around, but we have more work to do, more proof to provide. And by more work, I don't mean getting a handful of experts in a smoky back room mandating what the formats are going to be. In fact, I'm not so sure we need a single 'format'. If you've creating some kind of machine-readable schema to describe your data and your services, you're way ahead of the game.

In any case, don't wait for WADL to be finshed before starting to build out schema for your services. Use WADL if you can, use something else (hopefully simpler) if it's more appropriate for you.

Additional thoughts on Aristotle's post from Tim Bray, Stefan Tilkov and Mike Herrick.

audio vs video

Video and Audio in and out
© Ameer Ghanem

The Redmonkers have been starting to web publish video interviews along with their usual audio interviews. Coté seems to be doing most (all?) of the work, and you can catch these as he releases them on his blog.

I like to see people experimenting with new technology, and 'upgrading' from audio to video sounds like a fun experiment (pardon the pun). But it doesn't work for me.

My issues:

• There really isn't that much 'extra' in a video interview, over just the audio. You get to see faces. You get to see some body language. Maybe a picture or two.

• The idea of watching an interview means I have to have two senses trained on the media: eyes and ears. You're killing my continuous partial attention!

• I can't listen to it on my video-challenged iPod.

• The reason I don't have a video-capable iPod is that the situations in which I listen to my iPod don't lend themselves to allowing me to watch something on the device as well: driving, mowing the lawn, washing the dishes, etc.

I fully admit to being an old fuddy-duddy; even Rush Limbaugh does video of his show. Good luck guys, and, if you can, also make the audio portion of your videos available as an MP3. I'm not alone in this wish.

But let me change the direction here. Let's look at an environment that is high on video interaction, and absolutely bereft of audio interaction. Your programming environment. Your IDE, be it Eclipse, NetBeans, IntelliJ, Visual Studio, XCode, Emacs, or a text editor and a command-line. How many of these programs use audio to help you develop code? None. Well, I might be lying, I'm not familiar with all of these environments, but I don't recall any of these environments making use of audio like they do visuals.

When we're programming on all cylinders, we're in 'the zone'. Continuous full attention. Eyes reading and scanning, fingers typing and clicking and moving mice. Where's the audio? It ain't there.

Nathan Harrington has a number of articles up at developerWorks, such as "Monitor your Linux computer with machine-generated music" which discuss ways developers can use audio in their computing environment.

This is good stuff, and we need more of it.

I would be remiss in not pointing out here that audio feedback like this is only useful to those of us lucky enough to have decent audio hardware and software in our heads. Those of us without such luck wouldn't be able to take advantage of audio feedback. On the other hand, folks who lack decent video hardware and software in their heads would most likely appreciate more emphasis on a sense they are more dependant on.

The most obvious use case for audio in a development environment is with debugging. There are a lot of cases while debugging when you just want to know if you hit a certain point in your program. The typical way you'd do this is to set a breakpoint, and when the location is hit, the debugger stops, you see where you are, and hit continue. Breaking your attention and demanding your input. What if, instead, you could set an audio breakpoint, that would play a sound when the location was hit? So your attention wasn't broken. And you didn't have to press the Continue button to proceed.

With regard to audio debugging, I know this has been experimented with many times in the past. I've done it as well, a decade ago, when I was using a programming environment that I was able to easily reprogram: Smalltalk.

But audio usage in development environments is not yet mainstream. There's lots of research to be done here:

• What are the best sound palettes to use: audio clips, midi tones, short midi sequences, percussion vs tones?

• How should we take advantage of other audio aspects like volume and pitch and three dimensional positioning, especially regarding continuously changing quantities like memory or i/o usage by an application?

• How do we deal with 'focus' if I'm also listening to Radio Paradise while I'm working?

• Does text-to-speech make sense in some cases?

• How do we arrange multiple audio feedback to be presented to us in a way that's not unpleasing to listen to? Not just a cacophony of random sounds.

• Beyond debugging, where else can we make use of audio feedback?

• How might audio be integrated into diagnostic tools like DTrace?

ETags are not a panacea

panacea
© Rufino

panacea: A remedy for all diseases, evils, or difficulties; a cure-all.
From Answers.com

In my post "That Darned Cat! - 1", I complained about Twitter performance, and peeked at some of their HTTP headers noticing they didn't seem to respect ETags or Last-Modified header cache validator tests.

Since posting, Twitter performance is back on track. I haven't checked, but I'm guessing they didn't add ETag support. :-)

A number of people seemed to read into my post that ETags are a cause of Twitter's performance problems. I'd be the first to admit that such a proposition is a bit of a stretch. ETags are no panacea, and in fact you'll obviously have to write more code to handle them correctly. Harder even, if you're using some kind of high level framework for your app. This isn't easy stuff.

And in general, my 20+ years of programming have taught me that your first guess at where your performance problems in your code are, is dead wrong. You really need to break out some diagnostic tools, or write some, to figure out where your problems are. Since I don't have the Twitter code, I'm of course at a complete loss to guess where their problems are, when they have them.

ETags and Last-Modified processing is something you ought to do, if you can afford it, because it does allow for some optimization in your client / server transactions. To be clear, the optimization is that the server doesn't have to send the content it would have sent to the client, as the client has indicated it already has that 'version' of it cached. There is still a round-trip to the server involved. If you're looking for an absolute killer optimization though, you should be looking at Expires and Cache-Control headers. See Mark Nottingham's recent post "Expires vs. max-age" for some additional information, along with the link to his caching tutorial.

Expires and friends are killer, because they allow the ultimate in client / server transaction optimization; the transaction is optimized away completely. The client can check the expiration data, and determine that the data they have cached has not 'expired', and thus they don't need to ask the server for it at all. Unfortunately, many applications won't be able to use these headers, if their data is designed to change rapidly; eg, Twitter.

Sam Ruby also blogged about another great example of Expires headers. How often does the Google maps image data really change?

Here's another great example, applicable to our new web 2.0-ey future. Yahoo! is hosting their YUI library for any application to use directly, without copying the toolkit to their own web site. Let's peek at the headers from one of their files:

$ curl -I http://yui.yahooapis.com/2.2.2/build/reset/reset-min.css
HTTP/1.1 200 OK
Last-Modified: Wed, 18 Apr 2007 17:35:45 GMT
X-Yahoo-Compressed: true
Content-Type: text/css
Cache-Control: max-age=313977290
Expires: Tue, 02 May 2017 04:08:44 GMT
Date: Mon, 21 May 2007 04:13:54 GMT
Connection: keep-alive

Good stuff! The Expires and Cache-Control headers render this file pretty much immutable, as it should be. When Yahoo! releases the next version of the toolkit, it'll be hosted at a different url base, and so will be unaffected by the headers of this particular file; they will be different urls. This sort of behaviour is highly optimal for web 2.0-ey apps, which are wont to download a lot of static html, css and javascript files, which, for some particular version of the app, will never change. And thus, by having the files cached on the client in such a way that it never asks the server for them again, the app will come up all the quicker.

Good stuff to know, and take advantage of if you can.

For more specific information about our essential protocol, HTTP 1.1, see RFC 2616. It's available in multiple formats, here: http://www.faqs.org/rfcs/rfc2616.html.

blogging process

process
© jen light

In "Lesson learned", my colleague Robert Berry recounts 'losing' a blog post he was editing. Not the first time I've heard this recently. I thought I'd document my process of creating blog posts, in case it's of any use to anyone. Because I don't lose blog posts.

My secret: I use files.

Although many blogging systems let you edit your blog posts 'online', and even let you save them as drafts, I don't actually go into my blogging system to enter a blog post, until it's complete. The process is:

• Create a new blog entry by going into the Documents/blog-posts folder in my home directory of my primary computer, and creating a new file, the name of which will be the title of the blog post. The 'extension' of the file is .html.

• Edit the blog post in html, in a plain old text editor.

• While editing, at some point, double click on the file in my file system browser (Explorer, PathFinder, Nautilus, etc) to preview it in a web browser.

• Churn on the edit / proof-read-in-a-web-browser cycle, for hours or days.

• Ready to post? First, check all links.

• Surf over to blog editing site, enter the body of the post into the text editor via the clipboard, set the title, categories / links, etc.

• Preview the post on the blog editing site. Press the "Publish" button.

• Move the file with the blog post from Documents/blog-posts to Documents/blog-posts/posted .

HTML TextAreas are an extremely poor replacement for a decent text editor. Using HTML is handy, since some (most?) blogging systems will accept it as input, and you can preview it yourself with your favorite web browser. Saving the files, even after finished posting, is a convenient backup mechanism, should you ever lose your entire blog.

Besides these obvious advantages, I noticed some behaviours of other blogging systems that I really didn't like, when saving drafts of posts 'online':

• On one system I used, the title saved with the first draft was used as the slug of the blog URL. Even if I later changed the title, the slug remained some abbreviated version of the first saved title. Ick.

• On one system I used, tags I saved with a post ended up showing up in the global list of tags on the blog. Even if there weren't any published posts that had used that tag. Ick.

I should note that I also have a directory Documents/blog-posts/unused for posts which I've started, and decided not to post. The "island of misfit blog posts", as it were, but "unused" was shorter.

There you have it! Since you religiously backup files on your primary computer you'll have no concern about ever losing a blog post again!

That Darned Cat! - 2

The Twitter Maintenance Cat

Some more thoughts on Twitter performance, as a followup to "That Darned Cat! - 1".

Twitter supports three different communication mediums:

• SMS text messaging
• A handful of IM (Instant Messaging) services
• HTTP - which can be further subdivided into the web page access, the Twitter API, and RSS and Atom feeds

I'm not going to talk about the first two, since I'm not familiar with the technical details of how they work. Other than to notice that I don't see how Twitter can be generating any direct revenue off of HTTP (no ads on the web pages, even), whereas they could certainly be generating revenue of off the SMS traffic they drive to whoever hosts their SMS service. IM? Dunno.

It would appear, or at least I guess, that most of the folks I follow on Twitter are using HTTP, rather than the other communication mediums. Maybe I'm in a microcosm here, but I'm guessing there are a lot of people who only use the HTTP medium. And there's no money to be made there.

So, we got a web site getting absolutely pounded, that's generating no direct revenue for the traffic it's handling. And it's become a bottleneck. What might we do?

Distribute the load.

Here's a thought on how this might work. Instead of people posting messages to Twitter, have them post to their own site, just like a blog. HTTP-based Twitter clients could then feed off of the personal sites, instead of going through the Twitter.com bottleneck.

This sounds suspiciously like blogging, no? Well, it is a lot like blogging. Twitter itself is a lot like blogging to begin with. Only the posts have to be at most 140 bytes. So let's start thinking about it in that light, and see what tools and techniques we can bring from that world.

For instance, my Twitter 'friends' are nothing but a feed aggregator like Planet Planet or Venus. Only the software to do this would be a lot easier.

Josh notes: Hmm, but doesn't RSS/Atom already give us everything we need for a twitter protocol minus the SMS (and text limit)? Indeed. Twitter already does support both RSS and Atom (I didn't see explicit Atom links, but find an RSS link @ Twitter, and replace the .rss URL suffix with .atom). They aren't things of beauty, but it something to start with. While you can already using blogging tools to follow Twitter, I'm not sure that makes sense for most people. However, reusing the data formats probably makes a lot of sense.

So, why would Twitter ever want to do something like this? I already mentioned they don't seem to be making any direct revenue off the HTTP traffic, so off-loading some of that is simply going to lower their network bill. They could concentrate instead, in providing some kind of value, such as contact management and discovery. Index the TwitterSphere, instead of owning and bottlenecking it. And of course continue to handle SMS and IM traffic, if that happens to bring in some cash.

In the end, I'm not sure any one company can completely 'own' a protocol like this forever. Either they simply won't be able to afford to (expense at running it, combined with a lack of revenue), or something better will come along to replace it.

If you love something, set it free.

There are other ideas. In "Twitter Premium?", Dave Winer suggests building Twitter "peers". This sounds like distributing Twitter from one central site, to a small number of sites. I don't think that's good enough. Things will scale better with millions of sites.