how RESTy need we be

That's Good Enough for Somebody
© Edward O'Connor

A few blog posts reverberating in my mind:

I just ran across a post by Joe Gregorio today, where he's comparing WS-* RPC vs. REST, specifically talking about the fact that WS-* RPC only uses the POST request method of HTTP:

"That POST of a generic media type gives no indication if the request is safe, or idempotent, nor is there any indication of the cachability of the response."

I also just read this morning a post by Leonard Richardson on Amazon's new FPS service, commenting on the 'REST'y ness of the service:

"its 'REST' interface is about as RESTful as Flickr's and del.icio.us's 'REST' interfaces"

Note, the Flickr and del.icio.us aren't considered terribly RESTy. :-)

Joe notes a handful of things that "just using POST in HTTP" breaks. Leonard notes that FPS isn't truly RESTy, but is in fact just as non-RESTy as two other fairly popular services.

And I'm left wondering: do we really need to do everything in pure REST?

Perhaps we can identify a small set of characteristics that get us most of the benefits of REST, that would be easier to implement than going 'full' REST. And just focus on those. Because apparently, it's hard to do full REST. Or to be even more pessimistic, perhaps there are disadvantages to using REST. Why would Amazon and Yahoo not use REST?

Most of the obvious advantages with REST revolve around the GET verb. Make sure you only put safe, idempotent actions behind the logic of your GET processing. Use ETags, or even better, cache directives, in your HTTP responses on the GETs to allow clients to intelligently cache things.

What else? Are there really other benefits? If there are, why are people still not jumping on the RESTy bandwagon?

Here's my thought on the lack of adoption of REST: it's complicated for anything but the GETs. Namely, the mutating verbs POST, PUT, and DELETE. Not technically, but semantically. Check the rest-discuss mailing list sometime. I've been having conversations with folks here recently regarding some of the semantics of 'updates' as well; ask 5 people their thoughts, you'll get 10 different answers.

Martin Nally had told me a few months ago that he thought people should be able to do 90-95% of their services in a RESTy style, and for those interfaces that didn't fit the REST mold, you could do 'em in non-RESTy style (ie, some kind of RPC-ish non-GET mutation operation), but it should cost you a $1000. I think I'm ready to lower the price to $100.

---

Updated on 2007/08/03 to fix a syntax error in the image element.

client generation

Handmade burgers
© Becca G

Dan Jemiolo: What kind of client generation are you looking for?

~shiver~

I suppose I must not have gotten around to telling Dan my horror stories of using WSDL in the early, early days of Jazz. The low point was when it once took me two working days to get the code working again, after we made some slight changes to the WSDL. Of course, we were doing some evil, evil things, like parsing Java code with a JDT jar in an ant task, and replacing generated code from the WSDL generation process. But still, code generation of this ilk leaves a bad taste in my mouth.

Also see Dare's issues with bananas.

The best code generation is no code generation.

And that's what we changed in Jazz. Because we already had complete control over the data typing story (EMF), we had no problem generating the XML and JSON we wanted, completely dynamically, by reflecting over the data used in our services. But we had to do something about the service methods themselves.

So we rolled our own client and server side stack for this.

We kept the notion of defining the operations in a web service in a Java interface, because this makes a lot of sense to do in Java. We can reflect over it to look at the methods and signatures. On the server, you can write a class to implement the interface, and that's your service implementation. The low-level server interface (ie, Servlet for Java) can figure out what service to invoke, and then call the service implementation reflectively. And on the client, you can use Proxy and friends to build an object which implements the interface by making the HTTP request on your behalf.

(Quick aside to note that Jazz services are largely RPC styled, though there are some that are more RESTy flavored - stay tuned; they've caught the REST bug. I think the 'client library' or invocation style is largely independent of the architectural style, so I think everything I'm saying here completely holds for REST as well as RPC, and everything in between.)

By having the client and server work completely reflectively, all we had to do was make sure the data classes and service interfaces were the same (or at least compatible) between the client and server. Understand, "all we had to do" can be a problem; but at least we didn't have the generated code to deal with as well, nor did we have a separate build step for it that can be an agility killer.

It goes without saying that you can get by with a lot less configuration in such a situation. Reflection over Configuration. Don't Repeat Yourself.

Looking back on this, I think this was a great trade-off in the time expended to build the stacks. For instance, we were able to tweak the system in various ways that would have been impossible to do with a code-gen built system. I suspect this is going to be the case for any medium- to large-scaled system built using a number of services. You can either lock yourself into a system under which you have very limited control, and spend your time working around it and fighting it, or you can write code customized to your needs and spend time tweaking as you need.

Let's get back to your original question, but tweak it a bit: What should we do to make it easier for people to build clients?

• Provide machine-readable descriptions of the service interfaces. These machine-readable descriptions can be used to generate human-readable descriptions. For instance, instead of generating an HTML table for your Gregorio tables, how about generating JSON? Then you can generate an HTML file that uses some JavaScript to format the result in an HTML table. Perhaps you can use it, once you've built a little client invocation stack, to build some dynamic callable objects which serve as proxies to the server (if you're lucky enough to be working in a dynamic language).
• Provide machine-readable descriptions of the data flowing over the wire. Not just the HTTP content, but query string parameters where appropriate.

If you can reflect on your service interfaces and data dynamically in the server, then you can generate all this meta-data reflectively as well.

Or if you really do want to do some code gen, and your client is able to load directly runnable code from your server *cough* JavaScript *cough* then it's easy to imagine that you could code-gen a client library dynamically on the server as well. I suspect that's actually overkill though.

REST helpers

May 14 2006 resting table
© the .square

Dan Jemiolo posted today about his restdoc tool, which produces Gregorio Tables from comments in REST service implementations designed to be run in Project Zero. He also posted to the Project Zero forum, and included some screen shots of the output of his tool, here.

Here is what's cool about this:

• I love it when I can keep artifact information like this with my source code; easy to keep in sync.

• It's genuinely useful information.

• I'm tired of the "we don't need no stinkin' tools" attitude of some of the RESTafarians. Baloney. I think we can certainly live without overengineered tools like WSDL, but having small tools can certainly help.

Now, for some rocks:

• As I mentioned in a comment on Dan's blog, it would probably be useful to have this information available at runtime on the server; the server could actually validate what it's doing. And I'm not talking about the server reading it's restdoc information back in; I'm talking about the server having that information available to it, at runtime, obtained reflectively. The advantage is that you know the information is never stale. No more, "When was the last time I ran restdoc again?". In Java, you would presumably use annotations to do this.

• While Joe's tables included a non-precise description of the HTTP content for the method / uri rows (eg, "Employee Format"), restdoc only includes the 'format' (eg, JSON). I'd prefer to see more precise typing, but to start with, something as vague as Joe's would be good.

• That HTML table looks like it may be too wide to be viewed on an iPhone. The good news is, I don't have to worry about that problem.

Finally, not to knock Dan at all, but I really have to wonder what's going on if the best we can do is describe our services in human readable tables of text. Really? I mean, can't these multi-core, multi-gigahertz computers of ours really help out anymore than by rendering some HTML to a display that we read while hand-coding our HTTP invocations?

That just isn't right.

erlang meet-up

Erlang Song
© Chen Zhao

I went to the first erlounge RDU meet-up tonight to find out more about Erlang. The meet-up was arranged by Kevin Smith, who provided a short presentation.

I had done a little boning up on Erlang last night, so the presentation wasn't completely foreign to me, and I was able to squeeze a few questions in while the S5 presentation took forever to switch slides. Some of those questions:

• Q: Does Erlang compile to binary, or use bytecodes, or ???

  A: Compile to bytecodes.

• Q: What's the deployment story for web server code?

  A: Run yaws, a web server written in Erlang. Presumably, you'd be able to proxy to this via Apache, like other stand-alone servers (on a shared host, for instance; TextDrive doesn't currently have it installed, asking about it now).

After doing a bit of reading last night, I came away a little less enthusiastic than when I started. Not sure why. The meet-up re-invigorated me a bit. It's probably time to buy the book. Even if I don't actually do anything with Erlang, it's always nice to see what's going on with other languages, in hopes of maybe transferring some of those ideas, as appropriate, into other work you're doing.

On the negative side, it sounds like the error reporting (compilation and runtime) is fairly nasty. And the doc is largely man pages. And there's no real unicode support. And it's compiled.

It was a good omen to see my old young buddy, and fellow Building-50x-ite Chris Grindstaff at the meet-up. Chris has been yammering on about Ruby for years. Coincidently, the erlounge RDU meet-up group is an off-shoot of the Raleigh-area Ruby Brigade. Hmmm.

a history of transparency

transparency is just a concept
© DREASAN

Two projects in IBM have recently decloaked, both which I've had the pleasure of working on: Jazz and Project Zero. Both projects have come under some attack by folks as being "not open source". For instance, see Mark Pilgrim's typically humorous response to Project Zero.

I'm not interested in talking about that specific aspect of the projects. I'm an open source commie from way back, but the fact of the matter is that IBM contributes a lot of open source to various communities, and at the same time produces commercial, proprietary software.

What I'm quite happy about with both products is the transparency they provide to IBM's product development process. The term "transparent" I've borrowed from Stephen O'Grady, which he used to describe the Jazz development process. I thought I'd give a quick rundown on the transparency I've experienced in my 20+ years of software development at IBM.

From 1985 till about 1995 I worked on projects internal to IBM, and didn't really have a need to deal with customer feedback on the products I was working on, because they were internal to IBM. We did have a 'newsgroup'-like system called "IBMPC Conferencing" that was a fantastic resource for IBMers. Not a whole lot of people used it (relatively), which was unfortunate, but also kept the wheat/chaff percentage quite high. It did serve as a way to communicate with our internal customers though.

The IBMPC conferencing system expanded sometime in the early 1990's to allow customers access to a restricted set of 'newsgroups' called CFORUMs. This was quite convenient for IBM developers, since it used the same news system we already knew how to use, and customers also had access to it somehow. Sometime later, IBM created a real NNTP server named news.software.ibm.com, which served the same purpose as CFORUMs, but used the more 'standard' NNTP protocol.

By this time, I was working on VisualAge Smalltalk, and we created at least one newsgroup for it on the NNTP server. For other projects I was involved with later, we also created newsgroups on the server.

So, we've provided some kind of newsgroup-y access to product groups for a while. However, we were limited in what we could actually discuss on the newsgroups. While we would frequently accept bug reports from users posted to the newsgroups, we couldn't really provide bug tracking numbers, since there was no way for users to access our bug tracking systems. We did it anyway, to at least give our users a shorter handle by which to reference the bugs. For new feature requests, or requests for when the next release would be made available, you'd typically see a response of "We cannot provide information regarding future versions of the product". Which was insanely frustrating for us developers. But we bit our tongues and pasted that response into posts, a lot.

So now, roll forward to the mid-late 2000's, with Jazz and Project Zero. We have 'newsgroup'-y access like we have had for a while. But we also have access to the bug tracking systems. And source code management systems. And a general notion of talking more openly about future release content and dates (I hope!)

It certainly seems to me, that over time, we've become more transparent with our development processes for our commercial products. I think this is great for customers, who will have more direct access to the product development teams and the product development process itself. It's also great for the development groups within IBM, as these open processes are a great equalizer: both new hires and CTOs have the opportunity to converse equally with customers.

I'm a firm believer in this sort of transparency being a benefit to everyone, and I'm looking forward to this becoming the rule rather than the exception for more IBM products.

WADL waffling

Waddle Waddle Waddle
©

Joe Gregorio answered some questions about WADL in his post "Do we need WADL?". Also note that Leonard Richardson has chimed in recently on the WADL issue. And I of course have some different thoughts. :-)

Quotes from Joe in bold italic, mine in plain.

If I describe an Atom Syndication Feed in WADL, how close will the generated code be to a feed reader like Bloglines? Or even to a library like Abdera? If I write a really good WADL for (X)HTML how close will the generated code be to a web browser? The point is that generated code stubs are so far from a completed consumer of a web service that the utility seems questionable.

I don't think anyone is expecting a magic machine to appear that eats WADL and generates applications out the other side. At best you're going to get some code stubs. Which is what WSDL does today. And functionally works. I'm not saying it's nice, or pretty; just that it functionally works. It is easier than writing all the SOAP glorp yourself, so I would say such a scheme has a lot of value.

Code generation of data bindings from XML Schema, though, seems fraught with problems. You can either design a nice document, in which case the resulting code will be 'ugly', or you can design nice objects and your XML schema will be be 'ugly'. That's why I'm interested in JSON; perhaps we can have nice objects AND documents serialized formats!

Yes, people want to describe interfaces, and those descriptions are brittle. If I download a WADL and compile my client today it will break tomorrow when you change the service. If, instead, you use hypertext, link following and request construction based on the hypertext and client state, then the interface won't break when you change servers, or URI structures.

I think of interfaces described here in the same way as Java interfaces. Namely, it's an external description of the system that people interact with. The guts can change, the interface can remain the same. One of the nice features of Java, if you're in the 'binding contracts' business (ie, you use Java). So, no, just because the service changes, does not imply that the client breaks.

But even beyond that, there's no reason someone can't deploy a new interface for a service and leave older interfaces still working. Some people create new versions of their service interfaces every few weeks and support each version for about a year.

Code gen is brittle, and I generally dislike it. But some languages don't require any code-gen, like PHP's SOAP support. Just give it a WSDL, it provides an interface to make calls against, as long as you can figure out what the methods and data are. Even for Java, there a minimizations that can be made; for instance, using dynamic Proxy's against generated Java Interfaces could leave you with a story of just having to generate Java Interface 'stubs', the rest being all handled dynamically.

And of course it would be useful to mention non code-gen uses; even if WADL were totally useless as a code gen device, it might still be handy to have as a documentation format for someone's services. It could also be used to provide validation for the client and server.

... you can't expect me to believe that if you had a carefully crafted WADL you could hand it to WADL2Java and out would pop flickrfs.

Again, of course we're not expecting fully formed apps or filesystems to pop out of a schema-2-language grinder (though I'm curious about what it would mean to plug APP into FUSE). But perhaps something like the "API Kits" listed here? Absolutely! That's what I'm talking about!

Q: You don't expect everything to be built with APP, do you?

Paraphrasing Joe's answer: Not everything, but a lot.

I'm leery of the blog-based legacy of APP. For instance, the second-class nature of binary resources. Also, attributes on collections and entries such as categories, title, etc. A lot of human readable, textual attributes. The kind of stuff you see in ... what is that word again ... oh yeah ... hypermedia.

I'm leery of APP, but I'm hopeful. It's especially nice to think of someone creating all the infrastructure for the CRUD-like interfaces, especially being able to handle HTTP cache validators (I hope). In the end though, you still need to describe the non-Atom data you are transferring over APP.

DCampSouth 2007

DCampSouth discussion group
© Patrick Mueller

On Saturday, I attended DCampSouth at the rather bizarrely shaped, but very cool School of Communication Arts (aka "The Digital Circus") outside Raleigh.

Lots of people will tell you that the best part of conferences in general are the informal, impromptu conversations that happen in the hallways. Unconferences like DCampSouth are all day, just slightly more structured, hallway conversations.

Big thank you to Jackson Fox and the rest of the crew who put this together.

I'm looking forward to attending the Ruby Hoedown 2007 later this summer, and the next BarCamp RDU this summer or fall.

browser scripting

surfing competition
© heydrienne

Finally we have a programmable persistence engine for our browsers. Thank you, Googleplex.

Here's what I want next: more scripting languages. Obvious choices being python and ruby. Whatever happened to this? Slingshot would still have a slight advantage over such a contraption, as Slingshot has some additional desktop integration features that browsers don't currently have. It also has the advantage that it's not running in an application shell designed for browsing the entire web; there's no time machine (back button).

There's also Adobe Flex/Apollo to consider, since they will also have an embedded database available. On the language front with Flex, Adobe recently made an ActionScript Virtual Machine 2 (AVM2) Overview available. How long before someone ports some languages to that VM? Especially since dynamic languages like python and ruby are a fairly natural fit to the AVM2 engine (compared to the JVM anyway), and the AVM2 engine will likely be the most widely deployed VM in the near future (it's included in Flash 9).

The one thing I've been most excited about given the rash of new client products available, is that we've finally got a new "browser war" on our hands. Competition is fantastic; it's going to be a wild next couple of years.

not doing REST

Rest Stop
© Grant Lindsay

From Mark Baker:

So if you're writing (or generating) contract/interface-level code which can't late-bind to all resources, everywhere, you're not doing REST ...

Is this "We don't need no stinkin contracts!" meme a reaction to the non-web-friendly WS-* world, what with it's overly complex and verbose schemas? Because I think there's plenty of room for some people to apply contracts to parts of the web. I certainly don't believe the entire web can be fully described using some all-encompassing schema language; but small pieces? Sure.

I guess what I don't understand, is how you are supposed to describe your services to someone without some kind of meta-data describing the service. Every 'web api' I've ever seen has human readable text describing the URIs, the data expected as input, and the data expected as output. (Admittedly, most of these 'web api's violate best practice HTTP principles somehow, but I think that's not an issue here; they could all be refactored to be be better HTTP citizens.) That human readable text is a contract; an interface. In English. Which is terrible. I'd rather have a machine readble version of that contract, so I can generate something human readable from it. And perhaps a validator. And some client stubs. Maybe some some test cases. Diagnostic tools. Etc.

What is the the alternative to describing your services? How is anyone going to write code to use these services, if they don't know where to send requests, what verbs to use, what data to send, and what kind of data to expect? Instead of Flickr producing a description of their web services like this, they're simply supposed to say "Flickr is now fully REST-enabled. Start here, and have fun!" ??

As with data modelling, I don't feel like there is a single answer to what schema or contract language be used. I'm not initially sold on WADL (seems too verbose), and certainly wouldn't use it if there was something else, better, for whatever project I was working on. The shape of the schema language isn't important, as long as it works for you.

So I guess contract-driven HTTP interfaces aren't REST. But this is an area I'm interested in; what name should I use, so I can avoid be labelled as "not doing REST" while I'm optimizing my use of the web by being a good HTTP citizen?

typing rest

The One Thing We Can't Live Without
© Ray

Count me as someone who wants some typing in the REST world, based on the arguments made in the post by Aristotle Pagaltzis last week.

We're talking about contracts here. Contracts need to be formalized, somehow. English is not the best language to use, especially since we have much more precise languages available to us.

My thoughts here are really just an extension to my thoughts on data serialization. Services are just the next level of thing that need to be meta-described.

Several folks have pointed out WADL (Web Application Description Language) as a potential answer, but it has at least one hole: it doesn't have a way of describing non-XML data used as input or output. For example, JSON. It certainly is simpler and more direct than WSDL, so it does have that going for it.

All in all, good thoughts all around, but we have more work to do, more proof to provide. And by more work, I don't mean getting a handful of experts in a smoky back room mandating what the formats are going to be. In fact, I'm not so sure we need a single 'format'. If you've creating some kind of machine-readable schema to describe your data and your services, you're way ahead of the game.

In any case, don't wait for WADL to be finshed before starting to build out schema for your services. Use WADL if you can, use something else (hopefully simpler) if it's more appropriate for you.

Additional thoughts on Aristotle's post from Tim Bray, Stefan Tilkov and Mike Herrick.