That Darned Cat! - 1

The Twitter Maintenance Cat

The performance of Twitter as of late has been abysmal. I'm getting tired of seeing tweets like "Wondering what happened to my last 5 tweets" and "2/3 of the updates from Twitterrific never post for me. Is this normal?". I'm especially tired of seeing that darned cat!

Pssst! I don't think the cat is actually helping! Maybe you should get him away from your servers.

Here's a fun question to ask: do you support ETags?

In order to test whether Twitter is doing any of the typical sorts of caching that it could, via ETag or Last-Modified processing, I wrote a small program to issue HTTP requests with the relevant headers, which will indicate whether the server is taking advantage of this information. The program, http-validator-test.py, is below.

First, here are the results of targetting http://python.org/ :

$ http-validator-test.py http://python.org/
Passing no extra headers
200 OK; Content-Length: 15175; Last-Modified: Fri, 18 May 2007 01:41:57 GMT; ETag: "60193-3b47-b04e2340"

Passing header: If-None-Match: "60193-3b47-b04e2340"
304 Not Modified; Content-Length: None; Last-Modified: None; ETag: "60193-3b47-b04e2340"

Passing header: If-Modified-Since: Fri, 18 May 2007 01:41:57 GMT
304 Not Modified; Content-Length: None; Last-Modified: None; ETag: "60193-3b47-b04e2340"

The first two lines indicate no special headers were passed in the response, and that a 200 OK response was returned with the specified Last-Modified and ETag headers.

The next two lines show an If-None-Match header was sent with the request,indicating to only send the content if it's ETag doesn't match the value passed. It does match, so a 304 Not Modified is returned instead, indicating no content will be sent down (it hasn't changed since you last asked for it).

The last two lines show an If-Modified-Since header was sent with the request,indicating to only send the content if it's last modified date is later than the value specified. It's not later, so a 304 Not Modified is returned instead, indicating no content will be sent down (it hasn't changed since you last asked for it).

For content that doesn't change between requests, this is exactly the sort of behaviour you want to see from the server.

Now, let's look at the results we get back from going to my Twitter page at http://twitter.com/pmuellr :

$ http-validator-test.py http://twitter.com/pmuellr
Passing no extra headers
200 OK; Content-Length: 26491; Last-Modified: None; ETag: "a246e2e41e13726b7b8f911995841181"

Passing header: If-None-Match: "a246e2e41e13726b7b8f911995841181"
200 OK; Content-Length: 26504; Last-Modified: None; ETag: "1ef9e784fa85059db37831c505baea87"

Passing header: If-Modified-Since: None
200 OK; Content-Length: 26503; Last-Modified: None; ETag: "2ba91b02f418ed74e316c94c438e3788"

Rut-roh. Full content sent down with every request. Probably worse, generated with every request. In Ruby. Also note that no Last-Modified header is returned at all, and different ETag headers were returned for each request.

So there's some low fruit to be picked, perhaps. Semantically, the data shown on the page did not change between the three calls, so really, the ETag header should not have changed, just as it didn't change in the test of the python site above. Did anything really change on the page? Let's take a look. Browse to my Twitter page, http://twitter.com/pmuellr, and View Source. The only thing that really looks mutable on this page, given no new tweets have arrived, is the 'time since this tweet arrived' listed for every tweet. That's icky.

But poke around some more, peruse the gorgeous markup. Make sure you scroll right, to take in some of the long, duplicated, inline scripts. Breathtaking!

There's a lot of cleanup that could happen here. But let me get right to the point. There's absolutely no reason that Twitter shouldn't be using their own API in an AJAXy style application. Eating their own dog food. As the default. Make the old 1990's era, web 1.0 page available for those people who turn JavaScript off in their browser. Oh yeah, a quick test of the APIs via curl indicates HTTP requests for API calls do respect If-None-Match processing for the ETag.

The page could go from the gobs of duplicated, mostly static html, to just some code to render the data, obtained via an XHR request to their very own APIs, into the page. As always, less is more.

We did a little chatting on this stuff this afternoon; I have more thoughts on how Twitter should fix itself. To be posted later. If you want part of the surprise ruined, Josh twittered after reading my mind.

Here's the program I used to test the HTTP cache validator headers: http-validator-test.py

	#!/usr/bin/env python

	#--------------------------------------------------------------------
	# do some ETag and Last-Modified tests on a url
	#--------------------------------------------------------------------

	import sys
	import httplib
	import urlparse

	#--------------------------------------------------------------------
	def sendRequest(host, path, header=None, value=None):
	    headers = {}

	    if (header):
	        print "Passing header: %s: %s" % (header, value)
	        headers[header] = value
	    else:
	        print "Passing no extra headers"

	    conn = httplib.HTTPConnection(host)
	    conn.request("GET", path, None, headers)
	    resp = conn.getresponse()

	    stat = resp.status
	    etag = resp.getheader("ETag")
	    lmod = resp.getheader("Last-Modified")
	    clen = resp.getheader("Content-Length")

	    print "%s %s; Content-Length: %s; Last-Modified: %s; ETag: %s" % (
	        resp.status, resp.reason, clen, lmod, etag
	        )
	    print

	    return resp

	#--------------------------------------------------------------------
	if (len(sys.argv) <= 1):
	    print "url expected as parameter"
	    sys.exit()

	x, host, path, x, x, x = urlparse.urlparse(sys.argv[1], "http")

	resp = sendRequest(host, path)
	etag = resp.getheader("ETag")
	date = resp.getheader("Last-Modified")

	resp = sendRequest(host, path, "If-None-Match", etag)
	resp = sendRequest(host, path, "If-Modified-Since", date)

Update - 2007/05/17

Duncan Cragg pointed out that I had been testing the Date header, instead of the Last-Modified header. Whoops, that was dumb. Thanks Duncan. Luckily, it didn't change the results of the tests (the status codes anyway). The program above, and the output of the program have been updated.

Duncan, btw, has a great series of articles on REST on his blog, titled "The REST Dialog".

In addition, I didn't reference the HTTP 1.1 spec, RFC 2616, for folks wanting to learn more about the mysteries of our essential protocol. It's available in multiple formats, here: http://www.faqs.org/rfcs/rfc2616.html.

modelled serialization

It's Only a Model!
© Brett Weinstein

Too many times I've seen programmers writing their web services, where they are generating the web service output 'by hand'. Worse, incoming structured input to the services (XML or JavaScript), parsed by hand into objects. Maybe not parsed, but DOMs and JSON structures walked. Manually. Egads! Folks, we're using computers! Let the computer do some work fer ya!

In my previous project, we used Eclipse's EMF to model the data we were sending and receiving via RESTy, POXy web services. For an example of what I'm referring to as 'modelling', see this EMF overview and scroll down to "Annotated Java". For us, to model our data, meant adding EMF comments to our code. And then running some code to generate the EMF goop. What they goop ended up giving you was a runtime version of this model you could introspect on. Basically just like Java introspection and reflection calls, to examine the shape of classes, and the state of objects, dynamically. Only with richer semantics. And frankly, just easier, if I remember correctly.

Anyhoo, for the web services were we writing, we constrained the data being passed over the wire to being modelled classes. Want to send some data from the server to the client? Describe it with a modelled class. Because the structure of these modelled classes was available at runtime, it was (relatively) easy to write code that could walk over the classes and generate a serialized version of the object (XML or JSON). Likewise, we could take an XML or JSON stream and turn it into an instance of a modelled class fairly easily. Generically. For all our modelled classes. With one piece of code.

Automagic serialization.

One simplification that helped was that we greatly constrained the types of 'features' (what EMF non-users would call attributes or properties) of a class; it turned out to be basically what's allowed in JSON objects: strings, numbers, booleans, arrays, and (recursively) objects. We had a few other primitive types, like dates and uuid, but amazingly, were we able to build a large complex system from a pretty barebones set of primitives and composites. Less is more.

For folks familiar with WS-*, none of this should come as a huge suprise. There are basically two approaches to defining your web service data: define it in XML schema, and have tooling generate code for you. Or define it in code, and have tooling generate schema for you. In both cases, serialization code will be generated for you. Neither of these resulted in a pleasing story to me. Defining documents in schema is not simple, certainly harder than defining Java classes. And the code generated from tooling to handle schema tends to be ... not pretty. On the other hand, when starting with code, your documents will be ugly - some folks don't care about that, but I do. The document is your contract. Why do you want your contract to be ugly?

Model driven serialization can be a nice alternative to these two approaches, assuming you're talking about building RESTy or POXy web services. Because it's relatively simple to create a serializer that feels right for you. And you know your data better than anyone; make your data models as simple or complex as you actually need. If you're using Java, and have complex needs, consider EMF, because it can probably do what you need, or at least provide a solid base for what you need.

Besides serialization, data modelling has other uses:

• Generating human-readable documentation of your web service data. You were planning on documenting it, right? And what, you were going to do it by hand?

• Generating machine-readable documentation of your web service data; ie, XML schema. I know you weren't going to write that by hand. Tell me you weren't going to write that by hand. Actually, admit it, you probably weren't going to generate XML schema at all.

• Generating editors, like EMF does. Only these editors would be generated in that 'junky' HTML/CSS/JavaScript trinity, for your web browser client needs. Who wants to write that goop?

• Writing wrappers for your web services for other languages. At least this helps with the data marshalling. Again, JavaScript is an obvious target language here.

• Generating database schema and access code, if you want to go hog wild.

 

If it's not obvious, I'm sold on this modelling stuff. At least lightweight versions thereof.

So I happened to be having a discussion with a colleague the other day about using software modelling to make life easier for people who need to serialize objects over the web. I don't think I was able to get my message across as well as I wanted, and we didn't have much time to chat anyway, so I thought I'd whip up a little sample. Code talks.

This sample is called ToyModSer - Toy Modelled Serializer. It's a toy because it only does a small amount of what you'd really want it to be able to do; but there's enough there to be able to see the value in the concept, and how light-weight you can go. I hope.

structured javascript

Structure
© Salem Eames

I just listened to the Jon Udell podcast interview with John Lam, which was quite interesting. Highly recommended. I do, of course, have a bone to pick.

At 9:52, the conversation turns to talk about JavaScript. John Lam says JavaScript is "a very difficult language for programming in the medium or the large and by medium and large I'm going to say applications which exceed 5,000 to 10,000 lines of code. Once my JavaScript gets up to that, I have a really hard time maintaining that stuff because modularity is definitely one of the things that isn't really all that well thought out in the JavaScript language. Which is much better in languages like Ruby and Python."

Now, John is certainly correct that JavaScript lacks language level modularity features like namespaces, packages, and class definitions. Typically, a 'class' is defined by creating a constructor function and adding methods to it by adding functions to the constructor's prototype. Packages and namespaces are defined as a top-level objects with the top-level package segment, with a field defined for the next package segment, and then recursing through the remainder of the package segments.

Class definition by running plain old code.

Some JavaScript libraries like YUI and dojo actually do have some code and conventions around defining such things.

So, note that. No language level modularity features like a 'package' and 'class' keyword. It's all dynamic. And perhaps some conventions provided by libraries you happen to be using.

Which is quite similiar to Smalltalk. There are no 'language level' features for defining classes. There is no 'class' keyword. Instead, to define a new class, I'd go into a class browser, and fill in a template like:

    Number subclass: #Fraction
        instanceVariableNames: 'numerator denominator'
        classVariableNames: ''
        poolDictionaries: ''	

This is a class definition. However, literally, it's a message send. A message sent to a class (Number) to create a subclass (Fraction) with two instance variables (numerator and denominator).

I don't recall anyone who ever bothered to learn Smalltalk having made claims that it wasn't modular. So I don't think having language level modularity features is a neccessity for making the language usage modular.

My reference to Smalltalk isn't entirely spurious given the recent news of Dan Ingall's Project Flair. As Tom Waits would 'sing' ... "What's he building in there?".

This leads me to a number of questions:

• Could we build a set of conventions around package/namespace/class/method definition that would be usable in a number of different contexts? IDEs, live class browsers, etc.

• What would it mean to build 'images' of code and data in JavaScript? By image I don't mean a .gif file, I mean a collected set of code and data serialized into one or more files. An 'executable unit' for a language interpreter.

• Could we get these things to work with the rather crude code injection apparatus we currently have for web applications (<script src=...>)?

• In the unspecified future, we'll have declarative language features in JavaScript. If you'd like to get a taste for what this might look like, right now, look no further than ActionScript 3 from Adobe. Do we want this? Do we need this?

• WWSD - What Would Self Do? Self is probably the best known prototype-based language. It would be interesting to go back and look at some of the Self stuff; it's been years for me.

intrinsic qualities

quality sky
© Bruce Turner

A couple of meta-programming links for you, from Ward and Kent, and a slightly related topic I recently ran into.

From Ward, a blog post titled 'Is "Agile" Too Easy?'. While Ward ends up talking about agile, what I found more interesting was the general concept of "the essence".

From Kent, a movie of a talk he gave titled "Ease at Work". I'm 100% certain I've never sat down and watched one hour straight of YouTube, till I watched this. When starting the last segment, I realized I should have watched it via the wii instead. Next time.

Both Ward and Kent are discussing some of the intrinsic qualities and mind-states of programmers. Interesting stuff. The kind of stuff we all know, but never really think about too much.

I ran into another one of these intrinsic qualities the other night when I attended the panel "Breaking Into the Game Industry" presented by the Triangle chapter of the IGDA. The triangle, as it turns out, is home to a large number of game companies. Three triangle-based game company execs were answering questions on how to get a job in the game industry. One meme that kept coming up, was that the companies were looking for people with "passion". They don't just want good technical people, they want good technical people who will love their job. Like the "essence", passion is something "we know when we see it", Gladwell-ian Blink style. Gaming companies have the luxury of making this an absolute requirement due to the large number of applicants they get to select from, compared to the rest of the software industry.

Regarding "essence" and "passion", these are obviously qualities we desire to have in our own jobs. But they also seemed like the distillation of the qualities we look for in potential new hires, in co-ops who may become new hires eventually, and future team-mates, whether you're adding to your team, or shopping for a new team to work with. Good, simple memes to keep in mind.

Kent's talk provides what I would consider some suggestions on maintaining the passion, or more pessimistically, dealing with passion-killing work habits, for those times when you don't happen to be on the 'genius' side of the roller coaster.

new shuffle

shuffle in the car
© Patrick Mueller

As a little enticement / reward for doing more exercise recently, I decided to upgrade my iPod. My kids wanted me to get one of the big boys; they wanted to see the videos and the games. Not like they would have ever had a chance to play with it or anything. :-) I opted instead for the shuffle because I really wanted something smaller.

What I'm having a bit of a problem getting around is the lack of podcast support in iTunes with the shuffle device. Namely, that it's missing the "Podcast" tab for the device in iTunes, where you'd select the podcasts you wanted auto-synced. This is important to me, because my primary use of my iPod is to listen to podcasts.

Instead, you have to do things manually. You know, drag and drop files in the iTunes UI. And select files to be deleted. Etc. That's certainly a step down in functionality than I had with my crusty old 3G iPod. I have no idea why the functionality is missing in the first place.

I currently do have an "Autofill" set up for a Smart Playlist I created called "Podcasts" that contains, well, you can guess. Autofill is some special function, presumably just for the shuffle, that picks songs randomly from your library or from a playlist to put on the shuffle. I can't quite tell how well this is working yet; I'm sure I'll have to go a few weeks to see if it makes life easier. At a minimum, having a Smart Playlist with just podcasts in it makes life somewhat easy itself, in terms of copy en masse to the device.

One advantage of the simplicity factor here is that it's quite easy to arrange audio files, on the device, in whatever order you wish. Instead of turning and clicking to pick the things I listen to (while driving), I can now arrange them in the order I want, sitting on the computer.

The size, simplicity, and wacky colors of the device are hard to beat. I especially love the big play/pause button right in the middle, and the integrated clip.

Java, please evolve

Crusty and Rusty
© Thomas Haverkamp

As the Java language continues to add dysfunctional function to it's libraries, and as it doesn't add stuff we need to it's libraries, and we bicker about licensing details of test cases, Microsoft is doing something interesting in the CLR that Java should have done years ago.

Sigh.

We need to seriously stir some stuff up in the Java space. A lot.

I see signs of hope. Today in the #jruby irc channel on irc.freenode.net, there was some heretical chatter about using grizzly to tie directly into JRuby on Rails, avoiding Servlet (~shiver~) altogether.

This is excellent thinking. We need more heresy like this.

BTW, Charles Nutter (of JRuby fame) is running an informal "dynamic languages on the JVM" session at JavaOne. Contact him for more details.

Java process management arghhs

Cross-processed rust
© Rob Brewer

Pretend like you'd like to write a Java program that wants to launch and lightly manage some operating system processes. Know how to do this? You'll want to use one of the flavors of the Runtime exec() methods. If you happen to be running on a J2SE 1.5 or greater JVM, you can use the new ProcessBuilder class, but there's seems to be not much benefit to doing so, and of course, your code won't run on a J2SE 1.4 or earlier JVM if you do. Both Runtime.exec() and ProcessBuilder end up returning an instance of the Process class, which models the launched operating processes (the child).

One of the things I learned the hard way with the Process class, many years ago, was that you really need to process the stdout and stderr output streams of the child process, because if you don't, and your child process writes more than a certain amount (operating system dependent) of data on those streams, it'll block on the write, and thus 'freeze'. So, you need to get the stdout and stderr streams via the (confusingly named) getInputStream() and (logically named) getErrorStream() methods of the Process class.

The simplest thing to do to handle this is to launch two threads, each reading from these input streams, to keep the pipes from getting clogged. Do whatever you need to do with the data being output by your child process; I'm sure you want to do something interesting with it.

I've not really had a chance to work with the java.nio package before, so I happened to think that this would be a good chance to play; let's see if we can get from two threads handling the child process's output, down to one, by using the class Selector, which would seem to be the moral equivalent of the *nix function select(), which can be used to determine the readiness of i/o operations of multiple handles at the same time.

Looking at Selector, you can tell right from the top of the doc, that this class deals with SelectableChannel objects. Now, you need to figure out how to get from an InputStream (returned by Process.get[Input|Error]Stream()) to a SelectableChannel. Except, you can't. From any InputStream, you can call Channels.newChannel() to get a ReadableByteChannel, but a ReadableByteChannel is not a SelectableChannel. (ReadableByteChannel is an interface, and SelectableChannel is a class.) This is not the end of the world; it may just be that the object returned by Channels.newChannel() is actually an instance of SelectableChannel (or a subclass thereof). Never know. So, here's a little experiment for an Eclipse scrapbook page:

    Process process = Runtime.getRuntime().exec(new String[] {"sleep", "10"});
    java.io.InputStream iStream = process.getInputStream();
    java.nio.channels.ReadableByteChannel channel = java.nio.channels.Channels.newChannel(iStream);
    System.out.println(channel.getClass().getName());
    System.out.println(channel instanceof java.nio.channels.SelectableChannel);

and the result is ...

    java.nio.channels.Channels$ReadableByteChannelImpl
    false	

Bad news; it's not a SelectableChannel, and would appear to be an instance of an inner class of the Channels class itself. Poking into inner class, that class is a subclass of java.nio.channels.spi.AbstractInteruptibleChannel. Not selectable at all. All this inner class business is for the default 1.5 JRE I use on my mac. Other implementations might well be different (and better), but that doesn't help me on the mac.

So, unfortunately, you can't reduce my two threads to process a child process's stdout and stderr down to one, using this select technique. This might not sound so bad, but what if you wanted to be able to handle a lot of processes? To handle N simultaneous processes, you'll need N * 2 threads to process all the stdout and stderr streams. If you could have used the select logic, you'd only need 1 thread.

Note that you could also try polling these streams, by calling available() on them, to determine if they have anything to read. Polling isn't very elegant, and is obviously going to be somewhat cpu intensive. But for me, I got burned on available() a long time ago, don't trust it, and never use it.

Bummer.

But wait, it gets better!

Another thing you're going to want to do with these child processes is to determine when they're done. There's two ways of doing this. You can call Process.exitValue() which returns the exit value of the process. Unless the process hasn't actually exited, in which case it throws an IllegalThreadStateException. The other way to determine when the process is done is to call Process.waitFor(); this method will block until the process has exited.

Neither of these is very nice. If you use waitFor(), you'll burn a thread while it blocks (now up to N * 3 threads per process!). If you use exitValue(), you can poll, but every check of the process, while it's not complete, is guaranteed to throw an exception, which is going to burn even more cpu.

Double bummer.

Embedded Gnome

Tiny Wee Gnome
© Wedge

The announcement last week of the GNOME Mobile & Embedded Initiative was pretty interesting. I hope this breathes some fresh air into the embedded space.

It's been a while since I worked in the embedded software development arena, but boy was it fun. I highly recommend, if you ever get the chance, of doing some software development in the embedded world, if all you've ever done is desktop and server software development. You'll never look at multi-megabyte jar files of classes in the same way.

I still have the first embedded board I got code running on sitting on my desk; a 403GC PowerPC board. 25 or 33Mhz, can't remember how much RAM I had on it, using whatever version of QNX was available at the time. I finangled an expansion board from one of the embedded hardware guys that had an 8 segment LED so that I could actually output something somewhere besides a telnet session. This would have been ~1997, so I had the obligatory live stock ticker running on it. In Smalltalk, of course.

I'm guessing that Apple's re-entrance into the embedded space with the iPhone has prompted this action, to some extent. Embedded devices are sexy again. I've even been using my old Palm Tungsten-C now and again, if I think I'm going to be out somewhere with free wifi, so I can Twitter.

And Mike Wilson pointed out that Opera is running on the Palm, and is interestingly enough implemented on the J9 Palm Java VM.

Down at the bottom of the GNOME Mobile & Embedded Initiative announcement page, under "Technologies Under Consideration", is listed "Java ME" (Micro Edition). Hmm. Wonder what they're thinking. My wish would be for CLDC, which is the smallest useful Java class library available for the J2ME space, and then add SWT for the UI. Not sure if the eSWT libraries have GTK bindings, but I'm sure that's possible. I certainly hope they don't select just MIDP, since it's so weird and low-functional, and certainly hope they don't select Personal Profile or Personal Basis as the rich GUI, because that's just a step into the past (AWT).

Glad to see things are 'opening up' a bit, in any case.

turtles all the way down

turtles all the way down
© David Morimoto

There's been a bit of controversy over Mike Pence's article "Heresy and turtles (all the way down) with Avi Bryant"; it's an interesting read, if obviously, and seemingly intentionally, controversial.

There is one bit that particular cries out to me, and that's the notion of Smalltalk being implemented as "turtles-all-the-way-down". Meaning, Smalltalk 'libraries', like collections, http client access, numbers (heh) are implemented in ... Smalltalk. With a small core of 'natives' that do the lowest-level stuff like opening files and writing to sockets. Alternatively, if you look at a language like PHP, you realize that none of the 'base' library is implemented in PHP, it's implemented in C. Notice that these two languages are at opposite ends of the spectrum here, compared to other languages like Ruby and Python, where there's more C code than in Smalltalk, but less than in PHP (relatively, compared to libraries implemented in the target language). Meaning, a lot of the languages we use, have a lot of their base library code implemented in C. And not just any old C, but C code wrapped in layers of language- and interpreter-specific wrappers. Meaning, I can't share that code between languages. Ruby can't use 'extension libraries' built for Python (and vice versa). It gets worse. JRuby can't use 'extension libraries' built for Ruby (and vice versa).

Something's clearly a bit out of whack here. It shouldn't be this hard. There's so much other work involved in getting new languages, or new implementations of existing languages up and going, not to mention making sure you have good editors, debuggers, etc. This is clearly one area that we can make easier for language implementors, isn't it?

In VisualAge Smalltalk, we had both a 'native' interface, with icky wrappers you had to use around your C code, but we also had a great little framework called PlatformFunction. A way to call out directly to C code. We also had a framework to allow you to deal with native pointers called OSObject. With these facilities, in the 5 or so years I was shoulder deep in Smalltalk, I think I only ever wrote one real 'native'; I was able to use call outs to C code for everything else I needed. And I used this a lot. We also had a way to generate a pointer to a C function that hooked into an arbitrary method, so you could call in to Smalltalk from C, for callback purposes.

I've seen other libraries in other languages do this as well; usually referred to as ffi - foreign function interface. The CLR provides for this as well, via P/Invoke. It's a nice capability that every language environment should provide.

The downside of this is that you probably can't do anything with 'objects' using these interfaces. You're dealing with low-level C goop. Sometimes you really want to be dealing with objects.

So here's an interesting thought exercise. Would it be possible to create a portable 'native' interface, like Java's JNI, which can deal with 'objects', that could be used by multiple languages? So that I could compile an extension library as a binary, native shared library that multiple languages could make use of. I call out JNI specifically both because I'm intimately familiar with it, and it's actually a pretty nice API. For a number of reasons:

• JNI defines all the 'functions' used to talk to the runtime that you can use from your natives, as function pointers in a struct that gets passed to your 'native' functions. Since they're function pointers, you don't need to link against a library; the functions are resolved at runtime, not link time. This means I can compile a Java native library once, and use it on any VM that supports the same level of JNI.

• JNI provides pretty nice encapsulation from the VM implementation. Instead of getting pointers to internal VM structures and mucking with stuff, you have function calls to do your work. Instead of dealing with garbage collection bits and pieces (like reference counts), you use APIs to create/destroy references that the underlying VM will use to deal with the garbage collection bits and pieces.

• JNI has been protected very well against version to version changes, so that you have a very good expectation that a native library you compiled for version 1.x of Java will work with version > 1.x of Java.

What would it mean to do something like this to support multiple languages? There are obviously too many semantic differences between all the different languages we use to hope to be able to have a complete, universal 'extension' interface, but there's also a lot of commonality with these languages. It's not a simple problem to solve. But it's time to start thinking about it. In fact, it seems a bit embarrassing that we've not yet started thinking about it.

setting up Mercurial on TextDrive with ssh

Mercury, the Winged Messenger
© Toshiyuki IMAI

I've been thinking about setting up a simpler, change-set based source code repository for my little personal projects for a while now. For the past six months or so I've been using SVN on my TextDrive account but ... looking for something different.

I ran across Mercurial (hg) at least twice now recently, so that speaks to me "check it out". Getting it set up on my local box was straightforward, and of course the next step is getting it set up on your server. I host at TextDrive. And Bill de hÓra recently posted some instructions on how to do just that. Kind of.

The problem with Bill's instructions are that you're setting it up to run using Basic Auth but using http instead of https. Bill points out himself, that "This isn't secure". Using https with TextDrive is possible, but at times confusing. Luckily, there is a fairly straight-forward way to get to a secure solution.

The basic idea is to use the ssh instead of http for when accessing your hg repositories programmatically. Instructions on using ssh with hg are available in the hg(1) man page.

In a nutshell, follow Bill's instructions, with a few slight changes, and it just works, assuming you have ssh set up in the first place.

If this doesn't work, ping me; I perhaps forgot a step.

I also created a ~.hgrc file on my client box, with the following contents. The ssh setting causes ssh to use compression when transfering the data/files.

    [ui]
    username = your name <yourname@yourmail>
    ssh = ssh -C

The only thing I don't like about the resulting set up is having to update the hgweb.config file on the server whenever you create a new repository. Though I suppose it's nice that you can selectively make repositories easily publicly accessible or not this way. I don't think I'd rely on this trick to hide something.

I copied Bill's instructions, and then augmented them with the changes I made, which you can find here. I would have included them in this blog entry, but Roller decided to strip all the nice formatting I did out my post. And that's just unacceptable, given the stupid amount of time I spent adding it in, in the first place.