Links

pmuellr is Patrick Mueller

other pmuellr thangs: home page, twitter, flickr, github

Wednesday, April 25, 2007

Java, please evolve

As the Java language continues to add dysfunctional function to it's libraries, and as it doesn't add stuff we need to it's libraries, and we bicker about licensing details of test cases, Microsoft is doing something interesting in the CLR that Java should have done years ago.

Sigh.

We need to seriously stir some stuff up in the Java space. A lot.

I see signs of hope. Today in the #jruby irc channel on irc.freenode.net, there was some heretical chatter about using grizzly to tie directly into JRuby on Rails, avoiding Servlet (~shiver~) altogether.

This is excellent thinking. We need more heresy like this.

BTW, Charles Nutter (of JRuby fame) is running an informal "dynamic languages on the JVM" session at JavaOne. Contact him for more details.

Tuesday, April 24, 2007

Java process management arghhs

Pretend like you'd like to write a Java program that wants to launch and lightly manage some operating system processes. Know how to do this? You'll want to use one of the flavors of the Runtime exec() methods. If you happen to be running on a J2SE 1.5 or greater JVM, you can use the new ProcessBuilder class, but there's seems to be not much benefit to doing so, and of course, your code won't run on a J2SE 1.4 or earlier JVM if you do. Both Runtime.exec() and ProcessBuilder end up returning an instance of the Process class, which models the launched operating processes (the child).

One of the things I learned the hard way with the Process class, many years ago, was that you really need to process the stdout and stderr output streams of the child process, because if you don't, and your child process writes more than a certain amount (operating system dependent) of data on those streams, it'll block on the write, and thus 'freeze'. So, you need to get the stdout and stderr streams via the (confusingly named) getInputStream() and (logically named) getErrorStream() methods of the Process class.

The simplest thing to do to handle this is to launch two threads, each reading from these input streams, to keep the pipes from getting clogged. Do whatever you need to do with the data being output by your child process; I'm sure you want to do something interesting with it.

I've not really had a chance to work with the java.nio package before, so I happened to think that this would be a good chance to play; let's see if we can get from two threads handling the child process's output, down to one, by using the class Selector, which would seem to be the moral equivalent of the *nix function select(), which can be used to determine the readiness of i/o operations of multiple handles at the same time.

Looking at Selector, you can tell right from the top of the doc, that this class deals with SelectableChannel objects. Now, you need to figure out how to get from an InputStream (returned by Process.get[Input|Error]Stream()) to a SelectableChannel. Except, you can't. From any InputStream, you can call Channels.newChannel() to get a ReadableByteChannel, but a ReadableByteChannel is not a SelectableChannel. (ReadableByteChannel is an interface, and SelectableChannel is a class.) This is not the end of the world; it may just be that the object returned by Channels.newChannel() is actually an instance of SelectableChannel (or a subclass thereof). Never know. So, here's a little experiment for an Eclipse scrapbook page:

    Process process = Runtime.getRuntime().exec(new String[] {"sleep", "10"});
    java.io.InputStream iStream = process.getInputStream();
    java.nio.channels.ReadableByteChannel channel = java.nio.channels.Channels.newChannel(iStream);
    System.out.println(channel.getClass().getName());
    System.out.println(channel instanceof java.nio.channels.SelectableChannel);

and the result is ...

    java.nio.channels.Channels$ReadableByteChannelImpl
    false	

Bad news; it's not a SelectableChannel, and would appear to be an instance of an inner class of the Channels class itself. Poking into inner class, that class is a subclass of java.nio.channels.spi.AbstractInteruptibleChannel. Not selectable at all. All this inner class business is for the default 1.5 JRE I use on my mac. Other implementations might well be different (and better), but that doesn't help me on the mac.

So, unfortunately, you can't reduce my two threads to process a child process's stdout and stderr down to one, using this select technique. This might not sound so bad, but what if you wanted to be able to handle a lot of processes? To handle N simultaneous processes, you'll need N * 2 threads to process all the stdout and stderr streams. If you could have used the select logic, you'd only need 1 thread.

Note that you could also try polling these streams, by calling available() on them, to determine if they have anything to read. Polling isn't very elegant, and is obviously going to be somewhat cpu intensive. But for me, I got burned on available() a long time ago, don't trust it, and never use it.

Bummer.

But wait, it gets better!

Another thing you're going to want to do with these child processes is to determine when they're done. There's two ways of doing this. You can call Process.exitValue() which returns the exit value of the process. Unless the process hasn't actually exited, in which case it throws an IllegalThreadStateException. The other way to determine when the process is done is to call Process.waitFor(); this method will block until the process has exited.

Neither of these is very nice. If you use waitFor(), you'll burn a thread while it blocks (now up to N * 3 threads per process!). If you use exitValue(), you can poll, but every check of the process, while it's not complete, is guaranteed to throw an exception, which is going to burn even more cpu.

Double bummer.

Saturday, April 21, 2007

Embedded Gnome

The announcement last week of the GNOME Mobile & Embedded Initiative was pretty interesting. I hope this breathes some fresh air into the embedded space.

It's been a while since I worked in the embedded software development arena, but boy was it fun. I highly recommend, if you ever get the chance, of doing some software development in the embedded world, if all you've ever done is desktop and server software development. You'll never look at multi-megabyte jar files of classes in the same way.

I still have the first embedded board I got code running on sitting on my desk; a 403GC PowerPC board. 25 or 33Mhz, can't remember how much RAM I had on it, using whatever version of QNX was available at the time. I finangled an expansion board from one of the embedded hardware guys that had an 8 segment LED so that I could actually output something somewhere besides a telnet session. This would have been ~1997, so I had the obligatory live stock ticker running on it. In Smalltalk, of course.

I'm guessing that Apple's re-entrance into the embedded space with the iPhone has prompted this action, to some extent. Embedded devices are sexy again. I've even been using my old Palm Tungsten-C now and again, if I think I'm going to be out somewhere with free wifi, so I can Twitter.

And Mike Wilson pointed out that Opera is running on the Palm, and is interestingly enough implemented on the J9 Palm Java VM.

Down at the bottom of the GNOME Mobile & Embedded Initiative announcement page, under "Technologies Under Consideration", is listed "Java ME" (Micro Edition). Hmm. Wonder what they're thinking. My wish would be for CLDC, which is the smallest useful Java class library available for the J2ME space, and then add SWT for the UI. Not sure if the eSWT libraries have GTK bindings, but I'm sure that's possible. I certainly hope they don't select just MIDP, since it's so weird and low-functional, and certainly hope they don't select Personal Profile or Personal Basis as the rich GUI, because that's just a step into the past (AWT).

Glad to see things are 'opening up' a bit, in any case.

Friday, April 20, 2007

turtles all the way down

There's been a bit of controversy over Mike Pence's article "Heresy and turtles (all the way down) with Avi Bryant"; it's an interesting read, if obviously, and seemingly intentionally, controversial.

There is one bit that particular cries out to me, and that's the notion of Smalltalk being implemented as "turtles-all-the-way-down". Meaning, Smalltalk 'libraries', like collections, http client access, numbers (heh) are implemented in ... Smalltalk. With a small core of 'natives' that do the lowest-level stuff like opening files and writing to sockets. Alternatively, if you look at a language like PHP, you realize that none of the 'base' library is implemented in PHP, it's implemented in C. Notice that these two languages are at opposite ends of the spectrum here, compared to other languages like Ruby and Python, where there's more C code than in Smalltalk, but less than in PHP (relatively, compared to libraries implemented in the target language). Meaning, a lot of the languages we use, have a lot of their base library code implemented in C. And not just any old C, but C code wrapped in layers of language- and interpreter-specific wrappers. Meaning, I can't share that code between languages. Ruby can't use 'extension libraries' built for Python (and vice versa). It gets worse. JRuby can't use 'extension libraries' built for Ruby (and vice versa).

Something's clearly a bit out of whack here. It shouldn't be this hard. There's so much other work involved in getting new languages, or new implementations of existing languages up and going, not to mention making sure you have good editors, debuggers, etc. This is clearly one area that we can make easier for language implementors, isn't it?

In VisualAge Smalltalk, we had both a 'native' interface, with icky wrappers you had to use around your C code, but we also had a great little framework called PlatformFunction. A way to call out directly to C code. We also had a framework to allow you to deal with native pointers called OSObject. With these facilities, in the 5 or so years I was shoulder deep in Smalltalk, I think I only ever wrote one real 'native'; I was able to use call outs to C code for everything else I needed. And I used this a lot. We also had a way to generate a pointer to a C function that hooked into an arbitrary method, so you could call in to Smalltalk from C, for callback purposes.

I've seen other libraries in other languages do this as well; usually referred to as ffi - foreign function interface. The CLR provides for this as well, via P/Invoke. It's a nice capability that every language environment should provide.

The downside of this is that you probably can't do anything with 'objects' using these interfaces. You're dealing with low-level C goop. Sometimes you really want to be dealing with objects.

So here's an interesting thought exercise. Would it be possible to create a portable 'native' interface, like Java's JNI, which can deal with 'objects', that could be used by multiple languages? So that I could compile an extension library as a binary, native shared library that multiple languages could make use of. I call out JNI specifically both because I'm intimately familiar with it, and it's actually a pretty nice API. For a number of reasons:

  • JNI defines all the 'functions' used to talk to the runtime that you can use from your natives, as function pointers in a struct that gets passed to your 'native' functions. Since they're function pointers, you don't need to link against a library; the functions are resolved at runtime, not link time. This means I can compile a Java native library once, and use it on any VM that supports the same level of JNI.

  • JNI provides pretty nice encapsulation from the VM implementation. Instead of getting pointers to internal VM structures and mucking with stuff, you have function calls to do your work. Instead of dealing with garbage collection bits and pieces (like reference counts), you use APIs to create/destroy references that the underlying VM will use to deal with the garbage collection bits and pieces.

  • JNI has been protected very well against version to version changes, so that you have a very good expectation that a native library you compiled for version 1.x of Java will work with version > 1.x of Java.

What would it mean to do something like this to support multiple languages? There are obviously too many semantic differences between all the different languages we use to hope to be able to have a complete, universal 'extension' interface, but there's also a lot of commonality with these languages. It's not a simple problem to solve. But it's time to start thinking about it. In fact, it seems a bit embarrassing that we've not yet started thinking about it.

Thursday, April 19, 2007

setting up Mercurial on TextDrive with ssh

I've been thinking about setting up a simpler, change-set based source code repository for my little personal projects for a while now. For the past six months or so I've been using SVN on my TextDrive account but ... looking for something different.

I ran across Mercurial (hg) at least twice now recently, so that speaks to me "check it out". Getting it set up on my local box was straightforward, and of course the next step is getting it set up on your server. I host at TextDrive. And Bill de hÓra recently posted some instructions on how to do just that. Kind of.

The problem with Bill's instructions are that you're setting it up to run using Basic Auth but using http instead of https. Bill points out himself, that "This isn't secure". Using https with TextDrive is possible, but at times confusing. Luckily, there is a fairly straight-forward way to get to a secure solution.

The basic idea is to use the ssh instead of http for when accessing your hg repositories programmatically. Instructions on using ssh with hg are available in the hg(1) man page.

In a nutshell, follow Bill's instructions, with a few slight changes, and it just works, assuming you have ssh set up in the first place.

If this doesn't work, ping me; I perhaps forgot a step.

I also created a ~.hgrc file on my client box, with the following contents. The ssh setting causes ssh to use compression when transfering the data/files.

    [ui]
    username = your name <yourname@yourmail>
    ssh = ssh -C

The only thing I don't like about the resulting set up is having to update the hgweb.config file on the server whenever you create a new repository. Though I suppose it's nice that you can selectively make repositories easily publicly accessible or not this way. I don't think I'd rely on this trick to hide something.

I copied Bill's instructions, and then augmented them with the changes I made, which you can find here. I would have included them in this blog entry, but Roller decided to strip all the nice formatting I did out my post. And that's just unacceptable, given the stupid amount of time I spent adding it in, in the first place.

Wednesday, April 11, 2007

twit-growl woops

I would be remiss in not pointing out a huge problem I recently found with my twit-growl program that I previously blogged about.

The problem is that the program is a fairly simple python script that uses some 'command line' programs to do all it's heavy lifting; curl and growl-notify. To invoke these, I was using os.system() to invoke the program, building up a big command-line as the parameter. As the programs take parameters which are coming from the data being downloaded, I was trying to be careful to escape the data when constructing the command lines.

Unfortunately, I'm an idiot, and didn't understand the escaping rules. In particular, I wasn't taking into account backquote substitution.

So, I was really suprised one day, when I got a message displayed in growl that was humongous; and the tail end of of the message was a wad of html. Going to the twitter site, to see the message there, I immediately realized what happened. The twitter message included some text like this:

... `curl http://www.yahoo.com`

Woops.

Got that fixed up pretty quickly, by avoiding the use of os.system() and using an os.spawn() variant instead, which (hopefully) avoids the shell completely.

BTW, I continue to use my twit-growl to get 'popups' for incoming tweets, and generally use the wonderful tweetbar Firefox extension for posting, and reviewing tweets en masse.

json schema

James Clark has a great post on XML and JSON. But I have one point to pick at.

"JSON's primitive datatype support is weak."

First, the number issue. If you assume JSON datatypes map into JavaScript ones, I believe the JavaScript number type does in fact have a very specific representation. I believe, a 64 bit floating point number. Even for integers. I'd love to point to a reference for this, but I don't have one handy, and the one at mozdev is insufficient.

Note that outside of JavaScript, for languages that treat integers and floating point as separate types, a parser can obviously distinguish between integers and floating points and 'do the right thing' in terms of mapping the JSON value into the appropriate type.

Next, "the set of primitive datatypes is not extensible". But this is true for XML itself. Until you mix in XML schema. It's schema that allows you to interpret a string in XML as some sort of other data type. JSON has no standard schema. Yet. It's easy to imagine though. Defined in JSON, of course.

json array exploit

Unless you've been living in a cave for the last week or so, you are familiar with JSON exploit as publicized by Fortify Software. I believe this is the same exploit I first read about on Robert Yates' blog. Original reference was from Joe Walker.

I'm not going to claim to be an expert on this particular exploit, but if I understand the situation correctly, I have a few issues with the current concerns.

  1. It's clearly not just JSON data which is suspect here. Anything which appears to be a valid (or maybe even somewhat valid) chunk of JavaScript, which someone could access via <script src=> is suspect. How many resources like this might be available at your site? How 'valid' (in terms of valid JavaScript) does it have to be? Only folks who are intimately familiar with the code of the JavaScript interpreters we use, can say for sure. Let's also not forget E4X; could your XML data be suspect?

  2. This appears to be a problem today just for Mozilla / Firefox. And I believe the reason is the advanced functionality it provides in JavaScript with coolio getter/setter capabilities. (BTW, ActionScript 3 also supports getter/setter capabilities). Is this actually a bug? Doesn't seem like it. Is there some chance this functionality could be locked down during script/src inclusion? Seems unlikely (slippery slope; what else would you lock down?). Could the enhanced functionality be turned off for user-land scripts? Probably; I don't suspect too many people are actually making use of the functionality. There's an open bug @ Mozilla on the issue that has additional thoughts.

  3. I'm clearly showing my n00biness here, but why aren't even simpler exploits possible, on all the browsers? Before doing a script/src, redefine a well-known object/function with one that does some hijacking? The security model for JavaScript is complex; so complex; too complex.

  4. Ted Husted pointed out that one way to fix this problem is to enclose your JSON in comment characters, and then strip the comments before parsing the JSON. That's great, unless I want to actually include comments inside my JSON. And assuming that JavaScript comments can't be 'recursive' (not specified at moz's js doc). Why would I want comments in my JSON? Why not? If you can have them in XML, I'm not sure why we won't someday want them in JSON. Note that I don't consider RFC 4627 to be the final statement on JSON syntax.

  5. The fact that JSON is also a syntactically valid chunk of JavaScript is smelling worse and worse all the time. I can't imagine a situation where I'd eval() JSON data in a production app; it's just so much safer to parse the data yourself. So, why don't we use this opportunity to come up with YASDN (yet another simple data notation)? JSON is great because it's readable, but I've already pointed out how we could make it even MORE readable.

  6. Isn't it about time we got a little better modularity available in JavaScript, beyond the script/src story. Which is more or less the same thing as building a C program in one big monolithic file that pulls in functionality with #include. So 1970's ...

Sunday, March 25, 2007

How Twitter is Different

Now that I've been using Twitter for a few days, I thought I'd note how it's different from other forms of communication I'm used to using. Comparing to things like irc, instant messaging (im), email, mailing lists, newsgroups, etc.

One very noticeable difference is in the 140 character limit on the messages. This is due to the fact that most people (I guess) are twittering via SMS on their phones, which has a limit of 140 characters per SMS message. Actually, is that 140 characters, or bytes? What's the character encoding story? hmmmm ...

This limit if far less than most other communication messages. Which forces various forms of conservatism, like using TinyURL for any URLs you want to 'send'.

Twitter is also, generally, not directed. You are talking to your 'friends' and/or 'followers' (the distinction has not stuck with me yet). In IRC terms, you could think of this as being a separate channel that you are sending your messages out on; but it's not neccessarily two-way. You might send a message that a 'follower' sees, but if you aren't set up to see their messages, you won't see them.

Twitter is not as intrusive as im or irc. With im, I can actually see who's online, and even perhaps how long they've been online or offline. I'm not sure if it was a bug or feature, but in a version of Sametime 7.5 that I used, it would actually popup a messaging window when someone started sending you a message, but before they actually sent the first one. It was fun to freak people out, when no one knew about this, by getting a message to them first. "Hey, I have this weird feeling that you want to ask me something". :-) But the fact is, I think im in general exposes a bit too much information.

Twitter is persistant. When I send a message, it's out there for everyone to eventually see, even if they aren't online to see it. Compare to irc, where if you aren't online (and don't have easy access to irc logs), you missed out on the conversation. Mailing lists have the same poor quality; it's easy to join a mailing list, but getting access to previous conversations can be painful. Hint: use gmane.

Twitter tends to be personal. "That was a delicious ham sandwich I just had!". It's easy enough to believe that this would make some folks non-interested in the product. However, the content which people are providing today is not really a telling indicator of how this might be useful in the future. For instance, my first foray into NNTP (or it might have been NetNews then), in the mid 80's, was rec.humor. Today, NNTP is an extremely valuable communication channel for non-personal, technical information.

Twitter has an insanely easy API. I was able to whip up a little 'reader' app for it in 1.5 hours. And that involved dusting off my python skills, figuring out how to talk to Growl, and dealing with cron. Compare with any of the other channels, and, it's night and day. I've looked at XMMP a few times, and it's not insanely easy. I've also written code to deal with many of the other channels, reading and writing, including email and nntp clients and servers, etc. Twitter is just plain simple in comparison to all of them.

I think the technology here is interesting. Perhaps as it gets integrated more into the fabric of our other communication channels, and daily-use tools, we'll find that it's a useful niche for certain styles of communication. I'm bullish on it.

Perhaps we can look at expanding this out a bit more. It would be nice to be able to support > 140 characters per message. It would be nice to be able to support sending hypertext of some kind. It would be nice to be able to support sending structured data. It would be nice to have more 'endpoints' than just twitter.com. And do we need to start thinking about spam?

Of particular interest to me, is the use of automated sending of messages from other programs, like continuous builds. And then the automated reception of messages. Jazz folks, take note.

Updates:

2007/04/02: fixed a broken link to Jazz.

ActionScript Typing

In the blog post, "The Open Web and Its Adversaries", Brendan Eich writes "If I were VP of Engineering in a 10,000 person company, I would want the security blanket of the C-like syntax and a static type system for a well-known, big-bell-curve language like AS3, C#, or Java." It's interesting that he's lumping ActionScript 3 with C# and Java as 'strictly typed' languages, since ActionScript is an evolution of JavaScript, which is not strictly typed.

It's true that the documentation for ActionScript 3 uses the 'typing' notation everywhere. Here's an example from one of the Hello World samples.

	package { 
		public class Greeter { 
			public function sayHello():String  { 
				var greeting:String; 
				greeting = "Hello World!"; 
				return greeting; 
			} 
		} 
	} 

Now compare with the 'untyped' version, in which I also took the liberty in removing the completely unneccesary semicolons. One of my pet-peeves with JavaScript: You almost never need to use semicolons in JavaScript, but everyone does!

	package { 
		public class Greeter { 
			public function sayHello()  { 
				var greeting 
				greeting = "Hello World!"
				return greeting
			} 
		} 
	} 

To prevent the compiler from complaining about the missing type information, use the mxmlc option -compiler.warn-no-type-decl=false when compiling. I tried playing with some options to try to get non-typing warnings flagged as errors, instead of warnings ... and I couldn't. The compiler would complain without the option specified above, but always produced the .swf file in the end.

This example doesn't really do justice to the impacts of typing things. But I think you can imagine how much more un-scannable your code is going to be with typing information littered throughout.

What's the rationale for doing all this typing? From the "Programming ActionScript 3.0" document: "In ActionScript 3.0, type information is preserved at run time, and used for a number of purposes. Flash Player 9 performs run-time type checking, improving the system's type safety. Type information is also used to represent variables in native machine representations, improving performance and reducing memory usage."

So typing helps with two things: 'improving type safety' and 'improving performance'.

We've beaten the type safety issue into the ground over the years; I won't touch that one. But performance? This sounds like premature optimization to me. As in premature optimization is the root of all evil.

May I suggest a comprise: use typing when it provides the convenient side-effect of documentation, and use typing when you need to optimize something. In terms of the documentation-style typing, I'm thinking the typing of 'fields' in an object, and the parameters and return types of methds. In fact, if you just did that, couldn't the compiler infer, to a large extent, the types of local variables?

Brendan also seems to be implying that corporations as a whole seem to prefer to deal with strictly typed languages, compared to 'looser' typed languages. Implying that ActionScript 3, and thus Flex, and thus Apollo, are all being aimed at corporations instead of a more general audience. Assuming this, and combined with the pricing model for tools, which I've already blogged about and ... yeah, you could see this as being marketed at corporations.

I'm constantly reminded of the programming model for the Palm Pilot, since I use mine daily (but only for reading on my walks, using the fantastic iSilo). One of the reasons for the Palm's popularity was that approachable tools were made available to the general public, which allowed for a flood of software programs to be released for that platform.

If you want wild success for your programmable product, make sure your target hacking audience is as wide as you can get.