modjewel 0.2.0 - a CommonJS require() function for the browser

I decided to get off my butt and bring my "for the browser" CommonJS require() implementation at http://github.com/pmuellr/modjewel into the 20-teens.

some background

Just over a year ago the "ServerJS" group was formed based on the "What Server Side JavaScript needs" blog post from Kevin Dangoor. This effort was renamed "CommonJS" later in the year, and lives on the web here: http://commonjs.org/. Kris Kowal has a nice write up on CommonJS at Ars Technica in the article "CommonJS effort sets JavaScript on path for world domination", if you don't know anything about it. Note: I'm a sucker for software-based world domination.

One of the first items looked at by the CommonJS community was a "module" system, centered around a function named require(). More information on modules and require() is available here: http://wiki.commonjs.org/wiki/Modules/1.1. While most of the folks in the group were playing with server-side JavaScript, I was interested in the browser, so cooked up a crude version of require() in a github project just over a year ago; the crufty old hack is here.

I figured someone else would do a proper version, but haven't really come across anything besides David Flanagan's efforts, documented in his blog posts "CommonJS Modules implementation" and "A module loader with simple dependency management". I have to admit I don't watch the CommonJS stuff all that closely, I probably missed some other implementations.

David's code doesn't have any kind of a license associated with it, so I didn't really feel it was appropriate to use it for a real project. And there were some things I wanted to modify. So, at the very least David shamed me into sprucing up modjewel and I will admit to stealing bits of his design as well.

One thing I learned from David's blog post was that compliance tests exist. Loves having tests. They currently live here: http://github.com/kriskowal/commonjs. Note that David's blog post points to an older version of the tests.

Onto the goodies in the newly improved modjewel ...

nice ui for the compliance tests

As I mentioned above, there's a set of test cases for the require() function. Problem is, they're built for command-line invocation; remember, this CommonJS stuff is more server-side than client-side JavaScript. I was recently pawing through the source for the SunSpider JavaScript benchmarks, and realized I could use the same "run the tests serially in IFrames" technique here, so I lifted that design. I now have a single page that will load and run all the tests serially, and produce a green/red report card when done. It's not perfect, but it's very handy.

One weird thing to note was that I couldn't find any kind of license on the tests; I ended up importing them directly into my project, which is kind of silly. I should probably run them right out of the github repo.

support for module.setExports()

The existing facilities with modules make it easy to provide access to a number of objects within a particular module. However, for those times you only have a single object in a module, things get verbose. Here's an example: say you want to build and use a module called WebDav (so the file is named WebDav.js) which exports a single function, WebDav.

In the file WebDav.js, you would define the WebDav class as:

   exports.WebDav = function WebDav() {
       // the function is 'named' for debug purposes
   ...
   }

You can then use the WebDav class in another file as:

   var WebDav = require("WebDav").WebDav

All the RY of Java. The "WebDav" passed to require() specifies the module, and the next WebDav selects the function as a property of the exports object the module populated.

Alternatively, with the module.setExports() support you can define the class in WebDav.js as:

   module.setExports(function WebDav() {
   ...
   })

and then use it in another file as:

   var WebDav = require("WebDav")

Better! Very convenient if you have a 1-1 correspondence between classes and modules.

module.setExports() isn't official yet; see the wiki page for more information: http://wiki.commonjs.org/wiki/Modules/SetExports.

preloading modules via <script src="">

The way modjewel's version of require() works is like every other browser-based one: it use synchronous XHR. Which is ok for development-time, but completely unacceptable at deployment-time. So they say. Flanagan came up a preload story for his require2.js file, but the mechanism seemed a little crude.

Instead, I created a command-line utility which will convert a module into a file which can be embedded with <script src="">. So then the story would be that you run this utility as part of a build step, perhaps minize or whatever you kids do to your JavaScript afterwards. In the end, you have a "plain old" JavaScript file to "include". All the utility really does is wrap your code in a function. When this code is executed, it registers the module and the function, such that the first time require() is called to load the module, the function is executed and the module is loaded.

A nice aspect of this is that the order in which the <script src=""> are placed doesn't need to be in some kind of special pre-req order. Including the modules doesn't really load them, it just pre-loads them.

Anyway, it's one answer to the "you can't use synchronous XHR!" argument.

portability

I've run the tests successfully on Mac Chrome, Mac Safari, Mac Opera 10, the IPhone Simulator, Windows Chrome, and Windows FireFox 3.0. IE 7.0 was a no-go.

There is an issue with Chrome, where some tests fail when running against a file:// URL. See issue 1 for more info.

There are some jiggly issues with Windows - of course those being path separator issues (\ instead of /) and new-line issues (CR LF instead of LF). Should be easy to fix. Not sure what to do about IE 7, do I use FireBug Lite to debug that?

remote web inspector - baby steps

Last week, I did a little work getting a remote debug story between an iPhone and WebKit's Web Inspector limping. Barely limping. More like crawling. Anyhoo, the write-up is here:

http://muellerware.org/papers/remote-web-inspector-baby-steps.html

who called that function?

A question came up today in Web Inspector land - someone wanted to know why setTimeout() was being called so often in some web application. The suggestion was that a capability could be added to Web Inspector's timeline panel to show a function's invocations (setTimeout() in this case) as well as the stack traces for those calls, to figure out who was actually calling the function.

My immediate thought was - can't you do this with user-land code? If you can do it with plain old JavaScript, then I see less of a need to add the function to the debugger. Especially since:

• The timeline already collects a LOT of information, keeping track of stack traces just adds more. The UI will get even busier, and Web Inspector will eat even more memory.

• Some new gestures would be required to manage the list of functions that you wanted to see on the timeline, and whether you wanted the stack traces or not (if optional). Note that it's not possible to use the existing breakpoint gestures for this, as you can't set a breakpoint on built-in function (like setTimeout()).

I wasn't sure you could do this with user-land code, so I felt the need to figure out if you could or not. I got close, maybe close enough, and the result is up at GitHub as a gist with a wrapWithBackTrace() function. The idea is that you pass the function you want traced and back traces displayed for, into the wrapWithBackTrace() function, which returns a new function to use in place of the original. Invoking that function will dump information on the console, including a back trace of functions that called this function. I think there are enough comments in the code to figure out what's going on, but here are a few caveats:

• The existing code makes use of Array's new methods indexOf(), map(), and forEach(). How nice to have these! If you don't have them, there's a link in the code to a site to get portable versions of them. Note that all the browsers I tested with (listed below) seemed to have them already. Wonderful!

• The whole functionality revolves around the bizarre caller property of Function instances. Look it up in your favorite JavaScript reference. Supposedly this property will disappear in future implementations of JavaScript engines, and is already not available in some (IE, I think).

Some sample code that uses the wrapWithBackTrace() function is available in the same gist. Here's a bit of it:

function legend(message) {
    var dashes = new Array(60).join("-")
    console.log(dashes)
    console.log(message)
    console.log(dashes)
    console.log("")
}

// tests with user-land function

function factorial(n) {
    if (n <= 0) return 1
    return n * factorial(n-1)
}

function printFactorial(n) {
    console.log(n + "! == " + factorial(n))
    console.log("")
}

function a() { b() }
function b() { c() }
function c() { 
    printFactorial(0)
    printFactorial(5)
}

legend("calling factorial() before wrapping")
a()

// install the replacement function
factorial = wrapWithBackTrace(factorial)

legend("calling factorial() after wrapping")
a()

legend("calling factorial() at top level")
factorial(0)    

This code exercises some of the edge cases for the function - recursive functions, and functions run at the top level of a script (no back trace available). The output of running the code above, in the console, is:

-----------------------------------------------------------
calling factorial() before wrapping
-----------------------------------------------------------

0! == 1

5! == 120

-----------------------------------------------------------
calling factorial() after wrapping
-----------------------------------------------------------

backtrace for factorial()
   - printFactorial()
   - c()
   - b()
   - a()

0! == 1

backtrace for factorial()
   - factorial()
   - wrappedWithBackTrace()
   - ... recursion on factorial()

(... repeated invocations elided ...)

backtrace for factorial()
   - factorial()
   - wrappedWithBackTrace()
   - ... recursion on factorial()

backtrace for factorial()
   - printFactorial()
   - c()
   - b()
   - a()

5! == 120

-----------------------------------------------------------
calling factorial() at top level
-----------------------------------------------------------

backtrace for factorial()
   - {top level call}

Sample code is also provided that wraps setTimeout(), which worked the same way. Just wanted to make sure it worked with built-in functions as well as user-land functions.

The code works in the latest WebKit nightly, Safari 4.0.4, Firefox 3.5.5, Chrome 4.0.249.12, and Opera 10.10, all on Mac OS X 10.6. Note that Opera does not have console.log(), so I gave it one:

console = {log: opera.postError}

The code also works node.js. The node system also doesn't have a console.log() function, so I gave it one:

console = {log: require('sys').puts}

Actually, there's one tiny little WIBNI with node that shows up in the output - can you spot it? (you'll need to install node and try it yourself)

In the end, I kinda doubt the usefulness of this - the problem being that so much of JavaScript these days is anonymous functions. Without being able to get the "file name" and line number a function is defined on, and without a name for the function itself, having your backtrace be a list of {anonymous} is less that helpful. Seems like it would be handy to have a getStackTrace() method on Arguments to give you this information. This still wouldn't help with eval()'d code though, as it doesn't even have a "file name" associated with it.

evil eval

With all the nascent alt-JS things people are playing with, talked about on this InfoQ blog post, I got to thinking a while back how we could make debugging a little better for these folks in WebKit's Web Inspector.

With my familiarity of Java's innards, I happened to think about supporting a mode of using an alternative source file, and line number mappings between the original source and the generated source, to allow folks to have some level of source-level debugging in their original language. That would be for systems which actually generate JavaScript from the original language.

Java supports meta-data in it's .class files for this sort of information, and I've seen it used for things like debugging Groovy in Eclipse using the stock Java debugger. Not a perfect story, but useful.

So I opened up Bug 30933 - Web Inspector: support for debugging alt-JS files which generate JS. Feel free to drop ideas there.

Anyhoo ...

While chatting with someone who could make use of this, they noted that their biggest problem was that eval() doesn't do a very good job of providing information about syntax errors in the eval()'d code. Really? Time for some experiments.

First, why care about eval() in the first place? Since it's evil. If code is being generated from some alternate source, don't you end up with JavaScript code you can just use in the browser with a <script> element? Sure, you can do it that way. But why do an off-line compile if you could do the compile in the browser?

It's also the case that some 'traditional' JavaScript libraries like Dojo may end up (depending on your configuration), downloading your source via XHR and then running it via eval() instead of using <script> elements.

Getting good diagnostic information from eval() would be good for lots of people.

On to the experiment:

Here's the source of an HTML file to try out with different browsers. There's a link in the HTML which, when clicked, will run the clickMe() JavaScript function. That function builds a string which will be passed into eval(). I added 100 or so newline's in the source before a string that all the browsers would actually choke on (took me a minute; my first guesses at things that would cause syntax errors didn't). The 100 empty lines are to see what kind of line numbers JavaScript will tell us about, if at all; if line numbers with syntax errors are reported, we want to see a line number like 100, not 13 (the line in the source that invokes eval()).

The page is also available on my web site.

Below the source are the results you see in the browser after clicking; these are the properties of the exception object generated by eval(), one per line.

HTML source for test:

     1  <html>
     2  <head>
     3  </head>
     4  <body>
     5  <a href="javascript:clickMe();">click me!</a>
     6  <p>
     7  <div id="output" style="font-family: monospace"></div>
     8  <script>
     9  function clickMe() {
    10      var lines99 = new Array(100).join("\n");
    11      evalString = lines99 + "for ]](x does not compute)]]\n"; 
    12      try {
    13          eval(evalString); // this is line 13 of the source file
    14      }
    15      catch(e) {
    16          dumpException(e);
    17      }
    18  }
    19  
    20  function dumpException(obj) {
    21      var result = "Exception:<br>\n";
    22      for (var key in obj) {
    23          result += "   " + key + ": " + obj[key] + "<br>\n";
    24      }
    25      document.getElementById("output").innerHTML = result;
    26  }
    27  </script>
    28  </body>
    29  </html>

---

WebKit nightly r50423 on Mac (basically, Safari):

Exception:
   message: Parse error
   line: 100
   sourceId: 4837249224
   name: SyntaxError

---

Opera 10.00 Build 6652 on Mac:

Exception:
   message: Statement on line 6: Syntax error stacktrace: n/a; see opera:config#UserPrefs|Exceptions Have Stacktrace
   opera#sourceloc: 6
   stacktrace: false

---

Google Chrome 4.0.223.11 on Mac:

Exception:
   message: Unexpected token ]
   stack: SyntaxError: Unexpected token ]
    at clickMe (http://muellerware.org/exception-in-eval.html:12:8)
    at unknown source
   type: unexpected_token
   arguments: ]
   name: SyntaxError

---

Firefox 3.5.4 on Mac:

Exception:
   message: missing ( after for
   fileName: http://muellerware.org/exception-in-eval.html
   lineNumber: 112
   stack: eval("[100 \n's elided]for ]](x does not compute)]]\n")@:0 clickMe()@http://muellerware.org/exception-in-eval.html:13 @javascript:clickMe();:1
   name: SyntaxError

---

IE 8.0.7100.0 on Windows 7 RC:

Exception:
   name: SyntaxError
   message: Expected '('
   number: -2146827283
   description: Expected '('

---

Google Chrome 3.0.195.27 on Windows 7 RC:

Exception:
   message: Unexpected token ]
   stack: SyntaxError: Unexpected token ] at clickMe (http://muellerware.org/exception-in-eval.html:13:8) at unknown source
   type: unexpected_token
   arguments: ]
   name: SyntaxError    

---

Yikes, not a pretty picture! WebKit was the only one that got the line number I wanted. But it gave a pretty crappy "message". Nice seeing the "stack" entries for some of the implementations.

To be honest, things like the "message" - the reason why you got the syntax error - I expect to be somewhat implementation dependent. Although I expect something (looking at you, JavaScriptCore!).

There's a question of semantics, of course. The exception generated from WebKit, with the line number of the eval()'d source, would be confusing had I not caught the exception (though perhaps I'd see something different in the debugger had I not caught the exception). In reality, what you may really want to see, structurally, is something like Java's nestable InvocationTargetException, for eval(), because there are multiple levels of things going on here; for the ultimate evil, eval() may end up doing some of it's own eval()'s.

In any case, it would be nice to see some of this stuff standardized. In ECMAScript 6, I guess.

Dropbox HTML experiments

Dropbox. Have you heard of it? If not, a colleague Jason Wagner recently wrote "Have You Heard Of Dropbox?", which is a good introduction with gushing praise. Steve O'Grady also recently wrote "There's More to Dropbox Than Piracy", which is a good read.

OK. So I installed it. On my laptop and my iPod Touch. Tried a few files. Nice.

The iPod Touch / iPhone version of the app does all it's own rendering of the shared files as near as I can tell - that is, it renders them in the Dropbox app, rather than launching another app to view/edit the shared files. So I thought - does it handle HTML files? Let's try.

---

experiment 1 - simple HTML page

For the first experiment, I created an HTML file with some embedded styling and JavaScript.

Here's the HTML:

<div style="font-size:300%; font-face:Optima; text-align:center;">
<p>experiment-1:

<p><a href="javascript:alert('it works')">click me</a>
</div>

Note that I created and subsequently edited this file right from the Dropbox folder on my laptop.

Trying it on my iPod - it works! Here's the screen after pressing the "click me" link, showing the alert:

Next, I dropped my network connection, and tried again.

Woops. Turns out that the Dropbox iPhone app only downloads a file for offline viewing when you mark the file as a 'Favorite'. OK, easy to fix. Connect to the network, view the page, mark it as a Favorite, disconnect, try again - it works!

BTW, what an ugly "fail whale"!

---

experiment 2 - external JavaScript, CSS and image

That worked out ok, so let's try something a little more challenging. Basically the same test, only move the styling to a separate CSS file, move the JavaScript to a separate file, and reference a shared image.

Here's the HTML:

<link rel="stylesheet" href="experiment-2.css">
<script src="experiment-2.js"></script>

<div>
<p>experiment-2:

<p><a href="javascript:clickHandler()">click me</a>

<p><img src="experiment-2.jpg">

<p><i>"kittens"</i> from
<a href="http://www.flickr.com/photos/mathias-erhart/2562106102/">
http://www.flickr.com/photos/mathias-erhart/2562106102/
</a>
</div>

Here's the JavaScript:

function clickHandler() {
    alert('it works');
}

Here's the CSS:

div {
    font-size:  300%;
    font-face:  Optima;
    text-align: center;
}

And here's what I get when I view the HTML file:

Woops. None of my referenced files were loaded. On a hunch, I decided to "Favorite" all the files. After doing that, it works!

You can just make out those adorable kittens behind the alert dialog. You'll have to load the page yourself to see them in all their adorable-ness.

---

experiment 3 - using localStorage

OK, time to get serious. Does this thing support localStorage?

Here's the HTML:

<div style="font-size:300%; font-face:Optima; text-align:center;">
<p>experiment-3:

<p>This page has been viewed by you <span id="times">???</span> times.

<div id="message" style="color:red" ></div>    

</div

<script>
(function() {

var timesSpan  = document.getElementById("times");
var messageDiv = document.getElementById("message");

function log(message) {
    messageDiv.innerHTML += message + "<br>";
}

if (!this.localStorage) {
    log("localStorage is not available!");
    return;
}

var times = localStorage.getItem("times-run");
if (!times) times = "0";
times = parseInt(times);
times = times + 1;
timesSpan.innerHTML = times;
localStorage.setItem("times-run", "" + times);

})();
</script>

And here's the page when loaded, then reloaded a few times. It works!

---

Just a few experiments, whetting my appetite for more. I assume it must support SQL storage as well. What about app-cache? XHR to another shared file?

Where am I going with this? Well, seems to me like if could create a little "application" as a single HTML file - embedded style, JavaScript, data: URLs for images, etc - then you have a nice little way of deploying your application. Just leave it in your Dropbox folder, edit away, test on your device, rinse and repeat. Editing such a wad would be a bit painful. You could write the application as pieces, like in experiment-2, then use a pre-processor to stitch the bits together and copy to your Dropbox (or provide a link to it).

Seems like there are many aspects of deploying a little app in this way which would be much nicer than how you have to do this using Mobile Safari. Using Dropbox would be a bit of a low-fi experience in comparison, but might be easier to write and update.

All three experiments are available in my public Dropbox folder, which you can access via your browser:

• experiment-1.html

• experiment-2.html

• experiment-3.html

JavaScript tracing

Usually when I'm trying to diagnose a problem with some piece of code, I'll reach for a debugger. A debugger is a great tool to use if you want to watch your code as it's executing, to see where it goes, what it touches, etc. But sometimes, using a debugger is inappropriate. For example, timing problems in your code can be difficult-to-impossible to diagnose with a debugger, because the act of debugging changes the timing (the observer effect).

In these cases, I reach for a tracer. What's a tracer? A tracer is a debugger, of sorts, that shows you what functions are being executed, and produces some kind of hierarchical log of the function execution when you are "done". Here's an example of a tracer log:

ElementsPanel.reset() ➊
   ElementsPanel.rootDOMNode() ➋
      ElementsTreeOutline.rootDOMNode() ➌
   ElementsPanel.focusedDOMNode() ➍
      ElementsTreeOutline.focusedDOMNode()
         ElementsTreeOutline.revealAndSelectNode()
   ElementsPanel.searchCanceled()
Panel.searchCanceled()

In this example, the function ElementsPanel.reset() ➊ calls the function ElementsPanel.rootDOMNode() ➋ which calls ElementsTreeOutline.rootDOMNode() ➌. After the call to ElementsTreeOutline.rootDOMNode() returns, ElementsPanel.rootDOMNode() then calls ElementsPanel.focusedDOMNode() ➍. And so on.

You could instrument your code by hand, using printf() or whatever, but that's tedious. Let the computer do the work for you.

In a previous life, as a Java developer, I've written at least two Java tracers using the Java^TM Debug Interface (JDI). Once you figure out how to use JDI, it's straight-forward to write single-purpose debugging clients like a tracer.

In my current life, I'm a JavaScript developer. There is nothing like the JDI for JavaScript. Yet. Bummer. But all hope is not lost. Turns out it's fairly straight-forward to write a tracer for JavaScript "classes". Here's one: function-tracer.js. And here's how I used it in a real-life situation.

I'm currently looking at a bug which smelled to me like it had a timing issue. I knew the suspect classes, so I just traced all of them, using the following bit of code added to my HTML:

<script src="function-tracer.js"></script>

<script>
FunctionTracer.logFunction = function(message) { WebInspector.log(message); };

FunctionTracer.traceFunctionsInObject(WebInspector.View.prototype, "View.");
FunctionTracer.traceFunctionsInObject(WebInspector.Panel.prototype, "Panel.");
FunctionTracer.traceFunctionsInObject(WebInspector.ElementsTreeOutline.prototype, "ElementsTreeOutline.");
FunctionTracer.traceFunctionsInObject(WebInspector.ElementsTreeElement.prototype, "ElementsTreeElement.");
FunctionTracer.traceFunctionsInObject(WebInspector.ElementsPanel.prototype,       "ElementsPanel.");
</script>

What's going on here:

• Add the function-tracer code via the <script src=> element.

• Add a custom logging function which writes to a logger that my application uses (WebInspector.log)

• Instrument the suspect classes (WebInspector.View, etc)

"Instrument"ing a class means that when an "instance" method in the class runs (a function in the class's prototype), it will get logged by my logger. The tracing output sample above was copied straight from my logger. I'll let you figure out how it actually works.

The code I posted handles new age ECMAScript 5 getter/setter accessors, which the code I'm debugging is using. I think I've provided enough checking in the code that it will work if you're running on some crusty old ECMAScript 3-ish engine.

In the end, I believe I found the issue with the bug in question, by comparing a "successful" run of the code with a "failure" run. But what's really the most fun thing of all with a tracer, is to see how bloaty your code is. In this case, I not only found a problem in the code, I could see that it was re-running some of the same code, over and over, probably for no good reason. But that's another bug.

web beep()

I've blogged before about making use of audible feedback for debugging. One of the things I'd like to eventually get around to doing for WebKit's Web Inspector is to add first class audible breakpoints. Just like you can "click" on a line to add a breakpoint where the debugger will stop the next time that location is reached in your program, an audible breakpoint wouldn't stop, but play a sound clip instead.

There's some infrastructure that needs to be added to Web Inspector before tackling that feature, so I figured I'd go on the cheap instead. Using HTML5's new audio element, I created a beep() function which just plays a sound clip when invoked. Wherever you might put a console.log() invocation, you can put a beep() invocation instead. Actually, I created a boop() function also; a single sound is rather boring.

Of course, having to modify your source for debugging is so 1980's. So my beep() functions can also act as wrappers. Pass them a function, and they'll return a new function you can use as a replacement of the original function. When that new function is invoked, it beeps (or boops, or whatever), then invokes the original function. The new function also contains a reference to the old function, so you can unwrap as needed. Using this, you can augment addressable functions from within your REPL-y JavaScript debugger console, without having to muck with your source.

The sample code is up here: web-beep. Seems to run fine in Safari and FireFox 3.5.

A Survey of Data Display in JavaScript Debuggers

My mom has been asking me about the differences in the way data is displayed in some JavaScript debuggers, so I wrote something up for her.

offline web application cache abuse

I've been a user of hand-held computers since 1997, when IBM bought me a Palm Pilot to use in a customer demo I was putting together. Since then, I've been through a number of Palm devices, and up till last week had been using a Nokia N800 device. This week, I caved, and bought myself an iPod Touch.

One of the main uses I've had for these devices is reading. On the Palm, I used the wonderful iSilo program to convert HTML to some binary format that the Palm reader rendered as good as it could - good enough - even on a 160x160 pixel display. On the N800, HTML content could just be copied to an SD disk, and then you could view it via the built-in web browser using file: URLs.

It turns out that I've been able to find plenty of material over the years, in HTML, that renders well enough on my hand-held devices. Either ready made, or built via scraping, or whatever.

what to use on an iPhone or iPod Touch

So, what are the options for reading on the iPhone/iPod Touch? Stanza is the only generic reader I've looked at so far, and it's not bad. It's particularly nice to have access to FeedBooks and Project Gutenberg books, which can be downloaded and then read later if you aren't online. Compared to Google Books which appears to only support reading while online. There's also Kindle, but I don't have any need to buy books, there's plenty of free content to be had. More importantly with Kindle is the 1995 level of HTML support, and that's a big problem for me; I like to read technical content with code samples, etc.

Beyond generic readers, there are also content-specific readers from content producers like the New York Times, the BBC, and AP News. The BBC one is especially strange. When displaying a full article, they seem to be displaying the same content that's on their web page. Various navigation and other links eating up space in the left and right side of the page. You can double-tap the middle content to get that to zoom, but why wouldn't they do that in the first place? If you actually happen to click a link on the page, even to another story on the BBC, it exits their app, and launches a browser on that page. Amazing, and not in a good sense.

Stanza seems like the most useful reader at present, at least for literature. I'm not really happy about the page-turning metaphor; I prefer to scroll pages vertically. I can probably learn to live with it.

But how can I get my own content on there, or more generally, rando HTML content on there?

EPUB

If you look at the formats of "documents" that Stanza supports, one particular format is EPUB. I've not been able to make my way through the morass that is the specifications surrounding this format, but if you're interested, here's how to get started: an EPUB file is a .zip file, expand with your favorite zip utility; you'll find a number of XML files inside, you can kinda get a clue for how everything fits together by browsing those files.

The "meat" of an EPUB document consists of XHTML, CSS, and image files. Hmmm. Sounds like the web. Could you take some existing web site and easily "EPUB" it?

HTML5 Offline Application Cache

Turns out, there's something in HTML5, and more importantly, available on iPhone and iPod Touch devices, that can take an existing web site and make it available even if you aren't connected to a network. There is a W3C Working Group Note for this available as"Offline Web Applications". Apple has documentation on their support of this in the document "HTML 5 Offline Application Cache". See also the WHAT-WG version under development.

The basic idea is to do the least possible work. You need to create a new "manifest" file which lists all the files which should be cached for offline usage. And you need to identify that file in the <html> element of your document. That's all! The browser will arrange to cache all the content listed in the manifest.

I decided to try out how well this works with Mark Pilgrim's "Dive Into Python" book. I downloaded a copy of the book as separate HTML files, ran find on the result to create the manifest, whacked the <html> elements with a multi-file search-and-replace in my text editor, set up the files on my machine's local server, browsed to it from my iPod Touch, worked like a champ.

It seems to take 20-30 seconds to get all the files downloaded, and during that time there is no indication of what's going on in the browser. I was tail'ing my server's access log to tell when I was done. Presumably some of those events specified in the WHAT-WG version of the spec will help out with issues like that. After the files got downloaded, I could turn the wifi off on my device, and continue to browse the content.

"Dive Into Python" seemed like a good test case; non-trivial markup, especially code snippets; a significant amount of content (3MBs fully expanded), and presumably not really designed to be read on a hand-held device. I'm satisfied with the results, though I think most people will find the font sizes too small. The font sizes are too small for me when displaying in portrait mode, so make sure you try landscape as well. Tough decision point there, make the fonts bigger and then you'll either get text wrapping or have to do side-to-side scrolling.

I've got the files available on one of my servers, http://diveintopython-cached.muellerware.org/, which I'll leave up until my hosting provider kills me - each access to the page from an enabled browser will download the whole book (if it's not already been downloaded). Presumably this might work on other WebKit-based browsers, like the one on an Android device, or the Palm Pre.

On the iPhone or iPod Touch, after displaying the initial page, add a bookmark by clicking the + button, and then the "Add to Home Screen" button. Remember to wait 30 seconds or so after the initial page load before doing this, to make sure all the content gets downloaded. This will add the book as a new "app" on the main application screen. I've also done the appropriate horkiness to get the cover of Mark's book to show up as the application icon.

questions

This experience raises a number of questions regarding the HTML5 Offline Application Cache support:

• How much space does the browser set aside for this cache? I assume there's a fixed upper limit for the cache, but does it compete with all other browser caching? Or is there a per-site or per-url cache limit?

• How can I be informed when my cache content is thrown out of the cache? Is the entire set of files listed in the manifest thrown out atomically?

• Is there enough functionality in the existing and proposed APIs to make reliable use of this capability?

I'm left wondering if it wouldn't just be better off to provide a full-function JavaScript API to the cache in general, although clearly this could be done as well as this declarative approach. My fear is the declarative approach might take you 80% of the way there, but leave out critical capability in that missing 20%.

More experimentation required. Have at it.

Or maybe this is just simple abuse of the capability. Instead of caching web sites, you could certainly get more control by having all the book content stored in a client-side SQL database, and then build an HTML application to navigate through it. It's just a lot more work.

debugger friendly

(this is a bit of a followup to my previous blog post - "debugging")

Hit a milestone yesterday, submitting my first patch to WebKit, in support of "Bug 25474: Show the filename and first line for '(program)' in the Profiler/Debugger". The patch provides the same support that Firefox provides to allow JavaScript eval() invocations and Function() constructor invocations to name the code they are injecting into the environment, by adding a comment to the source interpreted:

//@ sourceURL=some-url-goes-here

Presumably some-url-goes-here is replaced in your source by some kind of moniker that will be useful to you.

Why is this useful? Here's what your "list of scripts" looks like when developing a Dojo app, without this support: [link]; here's what it looks like with this support: [link]. A little more useful.

Last month, another interesting feature for debugging was added to WebKit - the ability to 'name' anonymous functions, after they've been created. A full description of this, and the value it provides, was written up a blog entry "Building a Better JavaScript Profiler with WebKit". In a nutshell, you can add a property displayName to a function which will be used as it's name in the debugger and profiler if the function doesn't otherwise have a name.

Quick notes:

• the sourceURL feature is not yet committed to WebKit, and even if/when it is, would only be available in WebKit nightlies.
• the displayName feature is only available in WebKit nightlies right now.
• the sourceURL feature does not currently work with the WebKit profiler, just the debugger.
• the displayName feature works in the WebKit profiler and debugger.
• Firefox supports (and invented!) the sourceURL feature, but AFAIK does not support the displayName feature.
• Dojo currently has support for sourceURL feature, but only if you're running in a Mozilla-based browser - chime in here for WebKit support.
• Dojo has no support for the displayName feature - chime in here on that issue.

enhanced object display

Here's what I'm going to look into next:

In the WebKit bug, "Bug 25724: Incorrect collection output for Javascript console evaluation", the reporter claims that the output when displaying the result of a jQuery function invocation is incorrect. Incorrect isn't fair; it's better to say that it's not what the user expected, but then, as jjb mentions in comment #6, "we can't read [the user's] mind".

It's certainly true that the current output isn't useful, and it's certainly the case that the developer of a debugger can't read a user's mind, so the obvious answer is to let the user decide how to see their output. In Smalltalk, the debugger would call debugPrintOn: instead of the more typical printOn: method (printOn: being analogous to toString in Java or JavaScript). This allowed developers to provide a debug-friendly version of the human-readable representation of an object, without impacting runtime code. JavaScript debuggers should do the same thing.

The other bit that needs to be done is to somehow augment the "properties" that you see associated with an object in the debugger. Literally what you see in the debugger is the implementation goop of an object. Not terribly useful, especially if values associated with objects are computed lazily, via a function invocation. Seeing a function instead of perhaps the lazily computed value provides little value (though it is obviously much safer!).

What I'd like to do is provide an optional method on a function which can be called to obtain "debug properties". For instance, the debug properties for a jQuery object might return the array of matching DOM elements. DOM elements themselves might return, instead of the goop currently returned, their parent DOM object, their children, and their next/previous siblings. These DOM element's might have a "debug toString" that actually showed the HTML representation of the node (which the debuggers already support in some cases).

Of course feel free to post additional ideas on this to the existing but report.

soundpoints

Here's something I did back in the Smalltalk days; would be fun to do this for JavaScript as well:

We all know what breakpoints are, but I extended the concept in Smalltalk with soundpoints (stole the idea from somewhere). Instead of having the debugger pause the execution of the program when a breakpoint is reached, the debugger would play a sound when a soundpoint is reached. The nice thing about this capability is that you get some feedback without having to stop your program. You can set soundpoints to let you know when you venture into code you're not expecting to venture into (some kind of beep/boop), or set a very short blip sound for a soundpoint you expect to hit a lot - giving you a kind of geiger counter. Since WebKit already has programmatic support for playing audio, the "hard" part of this is already done. What's needed is to integrate into the debugger framework. OK, that is the hard part.

As sort of a side issue, when I did this in Smalltalk, I was able to do this with user-land code; I didn't need to be some kind of core developer to get the function added. That's not true today in WebKit. You need to modify existing WebKit code to add functionality like this; there's no simple way of me just adding it to my environment. That would be a nice thing then also, having a way of extending the debugging environment through plugins or equivalent. See "Bug 24273: Inspector Plugins or Extend Capability". Also note that I believe FireFox has or will have the ability to extend FireBug via plugins. I think it's too much to ask to have portable plugins, at this point, but something to think about.

what else

What else can we do beyond all that? What are your frustrations with JavaScript debugging?