accessible backed up git repos for transient work-related projects

preface: this is basically a "note to self", but thought others might find it useful

I tend to create a lot of "projects" at work. A "project" is a directory in my "~/Projects" directory. Who knows what might be in it; probably code, or maybe just documentation. Many times, it's a "project" that I think may be useful to other people, and I may eventually "publish" to an either a public SCM repo farm like GitHub or an internal IBM SCM repo farm. I currently have 317 projects in my "~/Projects" directory.

Before "publishing" it though, I'll usually want to crank on it for a bit, see if I can get the crap to work, and figure out if it's actually useful. So there's a window where my project isn't being "backed up" or available to anyone else, because it's sitting on my laptop's spinning disk of rust. That's not great.

In the past, I've used Subversion and/or Mercurial to help manage this "back up" process for personal, non-work-related projects. What I'd do is set up a repo on my shared server, and then just use it like any other remote SCM.

Problems:

• What about work-related stuff? I don't want to publish something that's work-related on a non-IBM site.

• I made it sound like "set up a repo on my shared server" was easy. It isn't. Depending on the capabilities of your server, it might be impossible.

• Yeah, we do have various internal-only SCM repo farms in IBM, but those still require some "setup" and "maintenance", just a different flavor than shared hosting, and even that can be too much sometimes. Especially for a project which may only be useful for a couple of days.

Bummer.

a new hope

What I'm doing now is using Git, and storing my Git repos on IBM's internal, backed up, rsync-/smb-/http-accessible cloud file store. Even if you don't have rsync or smb access, there are likely ways you can do something similar, with different tools, on your own server.

Here's how it works:

• mount the file store on my local machine, create a directory for the new repo in my "git-repos" directory, which is in a part of my cloud file store that is publically accessible via http, unmount the file store.

• initialize the project on my local machine with git (eg, "git init")

• write a "backup" script, which looks like this:

  #!/bin/sh
  
  #--------------------------
  # make the git repo accessible via:
  #   git clone http://{host-name}/~pmuellr/git-repos/{project-name}/.git
  #--------------------------
  
  cd `dirname $0`
  
  git update-server-info
  rsync -av . {host-name}:{path-to}/git-repos/{project-name}
  

• Instead of using a "git add/commit/push" workflow, I use a "git add/commit; ./backup" workflow.

• As the comment in the script notes, the git repo is accessible via http, to anyone who has access to that URL. For the general case for me, that's anyone in IBM.

• The "backup" script, as written, also backs up my working directory. Which is a win as far as I'm concerned. Now I can also point folks to web-accessible/-renderable resources in my working directory.

I've found this to be a very useful, very lightweight process to keep my immature projects backed up. I don't doubt there are ways to do similar things with SVN or Mercurial or whatever as well, I just never tried.

I can, but haven't yet, set up a similar story for personal projects, using my shared server, but that should be pretty easy; it supports rsync, but not smb. Instead of "mount and create project directory", I can "ssh and create project directory". Even that's probably not needed; rsync can likely create that project directory for me, but I'm an rsync n00b and I kind of like having to make that step explicit.

debugging concatenated JavaScript files

In James Burke's blog post "Why not AMD?", the subject of debugging briefly came up. In response to the claim that AMD is "Inefficient by default", James responds: (bold emphasis applied by me)

I assume this means that it allows separate modules to be loaded async. FWIW, the current state of ES harmony modules will allow separate file loading for each module. This works better for debugging.

For AMD, you can get the "one script file at the bottom of the page" loading with the 750 byte almond AMD shim and runtime http loading.

I cannot see a better solution to this issue. If you have ideas I would like to hear them.

I'd like to point out some futures/options for making concatenated scripts easier to debug. None of solutions deal with mangled names, they're just ways of letting the debugger show a concatenated script as the set of individual scripts that make up the concatenated script. Baby steps!

Source Map support in the web debuggers

Werner Schuster's blog post "Mozilla, WebKit To Support Debugging Minified JS As Well As CoffeeScript and other JS Languages" covers most of the ground for the Source Map work that's ongoing. Chris Heilmann's recent presentation from Velocity "Firefox Performance" mentions some work going on at Mozilla.

Of course, we need both ends working here; minizer's are going to have to be able to spit out Source Maps, as well as language translators like CoffeeScript. The CoffeeScript bug for Source Map work is here (I think).

cheating with eval() and //@ sourceurl

Back in the day, a feature in Firebug that I copied into Web Inspector was "sourceURL" support, described in "Give your eval a name with //@ sourceURL". I recently made use of this in my "unminified" version of the injected target weinre script. weinre ships with two versions of the code that gets injected into the target: target-script.js and target-script-min.js. The "min" version is what you should be using for typical weinre debugging, the non-"min" version is what I use when debugging the debugger.

The file target-script-min.js contains gently minized JS files, just concatenated together. The target-script.js file is a bit different. For every JS file, it adds a "//@ sourceurl" comment to the bottom of the file, converts the script content to a string with JSON.stringify(), and then writes an eval([that string here]) to the concatenated file.

The difference between debugging these in a browser which supports sourceurl is night and day. Try it: after visiting these pages, open your web debugger and look at the scripts which are loaded. Note, this works on Google Chrome 16.0.912.36 beta on a Mac (and likely other versions).

• target-script-min
• target-script

what's new in weinre 1.6.0

Notable changes in weinre 1.6, which was released last week. More weinre links here.

Element Highlighter Color Changes

The "element highlighter" is the coloring you get on your web page when you hover over DOM elements in weinre's Elements panel. Previously, the colors were very muddled.

Below are the new colors, with the orange-ish margin, turquoise-ish padding, and purple-ish body. Not shown is the border (this element has none), which is a bright yellow.

There are still some issues with this highlighter, which was about the 6th different one I had tried. As a result of all those attempts, the element highlighter is now fairly pluggable, and it should be straight-forward for you to create a better one. I dare ya.

JavaScript Callback Error Diagnostics

I often hear people talk about debugging their JavaScript with weinre, but weinre really just provides a REPL and console logging. New in 1.6 is an attempt to provide some additional JavaScript diagnostics.

For a set of built-in functions in JavaScript that have callbacks as parameters, weinre overrides those functions to run an instrumented version of the callback instead. The instrumented version just runs the specified callback within a try/catch statement and reports if an exception occurred. The functions that weinre overrides are:

• window.setTimeout()
• window.setInterval()
• window.addEventListener()
• Node.prototype.addEventListener()
• XMLHttpRequest.prototype.addEventListener()

The Node.prototype.addEventListener() override should handle document and all DOM elements in your document. The XMLHttpRequest.prototype.addEventListener() will handle your XHR's. Unfortunately, I'm not quite sure how to override the on-family of event properties, because browser host objects suck.

As an example of what you'll see in the Console window when an exception occurs, here's some JavaScript code which will generate a runtime error when the specified button is clicked:

Below is what you see in the console when the button is clicked.

Description of the five error messages in the console:

1. This is the toString() of the exception that was caught when the callback was invoked.
2. The callsite is an informational description of the call which registered the callback. In this case, it tells us the callback was registed to an <input> element's click handler, and that the callback registered was a function named buttonClicked().
3. If the caught exception has a stack property, the stack will be dumped following this line. Otherwise, you won't see this line or the stack.
4. This is the stack at the time the runtime exception occurred, if available. Note that the last two lines (at <error: TypeError: ...) seem to be bugs of some kind in the browser (couldn't be in my code!).
5. This is actually a line from the new onerror handler that weinre registers, unrelated to the new callback exception logging described here.

Not all browser's JavaScript engines support the error.stack property (eg, WebKit's JavaScriptCore), so this is what you'll see if you're running on one of those lame engines:

Also note the third line is, again, from the new onerror handler, which seems to like to not contain useful information. Why not? See here, here, here, and here.

The New Resources Panel

The Resources panel is brand new for version 1.6. It provides diagnostics for all the XHRs that you code issues. Below is an example of a web page that has issued five XHR's.

Clicking on one of those rows will display additional information in the right hand side of the window, as shown below.

There are two tabs here, one to show "header" information, the other to show the "content". The content for the XHR is shown below.

To dismiss the detail view and get back to the table view, click the circled X button to the left of the Headers/Content tabs.

weinre 1.5.0 is now available

weinre 1.5.0 is now available at https://github.com/phonegap/weinre/downloads.

ChangeLog: http://phonegap.github.com/weinre/ChangeLog.html

I also went through and neutered the bits at the old "pmuellr/weinre" site. Everything should now, somehow, point you to the "phonegap/weinre" site/repo. Let me know if you see dangling pointers. (note, I'm not going to change links in my old blog posts)

public weinre server hosted on the intertubes

Although I knew it was coming, the twitter post from Andrew Lunny yesterday made me very happy:

hosted weinre for PhoneGap Build is now live at http://debug.phonegap.com

Awesome! The fine folks at Nitobi are running this server, not me. Thank you so much! I owe everyone involved a cool beverage of your choice, next time we meet (OSCON?).

Of course, if you have any questions about using weinre, including this new public debug server, don't hesitate to ask:

• Google Groups: https://groups.google.com/group/weinre
• IRC: #phonegap @ irc.freenode.net
• GitHub issues - https://github.com/pmuellr/weinre/issues, for bug reports, feature requests, etc

some additional things to do for multi-user weinre servers

There's always more work to do, isn't there?

• make the front page easier to use and understand, if required

• optimize client and target resources - ok, this is kinda embarrassing, but the weinre server is not making use of some of the well-known best practices to optimize HTTP resource access

• provide some hints / tips / best practices for folks wanting to run their own multi-user weinre server

In case you're wondering why someone would want to run another multi-user weinre server, now that we have one "in the cloud":

• um, you're going to depend on the cloud?

• there's not much security around the current weinre client/server interaction patterns, and there is at least the problem that the current debug packets are flowing across the internet in plain text. That includes, at a minimum, any serialized versions of DOM nodes that you are seeing in the DOM elements panel.

Given that, seems like it would make sense for organizations to run their own in-house weinre servers.

I'm planning on setting up a weinre server inside the IBM firewall for IBMers to use. I'll post news on that subject to the new weinre "Community" I set up on the IBM internal "Connections" page. To find that community, search for weinre, mobile, debug, etc.

callback debugging - better mobile error diagnostics

One of the common issues that comes up regarding debugging mobile web apps, is that the error diagnostics for JavaScript are terrible. Sometimes non-existent. Current state of the art is to turn on the "Debug Console" switch in Mobile Safari, which is kinda horrifying, but does provide some information. Or use low-level debug tools like Android's logcat. These tools provide additional clues about JavaScript errors that have occurred.

But I've wanted to add something to weinre, and a while back opened issue 38 - "provide better error handling - by hook or by crook". My first attempt was to try overriding some addEventListener() functions, which would instrument the callback methods passed to them. The instrumentation was to wrap the invocation of the original callback in a try/catch, and if something was caught, report it. The reason this is a good avenue to go down is that after the top-level JavaScript code in your HTML and JavaScript files runs, everything else is run because of a callback somewhere. If you can control the callbacks, you control JavaScript.

Aside, there's a number of interesting links in that bug report:

• v8's stack trace API, callable from JavaScript
• what happens when you override Function.prototype.call() in JavaScriptCore (Holy Shit Really?!)

I decided to spend some time on the error diagnostics issue this week, but first did a cursory Google search, and found an article by Tom Robinson on the subject. Basically describes the approach I was heading down, so things still sounded promising.

Since I didn't see any kind of existing library to do this kinda stuff, I wrote one - CBWrapper. Putting it in a separate GitHub repository for now, because it's kind of a general purpose callback diagnostic tool. Could be useful in Node, perhaps. If it turns out to be useful and safe for weinre, I'll add it, but for now you can use it as a separate set of script files.

Here's what it can do. Keep in mind that the normal amount of JavaScript error handling you get in weinre is - nothing. Add the CBWrapper scripts as indicated on it's page, and you'll instead see the following output in the weinre console. Note that the information displayed is different depending on your web browser's JavaScript engine. I only tested under Mobile Safari and Android's Browser.

under JSC - all Apple devices and older Android devices

exception: TypeError: Result of expression 'xScript' [null] is not an object.
location:  https://[[url-elided]]/CBWrapper/tests/script-runtime-error-function.js:8
callback:  window.setTimeout(aScript(), 2000)

under V8 - newer Android devices

exception: TypeError: Cannot call method 'doSomething' of null
callback:  window.setTimeout(aScript(), 2000)
           [http://[[url-elided]]/CBWrapper/tests/test-weinre-runtime-function-script.html:10]
stack:
TypeError: Cannot call method 'doSomething' of null
    at bScript (http://[[url-elided]]/CBWrapper/tests/script-runtime-error-function.js:8:11)
    at aScript (http://[[url-elided]]/CBWrapper/tests/script-runtime-error-function.js:3:5)
    at cbWrapper (http://[[url-elided]]/CBWrapper/CBWrapper.js:253:27)
    at http://[[url-elided]]/target/target-script-min.js#test:3674:14

Curiously, the exception messages are wildly different here. The line that threw the exception was:

    xScript.doSomething() // where xScript == null

Note that besides the other information provided, there's a line labelled "callback", which gives you some information about the function that registered the callback that just threw the error. In this case, it was a window.setTimeout() call, and the message provides some contextual information on the call itself. The registration function is no longer on the stack, but I keep a reference to it in case you want it. Like in this case.

CBWrapper can do a lot more than log errors. It provides before/after callbacks before every callback that it wraps, so you can do things like get the elapsed time of all the callbacks, log callback events on the Web Inspector timeline, etc. In theory.

Practice, however, is a different story. Playing with it a bit, under weinre, led me to some issues. See, weinre itself uses some of the same callbacks that CBWrapper wraps, so it's pretty easy to fall into a trap where you do something in a CBWrapper callback (write to the console), which causes weinre code to set a timer or do an XHR, and you've just created an endless chain of console logs / CBWrapper callbacks that will never end. The callback-based async version of infinite recursion.

Finally, Kent Beck posted the following message on Twitter today:

the problem with callbacks triggering callbacks triggering callbacks is that when something goes wrong, reading the code doesn't help much

Seems like maybe you could whip something up with CBWrapper that could show you callbacks registered from other callbacks, and then show a trail of those callbacks when an error is thrown. Would that help? Only one way to find out and I'll leave that as an exercise for the reader.

weinre 1.4.0 available

weinre 1.4.0 is now available in the usual place: https://github.com/pmuellr/weinre/downloads. Changelog is available here: http://pmuellr.github.com/weinre/ChangeLog.html.

Major changes for this release:

• Google Groups mailing list
• extensions
• multi-user support

As always, for bugs or feature requests, create an an issue. Or use the new Google Group (see below).

Google Groups mailing list

I've created a Google Group for weinre, here: http://groups.google.com/group/weinre. Usual stuff applies, the group is set to moderate messages for "new" members.

extensions

Like all great platforms, Web Inspector has an extension mechanism allowing users to write code to extend the capabilities of the tool. The extension support for real Web Inspector (used in Safari/Chrome/etc) is not currently enabled, but I went ahead and enabled it in weinre.

I've written two extensions so far.

• https://github.com/pmuellr/weinre-ext-dom-monster - runs DOM Monster on your device, displaying the results in weinre
• https://github.com/pmuellr/weinre-ext-sample - an example extension you can use to base a new extension on.

Here's a screen-shot of DOM Monster running in weinre - click the image to see a larger version. Isn't that panel icon so cute!

The extension support is best-effort at present. Definitely open an issue for bugs or feature requests. If it's API related, I may forward them to the WebKit bugzilla component for Web Inspector.

I presume in real Web Inspector, all the various .html files that are loaded will be sandboxed, like Chrome and Safari extensions, but for weinre they're running as iframes of the main inspector window. As such, they have full access to everything in Web Inspector. Powerful. Dangerous. Have fun.

I decided to use a deployment structure that is easily mapped to a VCS, like Git/GitHub. This makes it easy to install and update extensions. To install the DOM Monster extension, run the following shell commands:

mkdir ~/.weinre
mkdir ~/.weinre/extensions
cd ~/.weinre/extensions    
git clone git@github.com:pmuellr/weinre-ext-dom-monster.git

To update the extension, after I make and publish changes, just run:

cd ~/.weinre/extensions/weinre-ext-dom-monster
git pull

multi-user support

This support will only be useful to folks who want to host a weinre server for multiple people to use simultaneously. For more information, see: http://pmuellr.github.com/weinre/MultiUser.html.

Although it's noted in the doc, I will mention here also. There is no real security here. Your client and target need to know a "shared secret" to connect to each other. On the bright side, there's never any information persisted on the weinre server, everything is transient, so if you aren't connected to a weinre server, there's nothing for anyone to steal.

There's likely more work to be done here, but looking for feedback.

weinre 1.3.0 available

weinre 1.3.0 is now available in the usual place: https://github.com/pmuellr/weinre/downloads. Changelog is available here: http://pmuellr.github.com/weinre/ChangeLog.html.

Major changes for this release:

• CSS style editing now available. I'm surprised more people haven't complained about this - I think this is one of my favorite features of big Web Inspector. If you're not familiar with it, there's some documentation on Chrome's DevTools documentation site about style editing.

• The Resources panel has been enabled, but currently only the localStorage and sessionStorage entries contain values. I couldn't resist adding localStorage support after seeing Steve Souder's blog post "Storager case study: Bing, Google".

A few notes on the style editing:

• Those little yellow warning icons to the left of the property names are error indicators; either the property name or value is incorrect. Unfortunately, for weinre, we can only catch such errors for properties you edit. We can't catch errors that are present in your CSS that the browser originally sees. The CSSOM doesn't provide access to error-ish stuff.

• Adding new style rules doesn't work yet. Vote on this issue if you really need it; pretty low priority for me.

• If the Matched CSS Rules section is empty for most of the elements you click on, it may be because of same origin restrictions for accessing CSS rules via the CSSOM. In other words, weinre may not be able to see the CSS rules for CSS files that you import from a different origin than the original document. Still investigating.

As always, for bugs or feature requests, create an an issue.

WTF are these scoop files?

At some point, someone's going to look at the weinre source code and ask, "WTF are all these .scoop files?"

• They're almost JavaScript files, intended to preprocessed into legal JavaScript files. The lines of the file which are not indented are "directives" that indicate how the following indented bits are to processed. The indented lines are plain old JavaScript which isn't touched by the preprocessor except to deal with "super" calls.

• They're my response to wanting to use classical OOP in JavaScript, without dealing with the crufty the way it's typically done in JavaScript. I like JavaScript, but I hate cruft.

• Since I'm also playing with CommonJS, I ended up folding some goodies for that in too.

Assuming you know JavaScript, and have some familiarity with classical OOP, I suspect you will get the gist of what's going on if you look at one of the weinre .scoop files.

I knew I'd be writing a lot of JavaScript for weinre, so I wrote scoop first to make it a pleasant experience.

The source for scoop is at GitHub. The doc is spartan, but my existing user base hasn't complained too much about that (existing user base == me).

Here's what I learned in the process of using scoop to build weinre:

• I like it. It's become second nature to author scoop files.

• The preprocessor runs fast enough that it doesn't impact my build in any significant fashion. In practice, all software I create has some kind of automated build that runs when I touch a source file. Including web sites. Adding one second to that process, if that, doesn't impact me. (weinre builds take 5 seconds, which includes compiling Java source - I could optimize that build more, but haven't needed to)

• There's no new language to learn. It's just JavaScript with some structuring. Just a few directives to learn, and rules to follow (indent your code). No offense to the altJS crowd - I love all that stuff, but really don't want to invest in learning another language, figure out how to debug it, etc.

• scoop and the CommonJS module runtime I use (modjewel) both take turns processing the source, but don't change the line or column numbers of the original source. When the JavaScript runtime complains about a syntax or runtime error with a line number, no mental mapping is required to find the line in the scoop source. The meat of code I step through in the debugger, function bodies, is the same as the scoop code I write.

• Exposing classes as modules seems to work fine, but requires the module system to support overriding the exports value that the module ends up populating. In practice, this works with node.js and modjewel, so, works for me. Specifically, the first class defined in a scoop file is assigned to module.exports, and thus any code which does a require() on that module will get the class as a result.

• The big downside of overriding module.exports is that recursively require()'d modules don't work right. I added a requireClass directive which will identify this condition at runtime, and fails fast with a decent diagnostic. Fixing these situations typically isn't difficult.

• Syntax-coloring text editors generally do the right thing for scoop files, once you lie to them and tell them that a scoop file is just a JavaScript file. The directives are usually colored oddly, but that's not a big problem. The actual JavaScript code is always colored correctly, for the text editors I use.

Part of my rationalization of doing this is based on Smalltalk's class structuring story. The Smalltalk language doesn't define any class structuring features like a "class statement" - all that is handled by the tooling and runtime. And it has a great classical OOP story. If Smalltalk can do it, then so can JavaScript.

I have a long laundry list of things I'd like to do with scoop, especially since I consider it just a stepping stone to having a nice, dynamic, live development environment for JavaScript, just like Smalltalk. The current incarnation is the ascii, text-file version of where I want to go. The thing I want to work on first is some read-only browsers for the module system and the class structure, that you can run right in your browser (or debugger). Next would come live editing, etc, etc. Probably would have worked on some of that stuff had my JSConf 2011 proposal for scoop been accepted. The pressure is off on that front though.

mobile debugging updates

Yesterday, mobile web developers got some refreshed debug tools to play with. I released a new version of weinre, and Remy Sharp released some YouTube movies showing off his JS Console tool.

The big new feature for weinre is the Timeline. The Chrome Dev Tools version of Timeline is documented here, and weinre implements some of it. A little of it. Timers, intervals, XHRs, some global events. It handles onerror, but I haven't seen onerror available on any mobile WebKit browsers, since it's a relatively new addition to WebKit.

Aside, if you're not familiar with WebKit Web Inspector / Chrome Dev Tools (same thing, different names), the Google Chrome Developer Tools site seems to be the best source of information.

With both jsconsole and weinre, there are limits to what you can do. Specifically, since they are both written in user-land JavaScript, and there are no user-land JavaScript APIs that deal with breakpoints, there's no way to do breakpoints (well, ...). Even just getting error information / stack traces is difficult-to-impossible.

What we really need is the real Web Inspector to work on mobile platforms. There's some code in WebKit to handle the remote-y-ness, as shown by this movie. Not shipping yet, though, AFAIK. Maybe on iOS 4.3 or already in Honeycomb but hidden? Would be a nice surprise for sure, but would make weinre - my baby - obsolete. That's fine with me, I've been developing software long enough I know that sometimes your creations need to die. There's plenty of other stuff to do in this arena.

What's next for weinre? You tell me. Or post a message on Twitter mentioning weinre, and I'll probably see it.

Here's the new movie for weinre:

Mobile, Debugging, Challenges, Hacks

Alex Russell has a great post today on software performance, titled Performance Innumeracy & False Positives. Having worked in the embedded/mobile space for the last decade (on and off), I certainly feel his pain. On the other hand, this is one of the reasons why I love the mobile space: it's challenging.

Performance is just one of the challenges that we face in the mobile space. Other issues are program and data size, RAM usage, and network speed. All constrained compared to desktop-class machines.

For some of these challenges, we need tools, because it's often not obvious where the problems really are. Sometimes the only tools you have is your eyes staring at the code and intuition. Performance, in particular, is an area that needs tools. If there's one thing I've learned about performance problems over the years, it's this: your intuition is almost always wrong.

Alex showed a stack trace from WebKit in his blog post, without any kind of time numbers in it - a pretty crude tool. But most people don't even have that luxury (instrumenting WebKit), for applications running in a browser. On some mobile platforms, recompiling parts of the OS isn't even a possibility for most developers.

In lieu of actual tools, I'm always up for crazy hacks. That's basically what weinre is - a crazy hack. Take the Web Inspector user interface, and plug in a fake runtime behind it running on the target browser - just enough to get the user interface to display something interesting. Until we have the real Web Inspector showing us the browser innards running on a mobile platform, maybe this cheat will be useful to folks.

On a slightly related note, an interesting bug got entered against Web Inspector the other day: Bug 53659 - Web Inspector: Better support for finding "leaked" DOM. The summary describes a debugging scenario for which some new tooling could help. Here's the sad part: if this were Smalltalk, there's a chance you could write a specialized tool to collect the desired information with user-land code. The ability to enumerate all the live objects in a running image was an especially helpful feature. How useful would it be, if you could do that in JavaScript? In a special debugging mode, of course - not exactly a function you want to expose to everyone. But it would be nice to not have to wait for the platform developers to write tools for us, if we could write our own.

Could you help diagnose the scenario described in Bug 53659 with a crazy hack? If not, what's the bare minimum additional runtime functionality you'd need in JavaScript to do it?