Links

pmuellr is Patrick Mueller, Senior Node Engineer at NodeSource.

other pmuellr thangs: home page, twitter, flickr, github

Friday, November 30, 2012

weinre installable at heroku (and behind other proxy servers)

I finally resolved an issue with weinre relating to using it behind proxy servers.

Matti Paksula figured out what the problem was, but I wasn't happy with the additional complexity involved, so I made a simpler and more drastic change (in the getChannel() function).

Used to be, subsequent weinre connections after the first were checked to make sure they came from the same "computer"; that was just a silly sort of security check that can be worked around, if need be. So why bother with it, if it just gets in the way.

So, now weinre can run on heroku. I have a wrapper project at GitHub you can use to create your own server: https://github.com/pmuellr/weinre-heroku

Since Heroku provides 750 free dyno-hours per app per month, and weinre is one app, and uses one dyno at a time, and 750 hours are in a month, you can run a weinre server on the Intertubes for free.

As usual, weinre has been updated on npm.

Friday, March 30, 2012

unofficial weinre binary packages for your convenience at my apache home page

weinre is now getting ready to go for the brass ring at Apache: an "official" release.

NOT THERE YET - NO OFFICIAL RELEASES YET.

Until that time, I have unofficial binary packages available for your convenience.

I've gone ahead and created a little download site at my personal Apache page, here: http://people.apache.org/~pmuellr/.

The download site contains all the doc and binary packages for every weinre version I've 'shipped', in the 1.x directories. It also contains unofficial binary packages of recent cuts of weinre for your convenience, as the bin, doc, and src directories in the weinre-builds directory, and the same version of the doc, unpacked, in the latest directory of weinre-docs.

As with previous binary releases for your convenience, you can npm install weinre. Here's the recipe:

npm install {base-url}apache-cordova-weinre-{version}-bin.tar.gz

{base-url} = http://people.apache.org/~pmuellr/weinre-builds/bin/
{version}  = 2.0.0-pre-H0FABA3W-incubating

That version number will change, so I'm not providing an exact link. Check the directory.

help me help you

If you'd like to accelerate the process of getting weinre to "official" status, you can help!

  • install it
  • use it
  • review it
  • build it
  • enhance it

Feel free to post to the cordova mailing list at Apache, the weinre google group, clone the source from Apache or GitHub, etc. All the links are here: http://people.apache.org/~pmuellr/weinre-docs/latest/.

One link I forgot to add, and feel free to open a bug about this, is the link to open a bug. It's hideous (the URL). Add component "weinre" and maybe a "[weinre]" prefix on the summary.

Thank you for helping yourself!

you f'ing moved the downloads again you f'er!

Chill. And some bad news. The official releases won't be here (I don't think). But I will provide a link on that site to the official releases, when they become real.

I intend to keep this page around until I can get all the weinre bits available in more appropriate places at Apache. Not there yet.

personal home pages at Apache

Apache provides "home pages" for Apache committers. That's what I'm using for this archive site. If you're a committer, and curious about how I did the directory indexing, the source for the site (modulo actual archives), is here: https://github.com/pmuellr/people.apache.org.

2012/03/30 1pm update

Jan Lehnardt pointed out to me on twitter that I should avoid the use of the world "release", so that word has been changed to things like "binary package for your convenience".

Friday, February 17, 2012

npm installable version of weinre

As part of the port of weinre from Java to node.js it seemed logical to make it, somehow, npm installable.

Success.

There is currently a tar.gz download of the weinre runtime available at the pmuellr/incubator-cordova-weinre downloads page. I may keep a 'recent' build here until I get the continuous builds of weinre at Apache working. At which point, the tar.gz should be available somewhere at apache.

Of course, let me know if you have any issues. Preferred place for discussion is the weinre Google Group.

install locally

npm install https://github.com/downloads/pmuellr/incubator-cordova-weinre/weinre-node-1.7.0-pre-2012-02-17--16-47-15.tar.gz

After that install, you can run weinre via

./node_modules/.bin/weinre

install globally

sudo npm -g install https://github.com/downloads/pmuellr/incubator-cordova-weinre/weinre-node-1.7.0-pre-2012-02-17--16-47-15.tar.gz

After that install, you can run weinre via

weinre

notes/caveats

  • The command-line options have not changed, although the --reuseAddr option is no longer supported.

  • There is no longer a Mac 'app' version of weinre.

  • This is not an 'official build' of weinre, in the Apache sense. I need to figure out what that even means first.

  • These 'unofficial' builds have a horrendous semver patch version names. Deal. If you have a better scheme, please let me know.

  • I don't have plans to ever put weinre up at npmjs.org, since I have no idea how this would work, in practice. I don't want to own the weinre package up there; ideally it would be owned by Apache. Can't see how that would work either. If I ever do put a weinre package up at npm, I think it would be a placeholder with instructions on how to get it from Apache. Or maybe there's a way we can auto-link the npmjs.org package to Apache mirrors somehow?

  • I need to look at whether I should continue to ship all my package's pre-reqs (and their pre-reqs, recursively) or not. And relatedly, whether I should actually make fixed version pre-reqs of all the recursively pre-req'd modules. An issue came up today that I "refreshed" the node_modules, and got an unexpected update to the formidable package (a pre-req of connect which is a pre-req of express which is a pre-req of weinre). That's not great. And could be fixed by having explicit versioned dependencies of all the recursively pre-req'd modules in weinre itself. Sigh. What do other people do here?

  • Prolly need to improve the user docs. The remainder of this blog post is the shipped README.md file. So, to read the docs, start the server, read the docs the server embeds, online. Sorry.

running

For more information about running weinre, you can start the server and browse the documentation online.

Start the server with the following command

node weinre

This will start the server, and display a message with the URL to the server. Browse to that URL in your web browser, and then click on 'documentation' link, which will display weinre's online documentation. From there click on the 'Running' page to get more information about running weinre.


Update: Patrick Kettner posted a link to this blog post on Hacker News, so you can discuss there, since I'm one of those "disable comments on blog" people. http://news.ycombinator.com/item?id=3604291.

Thursday, January 19, 2012

AMD - the good parts

Based my blog post yesterday "More on why AMD is Not the Answer", you might guess I'm an AMD-hater. I strive to not be negative, but sometimes I fail. Mea culpa.

I do have a number of issues with AMD. But I think it's also fair to point out "the good parts".

my background

I've only really played with a few AMD libraries/runtimes. I've been more fiddling around with ways of taking node/CommonJS-style modules, and making them consumable by AMD runtimes. The AMD runtimes I've played with the most are almond and (my own) modjewel. My focus has been on just the raw require() and define() functions.

But I follow a lot of the AMD chatter around the web, and have done at least reading and perusing through other AMD implementations. To find implementations of AMD, look at this list and search for "async". The ones I hear the most about are require.js, Dojo, PINF, and curl.

And now, "the good parts".

the code

Browsing the source repositories and web sites, you'll note that folks have spent a lot of time on the libraries - the code itself, documentation, test suites, support, etc. Great jobs, all. I know a lot of these libraries have been battle-tested by folks in positions to do pretty good battle-testing with them.

the evangelism

Special mention here to James Burke (not this one), who has done a spectacular job leading and evangelizing the AMD cause, both within the "JavaScript Module" community, and outside. In particular, he's been trying to get AMD support into popular libraries like jQuery, backbone, underscore and node.js. Sometimes with success, sometimes not. James has been cool and collected when dealing with jerks like me, pestering him with complaints, questions, requests and bug reports. Some open source communities can be cold and bitter places, but the AMD community is not one of those, and I'm sure that's in part due to James.

Thanks for all your hard work James!

the async support

AMD is an acronym for Asynchronous Module Definition. That first word, Asynchronous, is key. For the most part, if you're using JavaScript, you're living in a world of single-threaded processing, where long-running functions are implemented as asynchronous calls that take callback functions as arguments. It makes sense, at least in some cases, to extend this to the function of loading JavaScript code.

If you need to load JavaScript in an async fashion, AMD is designed for you, and uses existing async function patterns that JavaScript programmers should already be familiar with.

support for CommonJS

This is a touchy subject, because when you ask 10 people what "CommonJS" is, you'll get 15 different answers. For the purposes here, and really whenever I use the phrase "CommonJS", I'm technically referring to one of the Modules/1.x specifications, and more specifically, the general definitions of the require() function and the exports and module variables used in the body of a module. These general definitions are also applicable, with some differences, in node.js.

When I use the phrase "CommonJS", I am NEVER referring to the other specifications in the CommonJS specs stable (linked to above).

AMD supports this general definition of "CommonJS", in my book. Super double plus one good!

Aside: I'm looking for another 'name' to call my simplistic notion of CommonJS. So far, things like simple require()y support, seem to capture my intent. But you know how bad I am with naming.

the design for quick edit-debug cycles

One of the goals of AMD is to facilitate a quick edit-debug cycle while you are working on your JavaScript. To do this, AMD specifies that you wrap the body of your module in a specific type of function wrapper when you author it. Because of the way it's designed, these JavaScript files can be loaded directly with a <script src=> element in your HTML file, or via <script src=> elements injected into your live web page dynamically, by a library. This means that as soon as you save the JavaScript you're editing in your text editor/IDE, you can refresh the page in your browser without an intervening "build step" or special server processing - the same files you are editing are loaded directly in the browser.

the resources

Beyond just loading JavaScript code, AMD supports the notion of loading other things. The things you can load are determined by what Loader Plugins you have available. Examples include:

Coming from the Java world, this is familiar territory; see: Class.getResource*() methods. For the most part, I consider this to be amongst the core bits of functionality that a programming library should provide. It's a common pattern to ship data with your code, and you need some way to access it. And, of course, this resource loading is available in an async manner with AMD.

Aside: I'm not sure what the analog of this is in the node.js world. You can certainly do it by hand, making use of the __dirname global, and then performing i/o yourself. I'm just not sure if someone has wrapped this up nicely in a library yet.

the build

At the end of the day, hopefully, your web application is going to be available to some audience, who will be using it for realz. It's unlikely that you will want to continue to load your JavaScript code (and resources) as individual pieces as you do in development; instead, you'll want to concatenate and minimize your wads of goo to keep load time to a minimum for your customers.

And it seems like most AMD implementations provide a mechanism to do this.

Wednesday, January 18, 2012

More on why AMD is Not the Answer

Follow-on to Tom Dale's blog post "AMD is Not the Answer" and James Burke's response "Reply to Tom on AMD". I pretty much concur with everything Tom said. Here are some points brought up by James:

need an alternative

James says:

what is missing is a solid alternative to AMD. In particular, "use build tools to wrap up CommonJS modules" is not a generic solution. It does not allow for generic dynamic loading of resources, particularly from CDNs. Dynamic loading is needed for big sites that still do builds but want to stage loading of the site as the user uses it.

I don't think using build tools precludes you from building wads of modules that can be loaded dynamically. Generic solutions bother me. See: J2EE.

ceremony

James says:

On his concern about too much ceremony -- this is the minimum amount of ceremony:

    define(function(require) {
        //Module code in here
    });

On the other hand, here's some code pulled from dijit.Calendar.js:

    define([
        "dojo/_base/array", // array.map
        "dojo/date",
        "dojo/date/locale",
        "dojo/_base/declare", // declare
        "dojo/dom-attr", // domAttr.get
        "dojo/dom-class", // domClass.add domClass.contains domClass.remove domClass.toggle
        "dojo/_base/event", // event.stop
        "dojo/_base/kernel", // kernel.deprecated
        "dojo/keys", // keys
        "dojo/_base/lang", // lang.hitch
        "dojo/sniff", // has("ie")
        "./CalendarLite",
        "./_Widget",
        "./_CssStateMixin",
        "./_TemplatedMixin",
        "./form/DropDownButton"
    ], function(array, date, local, declare, domAttr, domClass, event, kernel, keys, lang, has,
                CalendarLite, _Widget, _CssStateMixin, _TemplatedMixin, DropDownButton){
        //Module code in here
    });

This isn't just ceremony. It's ugly ceremony of the sort that's difficult to debug when you screw up.

module consumption

James says:

the alternative he mentions does not handle the breadth of JS module consumption

The simple require()y story of npm modules seems to handle the JS module consumption pretty well, for node.js. For the browser, you can use npm modules with Browserify or other "build tools". Where is the equivalent of npm for AMD? Or maybe npm already contains AMD modules? Or let's say I want to just use a single module out of Dojo in my app. How do I do that?

Thursday, January 12, 2012

making Web SQL easier to use

One HTML5-ish area I've not yet played much with, but am starting to now, is Web SQL. Web SQL is the W3 standard for "SQL in the browser", which has been orphaned by the W3C. Here are some warnings from the Web SQL Database spec at w3.org:

  • "Beware. This specification is no longer in active maintenance and the Web Applications Working Group does not intend to maintain it further."

  • "This document was on the W3C Recommendation track but specification work has stopped. The specification reached an impasse: all interested implementors have used the same SQL backend (Sqlite), but we need multiple independent implementations to proceed along a standardisation path."

Despite the fact that the W3C no longer endorses Web SQL and will never 'standardize' it, implementations of Web SQL are shipping in browsers, including some mobile platforms. It's not dead yet.

what's the api like?

Here's an example right out of the spec, to count the number of rows in a table:

A function showDocCount()that calls a function db.readTransaction() that takes two anonymous functions as callbacks. One of those callbacks calls t.executeSql() passing it yet another callback, which finally, mercifully, gets the row count.

It's executing a single SQL statement.

But wait, it gets better.

If you need to run multiple SQL statements within a single transaction, you'll have to chain those executeSql() invocations within increasingly nested callbacks.

It's basically a horror show. At least as far as I'm concerned. Nested callbacks are things I hate to have to write, and things I really hate seeing other people have to write. The result is almost always unintelligible.

can we make this easier to understand?

For weinre, I implemented a stepper module used by the inspector's database interface. Not very clear what's going on there though, sorry.

The basic idea is that you provide an array of functions - the steps - that the stepper will arrange to run in order, by managing the callbacks itself. Double bonus good - you get to linearize your database transaction steps and you don't have to deal with callbacks!

I found another instance of this kind of processing in Caolan McMahon's async library - specifically the waterfall() function. Not really appropriate for Web SQL, but at least it seems to validate the approach.

I decided to implement a clean implementation of the stepper, trying to make it easy to use and understand. The result is available as WebSQLStepper at GitHub. As a use case, I took the Todos sample from Backbone, and modified it to use this library to persist the data to a Web SQL database. The complete sample is included in the GitHub repo, and here are the steps used to read all the rows of a database:

references

wr - an auto-build command for the command-line

For much of my programming career, I've been a user of "IDE"s, including BASIC, Turbo Pascal, various Smalltalks, IBM's VisualAge products and Eclipse.

Lately, I've been spending most of my time working without any IDEs. My tools are text editors, terminal windows, a boat load of unix commands built-in to my OS, some additional command-line tools from various open source locales, and web browsers. I think of this loosely organized/integrated set of tools as my IDE, they're just not packaged up in a nice single app.

But there's one thing I dearly love and really depend on in most IDE's that's missing from this command-line-based environment: an "auto-build" capability. Specifically, when I save a file I'm editing, I want a "build" to happen automagically. Without an "auto-build" capability, I'm left in a world where I save a change in my text editor, jump over to the terminal, run make, then jump to my browser to test the build. The jump to the terminal is not desired. Using some existing editor's capability of running shell commands doesn't work either, because I use a wide variety of tools to "edit" things - all the tools would need this capability.

(Quick aside about "builds". A lot of people see "build" and think "30 minutes of waiting for the compiles to be done". Yes, those builds are long and painful. I'm looking at you WebKit. But in all the IDEs I used to use, builds were insanely fast. Sub-second fast. If you're living the "big C++ project" lifestyle, I feel your pain, and "auto-build" is probably not applicable for you.

So, I'm talking about quick builds here. As an example, once you've done at least one build of weinre, subsequent builds take on the order of 6 seconds or so - I consider that slow, but it's doing a lot of work, and I could probably cut the time in half if I needed to - I just haven't felt the need to.)

a command to run something when a file changes

A few years ago, I wasn't aware of a tool that could watch for arbitrary file changes and run a command for me, so I wrote one - run-when-changed. This script served me well for a long time, but there were always little thingees I wanted to add to it. And since run-when-changed polled the file system to see when files changed, I was always interested in finding a better story than polling and the inevitable waiting for the polling cycle to hit - even waiting 3 seconds for run-when-changed to realize a file changed was bothersome.

A port to node.js seemed like it might be in order, especially since it has a non-polling file watch function fs.watch().

Earlier this week, I spent a few hours and now I have a replacement for run-when-changed, called wr, available at GitHub, written using node.js. The name wr comes from "watch and run".

Some features:

  • installable via npm

  • colorized stdout and stderr

  • reversed video status messages

  • when run with no arguments, will look for a .wr file in the current directory to use as arguments

Here's what it looks like, using it with one of my current projects:

Click the image for a slightly larger version.

Legend:

  • the reverse video blue lines are status messages
  • the reverse video green lines are command success messages
  • the reverse video red lines are command failure messages
  • the blue text is stdout
  • the red text is stderr

As another example, I'm generating this blog post by running the following command:

    wr --exec "markdown wr.md > wr.html" wr.md

Breaking this command down, we have:

  • wr - the command
  • --exec - a command-line option (see below in gotchas)
  • "markdown wr.md > wr.html" - the command to run
  • wr.md - the file to watch for changes.

The arguments to wr are options, the command to run, and the names of files or directories to watch. When the wr command is run, it waits for one of the specified files to change, runs the specified command, reports the results of running the command, and then starts waiting for a file to change again. Exit the program by pressing ^C.

The ability to store a .wr file in a project directory makes life even easier. I used to create a target in my make files to invoke run-which-changed ; now I just put the wr invocation for a project in the .wr file, and run wr with no arguments.

My work-flow for using wr goes something like this:

  • open a terminal window
  • cd into my project's home directory
  • launch an editor, returning control to the command-line
  • run wr
  • move the terminal window where I will always be able to see the last few lines of the window, to see the live status
  • switch between text editor and browser, for my edit-save-reload-test cycle, all day long, keeping an eye out for red stuff in the wr output
  • when the day is done, ^C out of wr, and then commit work to my SCM

This can all be yours, with a simple command-line incantation:

    sudo npm -g install wr

current gotchas

wr has two ways of invoking the command to run, determined by the command-line option --exec; see the README, in the ENVIRONMENT section, for more information on the differences.

The big gotcha though is the non-polling file watching. This is an awesome feature, as it means as soon as you save a file, wr will wake up and run a command. The problem, on the Mac anyway, is that there is a relatively small limit on the number of files that can be watched this way. Like, 200 or so. If you go over the limit, you'll see some error messages from wr, and you'll have to resort to using the --poll option, which uses a polling file watcher.