pmuellr: dustup

A few months ago, I took a look at Nathan Marz's Storm project. As someone who has used and implemented Actor-like systems a few times over the years, always fun to see another take. Although Storm is aimed at the "cloud", it's easy to look at it and see how you might be able to use it for local computation. In fact, I have a problem area I'm working in right now where something like Storm might be useful, but I just need it to work in a single process. In JavaScript.

Another recent problem area I've been looking at is async, and using Promises to help with that mess. I've been looking at Kris Kowal's q specifically. It's a nice package, and there's a lot to love. But as I was thinking about how I was going to end up using them, I imagined building up these chained promises over and over again for some of my processing. As live objects. Which, on the face of it, is insane. If I can build a static structure and have object flow through them instead. Should be able to cut down on the amount of live objects created, and it will probably be easier to debug.

So, with that in mind, I finally sat down and reinterpreted Storm this evening, with my dustup project.

Some differences from Storm, besides the super obvious ones (JavaScript vs Java/Clojure, and local vs distributed):

One of the things that didn't seem right to me in Storm was differentiating spouts from bolts. So I only have bolts.
I'm also a fan of the PureData real-time graphical programming environment, and so borrowed some ideas from there. Namely, that bolts should have inlets where data comes in, and outlets where they go out, and that to hook bolts together, that you'll connect an inlet to an outlet.
Designed with CoffeeScript in mind. So you can do nice looking things like this:
```
x = new Bolt
    outlets:
       stdout: "standard output"
       stderr: "standard error"
```
which generates the following JavaScript:
```
x = new Bolt({
    outlets: {
       stdout: "standard output",
       stderr: "standard error"
    }
})
```
and this:
```
Bolt.connect a: boltA, b: boltB
```
which generates the following JavaScript
```
Bolt.connect({a: boltA, b: boltB})
```
Even though I designed it for and with CoffeeScript, you can of course use it in JavaScript, and it seems totally survivable.

Stopping there left me with something that seems useful, and still small. I hope to play with it in anger over the next week or so.

Here's an example of a topology that just passes data from one bolt to another, in CoffeeScript:
(here's the JavaScript for you CoffeeScript haters)

In the lines 1-2, we get access to the dustup package and the Bolt class that it exports.

On lines 4-6, we're creating a new bolt that only has a single outlet, named "a" (the string value associated with the property a - "outlet a" - is ignored).

On lines 8-11, we're creating a new bolt that only has a single inlet, named "b". Inlets need to provide a function which will be invoked when the inlet receives data from an outlet. In this case, the inlet function will write the data to "the console".

On line 13, we connect the a outlet from the first bolt to the b inlet of the second bolt. So that when we send a message out the a outlet, the b inlet will receive it.

Which we test on line 15, by having the first bolt emit some data. Guess what happens.

next

The next obvious thing to look at, is to see how asynchronous functions, timers, and intervals fit into this scheme.

pmuellr

Links

Thursday, February 07, 2013

dustup