setting up Mercurial on TextDrive with ssh

Mercury, the Winged Messenger
© Toshiyuki IMAI

I've been thinking about setting up a simpler, change-set based source code repository for my little personal projects for a while now. For the past six months or so I've been using SVN on my TextDrive account but ... looking for something different.

I ran across Mercurial (hg) at least twice now recently, so that speaks to me "check it out". Getting it set up on my local box was straightforward, and of course the next step is getting it set up on your server. I host at TextDrive. And Bill de hÓra recently posted some instructions on how to do just that. Kind of.

The problem with Bill's instructions are that you're setting it up to run using Basic Auth but using http instead of https. Bill points out himself, that "This isn't secure". Using https with TextDrive is possible, but at times confusing. Luckily, there is a fairly straight-forward way to get to a secure solution.

The basic idea is to use the ssh instead of http for when accessing your hg repositories programmatically. Instructions on using ssh with hg are available in the hg(1) man page.

In a nutshell, follow Bill's instructions, with a few slight changes, and it just works, assuming you have ssh set up in the first place.

If this doesn't work, ping me; I perhaps forgot a step.

I also created a ~.hgrc file on my client box, with the following contents. The ssh setting causes ssh to use compression when transfering the data/files.

    [ui]
    username = your name <yourname@yourmail>
    ssh = ssh -C

The only thing I don't like about the resulting set up is having to update the hgweb.config file on the server whenever you create a new repository. Though I suppose it's nice that you can selectively make repositories easily publicly accessible or not this way. I don't think I'd rely on this trick to hide something.

I copied Bill's instructions, and then augmented them with the changes I made, which you can find here. I would have included them in this blog entry, but Roller decided to strip all the nice formatting I did out my post. And that's just unacceptable, given the stupid amount of time I spent adding it in, in the first place.

twit-growl woops

woops
© Andy Smith

I would be remiss in not pointing out a huge problem I recently found with my twit-growl program that I previously blogged about.

The problem is that the program is a fairly simple python script that uses some 'command line' programs to do all it's heavy lifting; curl and growl-notify. To invoke these, I was using os.system() to invoke the program, building up a big command-line as the parameter. As the programs take parameters which are coming from the data being downloaded, I was trying to be careful to escape the data when constructing the command lines.

Unfortunately, I'm an idiot, and didn't understand the escaping rules. In particular, I wasn't taking into account backquote substitution.

So, I was really suprised one day, when I got a message displayed in growl that was humongous; and the tail end of of the message was a wad of html. Going to the twitter site, to see the message there, I immediately realized what happened. The twitter message included some text like this:

... `curl http://www.yahoo.com`

Woops.

Got that fixed up pretty quickly, by avoiding the use of os.system() and using an os.spawn() variant instead, which (hopefully) avoids the shell completely.

BTW, I continue to use my twit-growl to get 'popups' for incoming tweets, and generally use the wonderful tweetbar Firefox extension for posting, and reviewing tweets en masse.

json schema

schema tic
© Jussi You-S-See

James Clark has a great post on XML and JSON. But I have one point to pick at.

"JSON's primitive datatype support is weak."

First, the number issue. If you assume JSON datatypes map into JavaScript ones, I believe the JavaScript number type does in fact have a very specific representation. I believe, a 64 bit floating point number. Even for integers. I'd love to point to a reference for this, but I don't have one handy, and the one at mozdev is insufficient.

Note that outside of JavaScript, for languages that treat integers and floating point as separate types, a parser can obviously distinguish between integers and floating points and 'do the right thing' in terms of mapping the JSON value into the appropriate type.

Next, "the set of primitive datatypes is not extensible". But this is true for XML itself. Until you mix in XML schema. It's schema that allows you to interpret a string in XML as some sort of other data type. JSON has no standard schema. Yet. It's easy to imagine though. Defined in JSON, of course.

json array exploit

exploited
© Radoslaw Stypczynski

Unless you've been living in a cave for the last week or so, you are familiar with JSON exploit as publicized by Fortify Software. I believe this is the same exploit I first read about on Robert Yates' blog. Original reference was from Joe Walker.

I'm not going to claim to be an expert on this particular exploit, but if I understand the situation correctly, I have a few issues with the current concerns.

1. It's clearly not just JSON data which is suspect here. Anything which appears to be a valid (or maybe even somewhat valid) chunk of JavaScript, which someone could access via <script src=> is suspect. How many resources like this might be available at your site? How 'valid' (in terms of valid JavaScript) does it have to be? Only folks who are intimately familiar with the code of the JavaScript interpreters we use, can say for sure. Let's also not forget E4X; could your XML data be suspect?

2. This appears to be a problem today just for Mozilla / Firefox. And I believe the reason is the advanced functionality it provides in JavaScript with coolio getter/setter capabilities. (BTW, ActionScript 3 also supports getter/setter capabilities). Is this actually a bug? Doesn't seem like it. Is there some chance this functionality could be locked down during script/src inclusion? Seems unlikely (slippery slope; what else would you lock down?). Could the enhanced functionality be turned off for user-land scripts? Probably; I don't suspect too many people are actually making use of the functionality. There's an open bug @ Mozilla on the issue that has additional thoughts.

3. I'm clearly showing my n00biness here, but why aren't even simpler exploits possible, on all the browsers? Before doing a script/src, redefine a well-known object/function with one that does some hijacking? The security model for JavaScript is complex; so complex; too complex.

4. Ted Husted pointed out that one way to fix this problem is to enclose your JSON in comment characters, and then strip the comments before parsing the JSON. That's great, unless I want to actually include comments inside my JSON. And assuming that JavaScript comments can't be 'recursive' (not specified at moz's js doc). Why would I want comments in my JSON? Why not? If you can have them in XML, I'm not sure why we won't someday want them in JSON. Note that I don't consider RFC 4627 to be the final statement on JSON syntax.

5. The fact that JSON is also a syntactically valid chunk of JavaScript is smelling worse and worse all the time. I can't imagine a situation where I'd eval() JSON data in a production app; it's just so much safer to parse the data yourself. So, why don't we use this opportunity to come up with YASDN (yet another simple data notation)? JSON is great because it's readable, but I've already pointed out how we could make it even MORE readable.

6. Isn't it about time we got a little better modularity available in JavaScript, beyond the script/src story. Which is more or less the same thing as building a C program in one big monolithic file that pulls in functionality with #include. So 1970's ...

How Twitter is Different

Humans Leave Many Different Tracks
© Silus Grok

Now that I've been using Twitter for a few days, I thought I'd note how it's different from other forms of communication I'm used to using. Comparing to things like irc, instant messaging (im), email, mailing lists, newsgroups, etc.

One very noticeable difference is in the 140 character limit on the messages. This is due to the fact that most people (I guess) are twittering via SMS on their phones, which has a limit of 140 characters per SMS message. Actually, is that 140 characters, or bytes? What's the character encoding story? hmmmm ...

This limit if far less than most other communication messages. Which forces various forms of conservatism, like using TinyURL for any URLs you want to 'send'.

Twitter is also, generally, not directed. You are talking to your 'friends' and/or 'followers' (the distinction has not stuck with me yet). In IRC terms, you could think of this as being a separate channel that you are sending your messages out on; but it's not neccessarily two-way. You might send a message that a 'follower' sees, but if you aren't set up to see their messages, you won't see them.

Twitter is not as intrusive as im or irc. With im, I can actually see who's online, and even perhaps how long they've been online or offline. I'm not sure if it was a bug or feature, but in a version of Sametime 7.5 that I used, it would actually popup a messaging window when someone started sending you a message, but before they actually sent the first one. It was fun to freak people out, when no one knew about this, by getting a message to them first. "Hey, I have this weird feeling that you want to ask me something". :-) But the fact is, I think im in general exposes a bit too much information.

Twitter is persistant. When I send a message, it's out there for everyone to eventually see, even if they aren't online to see it. Compare to irc, where if you aren't online (and don't have easy access to irc logs), you missed out on the conversation. Mailing lists have the same poor quality; it's easy to join a mailing list, but getting access to previous conversations can be painful. Hint: use gmane.

Twitter tends to be personal. "That was a delicious ham sandwich I just had!". It's easy enough to believe that this would make some folks non-interested in the product. However, the content which people are providing today is not really a telling indicator of how this might be useful in the future. For instance, my first foray into NNTP (or it might have been NetNews then), in the mid 80's, was rec.humor. Today, NNTP is an extremely valuable communication channel for non-personal, technical information.

Twitter has an insanely easy API. I was able to whip up a little 'reader' app for it in 1.5 hours. And that involved dusting off my python skills, figuring out how to talk to Growl, and dealing with cron. Compare with any of the other channels, and, it's night and day. I've looked at XMMP a few times, and it's not insanely easy. I've also written code to deal with many of the other channels, reading and writing, including email and nntp clients and servers, etc. Twitter is just plain simple in comparison to all of them.

I think the technology here is interesting. Perhaps as it gets integrated more into the fabric of our other communication channels, and daily-use tools, we'll find that it's a useful niche for certain styles of communication. I'm bullish on it.

Perhaps we can look at expanding this out a bit more. It would be nice to be able to support > 140 characters per message. It would be nice to be able to support sending hypertext of some kind. It would be nice to be able to support sending structured data. It would be nice to have more 'endpoints' than just twitter.com. And do we need to start thinking about spam?

Of particular interest to me, is the use of automated sending of messages from other programs, like continuous builds. And then the automated reception of messages. Jazz folks, take note.

Updates:

2007/04/02: fixed a broken link to Jazz.

ActionScript Typing

Type
© Tony Unruh

In the blog post, "The Open Web and Its Adversaries", Brendan Eich writes "If I were VP of Engineering in a 10,000 person company, I would want the security blanket of the C-like syntax and a static type system for a well-known, big-bell-curve language like AS3, C#, or Java." It's interesting that he's lumping ActionScript 3 with C# and Java as 'strictly typed' languages, since ActionScript is an evolution of JavaScript, which is not strictly typed.

It's true that the documentation for ActionScript 3 uses the 'typing' notation everywhere. Here's an example from one of the Hello World samples.

	package { 
		public class Greeter { 
			public function sayHello():String  { 
				var greeting:String; 
				greeting = "Hello World!"; 
				return greeting; 
			} 
		} 
	} 

Now compare with the 'untyped' version, in which I also took the liberty in removing the completely unneccesary semicolons. One of my pet-peeves with JavaScript: You almost never need to use semicolons in JavaScript, but everyone does!

	package { 
		public class Greeter { 
			public function sayHello()  { 
				var greeting 
				greeting = "Hello World!"
				return greeting
			} 
		} 
	} 

To prevent the compiler from complaining about the missing type information, use the mxmlc option -compiler.warn-no-type-decl=false when compiling. I tried playing with some options to try to get non-typing warnings flagged as errors, instead of warnings ... and I couldn't. The compiler would complain without the option specified above, but always produced the .swf file in the end.

This example doesn't really do justice to the impacts of typing things. But I think you can imagine how much more un-scannable your code is going to be with typing information littered throughout.

What's the rationale for doing all this typing? From the "Programming ActionScript 3.0" document: "In ActionScript 3.0, type information is preserved at run time, and used for a number of purposes. Flash Player 9 performs run-time type checking, improving the system's type safety. Type information is also used to represent variables in native machine representations, improving performance and reducing memory usage."

So typing helps with two things: 'improving type safety' and 'improving performance'.

We've beaten the type safety issue into the ground over the years; I won't touch that one. But performance? This sounds like premature optimization to me. As in premature optimization is the root of all evil.

May I suggest a comprise: use typing when it provides the convenient side-effect of documentation, and use typing when you need to optimize something. In terms of the documentation-style typing, I'm thinking the typing of 'fields' in an object, and the parameters and return types of methds. In fact, if you just did that, couldn't the compiler infer, to a large extent, the types of local variables?

Brendan also seems to be implying that corporations as a whole seem to prefer to deal with strictly typed languages, compared to 'looser' typed languages. Implying that ActionScript 3, and thus Flex, and thus Apollo, are all being aimed at corporations instead of a more general audience. Assuming this, and combined with the pricing model for tools, which I've already blogged about and ... yeah, you could see this as being marketed at corporations.

I'm constantly reminded of the programming model for the Palm Pilot, since I use mine daily (but only for reading on my walks, using the fantastic iSilo). One of the reasons for the Palm's popularity was that approachable tools were made available to the general public, which allowed for a flood of software programs to be released for that platform.

If you want wild success for your programmable product, make sure your target hacking audience is as wide as you can get.

Slingshot

The Human Slingshot
© Djenan Kozic

Yesterday, Joyent and Magnetk introduced a new 'product' (or is it a 'platform'?) called Slingshot. From the Joyent blog: "Joyent Slingshot allows developers to deploy Rails applications that work the same online and offline (with synchronization) and with drag into and out of the application just like a standard desktop application."

This is pretty cool. The story is: develop your Rails application like normal, add some special processing to handle sync'ing data between an off-line and on-line mode, and you can run your app in a local browser. Well, not quite a local browser. It's actually an application 'shell' which has an HTML widget as it's client area. But it can do more than your average browser; it has additional desktop-y capabilities available to it, like drag-n-drop, native file-system integration, etc.

That last bit: 'more than the average browser', is what I really like. It's actually more, and less. Watch the movie on the Joyent site. Where are the address bar, and Home, Back, and Forward buttons? Banished. As they should be. At least for 'apps'. They're great for hypertext.

I suppose there will be the eventual question about security. I'm guessing this is a fairly safe environment, since the opportunity to run someone else's code is very constrainable. Don't add links to someone else's pages in your app. Constrained, for security, but a loss of function as well. Probably a reasonable trade-off.

As other folks have noted, this seems to compete with Apollo. Excellent. Competition.

Can't wait to see more technical details. Dare we hope for some 'open-ness'?

Lastly, I kind of wonder how far you could take Eclipse (or even just SWT) to build something like this; use the HTML widget and see how far you can extend/integrate it into an application. If I wasn't tied up in server-y goop right now, I'd be poking around there. And what ever happened to XULRunner (last update: 10-Jun-2006)?

Wake County Library Lookup

Central Library Interior #2
© Light Gazer

In my previous post regarding using libraries, I mentioned using Josh Staiger's Wake County Library Lookup Greasemonkey script, which annotates Amazon.com pages with links to the library, if the library has the book in it's collection.

What I neglected to mention is that I have been having problems with the script for the past few weeks. One problem was that a number of Amazon pages no longer contained the library links at all; I think this is due to Amazon's new URL scheme. The ISBN was being pulled from the URL, and it's not always there anymore, or in a different place. The search for the ISBN has been expanded so that now more books will be candidates for library lookup.

The other problem was that the page layout of the Amazon pages changed, such that the links Josh was adding were no longer appearing. Dark XPath and DOM hackery. ~shiver~ I fixed that by not placing the library information in-line with the page the way Josh was doing, but placing it at the top of the page. A bit icky, but it's pretty much guaranteed that the links will show up now; the script is not currently dependent on the shape of the HTML on the page.

I talked to Josh about the issues earlier this week, and he mentioned he hasn't been keeping a close eye on the script, because he doesn't actually live in Wake County anymore. Although he has been making some updates, as recent as this January. What a trooper!

So it only makes sense at this point to fork and maintain a new version of the code, since Josh really doesn't have a reason to do this anymore. My version of the script is here. It's highly likely that I won't be updating it, till Amazon or Wake County make enough changes to their pages to break the script again. Which is pretty much guaranteed. The perils of screen scraping. But I'm certainly open for any thoughts on the functionality.

Thanks for putting the time into this Josh!

twit-growl

So, I'm not totally convinced on the Twitter thing yet, but I'm an information junkie, so how can I resist?

Twitter will feed you messages via SMS, IM, or you can go to the website. However, I don't do SMS, the IM has been broken since I started playing with Twitter, and the web site is a web site. I don't really 'check' web sites anymore.

So, I was thinking on the drive home from work that I should hook Twitter up with Growl. In fact, I figured someone must have already done this. And they have. The first couple I looked at on google were Ruby, and didn't seem fully operational, so I thought I'd write one. And here it is. It's rough; you'll have to install a Python library, and install wget. You can do it.

Took a total of 1.5 hours; my Python is a bit rusty; had to consult the manual constantly. But there's something just fun about writing Python; I should do more.

Tivo'd baseball?

Free Books!

Free_books
© Linda

Like Bill, I've had a problem accumulating books, but I've been much better about this for the last few years. I was using Safari Books Online for a while, but didn't really use it enough to justify the $120/year or whatever it was. It appears the price has gone up to $20/month, which is double what I think I used to pay; definitely can't justify that.

A handy little Greasemonkey script from my new next-door cube-mate, Josh Staiger, Wake County Library Lookup, opened my eyes up to a free resource: my local public library. (Thank you Ben Franklin!) If you don't happen to live in Wake County, North Carolina, perhaps you can find a Greasemonkey for your library, here.

I've been a life-long library user, so it's not like my eyes were just opened up to the possiblity of free books, but most libraries don't have a great selection of tech books, or at least recently available tech books. There are options; you're a tax-payer (probably), so why not ask your library to buy a book you want. For Wake County Public Libraries, head over to the Adult Services section at the library, and ask the librarian about this. They'll give you a card to fill out; ours wants to know, along with the usual stuff, the publisher, publication date, and ISBN. Stuff you can get from Amazon.

In lieu of ordering the book, at Wake County you can make a request for an InterLibrary Loan, where the library will attempt to get the book from a different library system. You can do this from your comfy chair at home, via the web. Our library is great; I request books from their (fairly yucky) web interface, they email me when the book is ready to be picked up. Head over to the library, walk over to 'my' shelf and find the labelled book, and checkout.

I just checked out Pragmatic Project Automation: How to Build, Deploy, and Monitor Java Apps, from my library this afternoon. It happened to be the only book, besides the GTD book, on Bill de hÓra's Ten books for the working programmer, which my library had available. I put in an InterLibrary Loan request for another one.

BTW, I just created an Amazon list for Bill's top ten list, plus some of the other books mentioned in the comments.

Of course, sometimes you need to own the book. But many times, you don't. You just need to read it. Or you might want to preview it, before actually purchasing it. A more extended preview than you can get from a visit to your local bookseller.