Hopefully everyone who builds stuff for the web knows about gzip compression available in HTTP. Here's a quick intro if you don't: https://developers.google.com/speed/articles/gzip
I use connect or express when building web servers in node, and you can use the compress middleware to have your content gzip'd (or deflate'd), like in this little snippet.
However ...
Let's think about what's going on here. When you use the compress middleware like this, for every response that sends compressable content, your content will be compressed. Every time. Of course, for your "static" resources, the result of that compression is the same every time, and so for those resources, it's really kind of pointless to run the compression for each request. You should do it once, and then reuse that compressed result for future requests.
Here are some tests using the play Waste, by Harley Granville-Barker. I pulled the HTML version of the file, and then also gzip'd the file manually from the command-line for one of tests.
The HTML file is ~300 KB. The gzip'd version is ~90 KB.
And here's a server I built to serve the files:
The server runs on 3 different HTTP ports, each one serving the file, but in a different way.
Port 4000 serves the HTML file with no compression.
Port 4001 serves the HTML file with the compress middleware
Port 4002 serves the the pre-gzip'd version of the file that I stored in
a separate directory; the original file was waste.html, but the gzip'd
version is in gz/waste.html. It checks the incoming request to see
if a gzip'd version of the file exists (caching that result), and
internally redirects the server to that file by resetting request.url.
Setting the appropriate Content-Encoding, etc headers.
What a hack! Not
quite sure that "fixing" request.url is kosher, but, worked great for
this test.
Here's some curl invocations.
$ curl --compressed --output /dev/null --dump-header -
--write-out "%{size_download} bytes" http://localhost:4000/waste.html
X-Powered-By: Express
Accept-Ranges: bytes
ETag: "305826-1384296482000"
Date: Wed, 13 Nov 2013 01:21:13 GMT
Cache-Control: public, max-age=0
Last-Modified: Tue, 12 Nov 2013 22:48:02 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 305826
Connection: keep-alive
305826 bytes
Looks normal.
$ curl --compressed --output /dev/null --dump-header -
--write-out "%{size_download} bytes" http://localhost:4001/waste.html
X-Powered-By: Express
Accept-Ranges: bytes
ETag: "305826-1384296482000"
Date: Wed, 13 Nov 2013 01:21:13 GMT
Cache-Control: public, max-age=0
Last-Modified: Tue, 12 Nov 2013 22:48:02 GMT
Content-Type: text/html; charset=UTF-8
Vary: Accept-Encoding
Content-Encoding: gzip
Connection: keep-alive
Transfer-Encoding: chunked
91071 bytes
Nice seeing the Content-Encoding and Vary headers, along with the reduced
download size. But look ma, no Content-Length header; instead the content
comes down chunked, as you would expect with a server-processed output stream.
$ curl --compressed --output /dev/null --dump-header -
--write-out "%{size_download} bytes" http://localhost:4002/waste.html
X-Powered-By: Express
Content-Encoding: gzip
Vary: Accept-Encoding
Accept-Ranges: bytes
ETag: "90711-1384297654000"
Date: Wed, 13 Nov 2013 01:21:13 GMT
Cache-Control: public, max-age=0
Last-Modified: Tue, 12 Nov 2013 23:07:34 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 90711
Connection: keep-alive
90711 bytes
Like the gzip'd version above, but this one has a Content-Length!
Here are some contrived, useless benches using wrk, that confirm my fears.
$ wrk --connections 100 --duration 10s --threads 10
--header "Accept-Encoding: gzip" http://localhost:4000/waste.html
Running 10s test @ http://localhost:4000/waste.html
10 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 71.72ms 15.64ms 101.74ms 69.82%
Req/Sec 139.91 10.52 187.00 87.24%
13810 requests in 10.00s, 3.95GB read
Requests/sec: 1380.67
Transfer/sec: 404.87MB
$ wrk --connections 100 --duration 10s --threads 10
--header "Accept-Encoding: gzip" http://localhost:4001/waste.html
Running 10s test @ http://localhost:4001/waste.html
10 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 431.76ms 20.27ms 493.16ms 63.89%
Req/Sec 22.47 3.80 30.00 80.00%
2248 requests in 10.00s, 199.27MB read
Requests/sec: 224.70
Transfer/sec: 19.92MB
$ wrk --connections 100 --duration 10s --threads 10
--header "Accept-Encoding: gzip" http://localhost:4002/waste.html
Running 10s test @ http://localhost:4002/waste.html
10 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 48.11ms 10.66ms 72.33ms 67.39%
Req/Sec 209.46 24.30 264.00 81.07%
20795 requests in 10.01s, 1.76GB read
Requests/sec: 2078.08
Transfer/sec: 180.47MB
Funny to note that the server using compress middleware actually handles less requests/sec than the one that doesn't compress at all. But this is a localhost test so the network bandwidth/throughput isn't realistic. Still, makes ya think.
