Http.>> and other Handler-generators never return when resource is super-long (or infinite)

[bmaso]

For example, the following code never returns from http.apply() because the targeted resource is a Twitter Stream API URL, whose content is essentially infinitely long:

----------------------------------------------------------
import dispatch._
import Http._

val http = new Http
  //...using basic auth for simplicity -- could always subst oauth or xauth...
val sampleStream = :/("stream.twitter.com") / "1" / "statuses" / "sample.json" as ("username", "password")

  //...the following line never returns...
val is = http(sampleStream >> {(is,_) => is})
-----------------------------------------------------------

I think the culprit is line 96 in Http.scala (using Dispatch version 0.7.5). The line invokes HttpEntity.consumeContent() whenever a Handler's block finishes executing. For an infinitely-long resource, this call never returns. I believe HttpEntity is attempting to consume the content in case the HTTP connect is reusable with keep-alive. But of course since the resource is infinite it can never be completely consumed.

Possible solution:

* Perhaps a way to "mark" a Request object so that consumeContent() would never be called on a resulting HttpEntity? This would avoid the problem.
* Maybe there already is an API in HttpClient to short-circuit consumeContent()?

Best regards,
Brian Maso
[phone]
[e-mail]