Handlers.>~ not compatible with infinitely-long resource bodies

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Handlers.>~ not compatible with infinitely-long resource bodies

bmaso
Handlers.>~() attempts to create a Source from the body of a HttpResponse by consuming the entire resource in to an in-memory String, and wrapping the in-memory copy with a Source (using Source.fromString()). Obviously for an infinitely-long resource (such as one of the Twitter Streamed API resources), this is impossible. For example, the Handler block in the following code snippet never gets executed:

-------------------------------------------------
import dispatch._
import Http._

val http = new Http
val twitterSampleStream = :/("stream.twitter.com") / "1" / "statuses" / "sample.json" as ("username", "password")

http(twitterSampleStream >~ {ssrc => src.getLines foreach {l => println(l)}})
-------------------------------------------------

Construction of the "src" object takes infinitely long, I think because the Http.scala code attempting to consume the entire resource (lines 321-329 Http.scala, Dispatch version 0.7.5).

Looks like line 320, currently commented out, attempts to use Source.fromInputStream() from the 2.8 API. Since 0.7.5 is marked in the maven repo as artifact "dispatch-http_2.8.0" we could go ahead and use the 2.8 API.

Best regards,
Brian Maso
(949) 395-8551
brian@blumenfeld-maso.com
Reply | Threaded
Open this post in threaded view
|

Re: Handlers.>~ not compatible with infinitely-long resource bodies

n8han
Administrator
Correct, >~ is not going to work with the streaming API or anything that produces a gigantic response, until the 2.7/2.8 problem is resolved. And although Dispatch does (and must) produce specific artifacts for each version of Scala, it currently isn't possible to supply different source files for each one.

If you want to use Source it's not difficult. You can apply >> and pass the input stream it gives you to Source.fromInputStream yourself.

The handler and consumeEntity problem is where things get interesting. To consume this indefinitely long stream you are going to need some kind of threading model within your application, and in that case a function that does not return (until an error occurs, or you abort it) is manageable. Instead of trying to get back a reference to a stream that needs to be manually closed, you can pass a function into the handler that contains everything you want to do with that stream reference. e.g.

  http(Streaming.statuses >> some.package.SomeObject.myStreamHandler)

And since that won't return, you can wrap the above in

    dispatch.futures.DefaultFuture.future { ... }

(which uses the very stable java.util.concurrent futures implementation) or some other threading structure. I wouldn't recommend using "Http with Threads" initially as that activates a different HttpClient connection manager which hasn't been as forgiving as the standard connection manager in my experience. It's safe to use the standard Http in a thread/future/actor as long as you use the instance only within that thread.

For the oauth access, I will definitely do some tracing to see what earlybird is doing differently. Thanks for figuring out what the kinks are in their setup.

Nathan


On 8/23/10 1:45 , bmaso [via Databinder] wrote:
Handlers.>~() attempts to create a Source from the body of a HttpResponse by consuming the entire resource in to an in-memory String, and wrapping the in-memory copy with a Source (using Source.fromString()). Obviously for an infinitely-long resource (such as one of the Twitter Streamed API resources), this is impossible. For example, the Handler block in the following code snippet never gets executed:

-------------------------------------------------
import dispatch._
import Http._

val http = new Http
val twitterSampleStream = :/("stream.twitter.com") / "1" / "statuses" / "sample.json" as ("username", "password")

http(twitterSampleStream >~ {ssrc => src.getLines foreach {l => println(l)}})
-------------------------------------------------

Construction of the "src" object takes infinitely long, I think because the Http.scala code attempting to consume the entire resource (lines 321-329 Http.scala, Dispatch version 0.7.5).

Looks like line 320, currently commented out, attempts to use Source.fromInputStream() from the 2.8 API. Since 0.7.5 is marked in the maven repo as artifact "dispatch-http_2.8.0" we could go ahead and use the 2.8 API.

Best regards,
Brian Maso
(949) 395-8551
[hidden email]




Reply | Threaded
Open this post in threaded view
|

Re: Handlers.>~ not compatible with infinitely-long resource bodies

bmaso
Worked like a charm! Thanks for the tip about DefaultFuture.future -- I hadn't looked at that before.

Best regards,
Brian Maso
(949) 395-8551
brian@blumenfeld-maso.com