Futures example or documentation?

[hbf]

Dear all,

I have seen some posts on the forum that mention the dispatch-futures. Are there some more examples or documentation available. I want to avoid concurrency problems (dirty connections being reused, etc.) and am looking for some best practices.

I suppose I need a ThreadPoolExecutor (with an unbounded queue) and tell dispatch to use this one. Then any requests submitted via dispatch's futures will be executed in parallel or put in the queue to be executed when threads become available, right?

Not knowing about Java NIO I am wondering whether NIO is related in any way?

Many thanks,
Kaspar

[hbf]

Hi everybody,

Here is a first attempt:

package org.hbf.util.throttler;

import org.junit.runner.RunWith
import org.scalatest.junit.JUnitRunner
import org.scalatest.FlatSpec
import org.scalatest.matchers.ShouldMatchers
import scala.xml._
import dispatch._
import java.lang.Thread._

@RunWith(classOf[JUnitRunner])
class DispatchTest extends FlatSpec with ShouldMatchers {
  "Scraper" should "fulfil requests for less than 100 hits" in {
    val http = new thread.Http
    println(currentThread)
    http {
      url("http://www.google.com/") >> { h =>
        {
          Thread.sleep(2000)
          println(currentThread)
        }
      }
    }
    http {
      url("http://www.google.com/") >> { h =>
        {
          Thread.sleep(2000)
          println(currentThread)
        }
      }
    }
    Thread.sleep(4000)
    http.shutdown()
    println("Shut down.")
  }
}

I have the following questions:

1. How can I wait until all threads have finished? I'd need to call shutdown() and awaitTermination() on the underlying ThreadPoolExecutor. (http.shutdown() shuts down the connection manager only IIUC and therefore, in the above code without the "Thread.sleep(4000)" statement, not all futures will necessarily terminate.)

2. Am I correct that now matter how many dispatch.thread.Http clients I construct, they all use the same thread pool?

Many thanks for any feedback,
Kaspar

[n8han]

On 05/03/2011 11:53 AM, hbf [via Databinder] wrote:
> I have seen some posts on the forum that mention the dispatch-futures.
> Are there some more examples or documentation available.

I'm working on that now (if anyone can believe it). It's hard to know
where to start but the initial version of it will at least have examples
for the same request carried out with a blocking interface, a threaded
future, and an nio future.

> I want to avoid concurrency problems (dirty connections being reused,
> etc.) and am looking for some best practices.

As far as I could tell, that resulted from a bug in the version of
httpclient included in Android. In general I think the
ThreadSafeClientConnManager is widely and heavily used enough to trust.

> I suppose I need a ThreadPoolExecutor (with an unbounded queue) and
> tell dispatch to use this one. Then any requests submitted via
> dispatch's futures will be executed in parallel or put in the queue to
> be executed when threads become available, right?

There is a default thread pool executor [1] but if it is not suitable
you can construct your own.

[1]:
http://sourced.implicit.ly/net.databinder/dispatch-futures/0.8.0/Futures.scala.html#9079

> Not knowing about Java NIO I am wondering whether NIO is related in
> any way?

It's an option. dispatch.nio.Http uses Apache's HttpAsyncClient which
still employs a thread pool but does not require one thread per open
connection. Applications that could require a massive number of open
connections are probably better off with this implementation, with the
caveat that it is new and relatively unproven compared to the
thread-blocking client.

Also, standard Java IO has been shown to have greater throughput than
Java NIO; there's a fair bit of debate on the web about this. If you
decide to build for the NIO version, I would recommend using it
abstractly enough that you can swap in the threaded variant for
comparative benchmarking and as an emergency fallback.

Nathan

[n8han]

On 05/04/2011 07:33 AM, hbf [via Databinder] wrote:

...

I have the following questions:

1. How can I wait until all threads have finished? I'd need to call shutdown() and awaitTermination() on the underlying ThreadPoolExecutor. (http.shutdown() shuts down the connection manager only IIUC and therefore, in the above code without the "Thread.sleep(4000)" statement, not all futures will necessarily terminate.)

You'll need to keep a reference to the returned Future to do this. e.g.

(1 to 5) map { _ =>
  http { ... }
} foreach { _() }

The apply method on Dispatch's Future type [1] waits for the value produced by the handler to be available.

[1]: http://sourced.implicit.ly/net.databinder/dispatch-futures/0.8.0/Futures.scala.html#9507

2. Am I correct that now matter how many dispatch.thread.Http clients I construct, they all use the same thread pool?

This is the default behavior, yes. The Future trait [2] delegates to a DefaultFuture object [3] which initializes one thread pool.

[2]: http://sourced.implicit.ly/net.databinder/dispatch-http/0.8.0/thread/thread.scala.html#21083
[3]: http://sourced.implicit.ly/net.databinder/dispatch-futures/0.8.0/Futures.scala.html#9079/Futures.scala.html#9079

Nathan

[hbf]

Nathan, thank you very much for your clear explanations.

Best,
Kaspar