Dear all,
I have seen some posts on the forum that mention the dispatch-futures. Are there some more examples or documentation available. I want to avoid concurrency problems (dirty connections being reused, etc.) and am looking for some best practices. I suppose I need a ThreadPoolExecutor (with an unbounded queue) and tell dispatch to use this one. Then any requests submitted via dispatch's futures will be executed in parallel or put in the queue to be executed when threads become available, right? Not knowing about Java NIO I am wondering whether NIO is related in any way? Many thanks, Kaspar |
Hi everybody,
Here is a first attempt: package org.hbf.util.throttler; import org.junit.runner.RunWith import org.scalatest.junit.JUnitRunner import org.scalatest.FlatSpec import org.scalatest.matchers.ShouldMatchers import scala.xml._ import dispatch._ import java.lang.Thread._ @RunWith(classOf[JUnitRunner]) class DispatchTest extends FlatSpec with ShouldMatchers { "Scraper" should "fulfil requests for less than 100 hits" in { val http = new thread.Http println(currentThread) http { url("http://www.google.com/") >> { h => { Thread.sleep(2000) println(currentThread) } } } http { url("http://www.google.com/") >> { h => { Thread.sleep(2000) println(currentThread) } } } Thread.sleep(4000) http.shutdown() println("Shut down.") } } I have the following questions: 1. How can I wait until all threads have finished? I'd need to call shutdown() and awaitTermination() on the underlying ThreadPoolExecutor. (http.shutdown() shuts down the connection manager only IIUC and therefore, in the above code without the "Thread.sleep(4000)" statement, not all futures will necessarily terminate.) 2. Am I correct that now matter how many dispatch.thread.Http clients I construct, they all use the same thread pool? Many thanks for any feedback, Kaspar |
Administrator
|
In reply to this post by hbf
On 05/03/2011 11:53 AM, hbf [via Databinder] wrote:
> I have seen some posts on the forum that mention the dispatch-futures. > Are there some more examples or documentation available. I'm working on that now (if anyone can believe it). It's hard to know where to start but the initial version of it will at least have examples for the same request carried out with a blocking interface, a threaded future, and an nio future. > I want to avoid concurrency problems (dirty connections being reused, > etc.) and am looking for some best practices. As far as I could tell, that resulted from a bug in the version of httpclient included in Android. In general I think the ThreadSafeClientConnManager is widely and heavily used enough to trust. > I suppose I need a ThreadPoolExecutor (with an unbounded queue) and > tell dispatch to use this one. Then any requests submitted via > dispatch's futures will be executed in parallel or put in the queue to > be executed when threads become available, right? There is a default thread pool executor [1] but if it is not suitable you can construct your own. [1]: http://sourced.implicit.ly/net.databinder/dispatch-futures/0.8.0/Futures.scala.html#9079 > Not knowing about Java NIO I am wondering whether NIO is related in > any way? It's an option. dispatch.nio.Http uses Apache's HttpAsyncClient which still employs a thread pool but does not require one thread per open connection. Applications that could require a massive number of open connections are probably better off with this implementation, with the caveat that it is new and relatively unproven compared to the thread-blocking client. Also, standard Java IO has been shown to have greater throughput than Java NIO; there's a fair bit of debate on the web about this. If you decide to build for the NIO version, I would recommend using it abstractly enough that you can swap in the threaded variant for comparative benchmarking and as an emergency fallback. Nathan |
Administrator
|
In reply to this post by hbf
... You'll need to keep a reference to the returned Future to do this. e.g. (1 to 5) map { _ => http { ... } } foreach { _() } The apply method on Dispatch's Future type [1] waits for the value produced by the handler to be available. [1]: http://sourced.implicit.ly/net.databinder/dispatch-futures/0.8.0/Futures.scala.html#9507 2. Am I correct that now matter how many dispatch.thread.Http clients I construct, they all use the same thread pool? This is the default behavior, yes. The Future trait [2] delegates to a DefaultFuture object [3] which initializes one thread pool. [2]: http://sourced.implicit.ly/net.databinder/dispatch-http/0.8.0/thread/thread.scala.html#21083 [3]: http://sourced.implicit.ly/net.databinder/dispatch-futures/0.8.0/Futures.scala.html#9079/Futures.scala.html#9079 Nathan |
Free forum by Nabble | Edit this page |