Getting exception with Exception message null

[Jack Widman]

Here is a method that tries to load a page and return either the length of the response or an exception if one is thrown.  When I apply it to certain urls, e.g. http://bae.cf.huffingtonpost.com/
an exception is thrown but the message is null. Any ideas? Thanks.

 def testCheckPage(url:String):String = {
    try {  
      var http = new Http
      var request = new Request(url)                                      
      val req_with_agent = request <:< Map("User-Agent" -> "Mozilla/4.0")  
      val responseBody = Http (req_with_agent as_str)
      responseBody.length.toString
    } catch {
       case ex: Exception => {
         "Message: " + ex.getMessage
       }
    }
  }

[Jack Widman]

Now I see I am getting a java.nio.BufferUnderflowException. Any idea what is causing this?

[n8han]

Yes, it's scala.io.Source being bad for long inputs or those with certain characters. I think I'm going to have to implement as_str without it (for 0.6.1), but in the mean time using the >> method and processing the InputStream should always work.

Nathan

[Jack Widman]

Excellent. Thanks so much.

On Fri, Nov 6, 2009 at 9:28 AM, n8han [via Databinder] <[hidden email]> wrote:

Yes, it's scala.io.Source being bad for long inputs or those with certain characters. I think I'm going to have to implement as_str without it (for 0.6.1), but in the mean time using the >> method and processing the InputStream should always work.

Nathan

---

View message @ http://n2.nabble.com/Getting-exception-with-Exception-message-null-tp3956879p3959051.html
To unsubscribe from Re: Getting exception with Exception message null, click here.

--
Jack Widman

co-founder / cto,  Authoritude, Inc.

203-641-9355

[Jack Widman]

Sorry to bother you again Nathan but could you show me how to use the >> method?

On Fri, Nov 6, 2009 at 9:28 AM, n8han [via Databinder] <[hidden email]> wrote:

Yes, it's scala.io.Source being bad for long inputs or those with certain characters. I think I'm going to have to implement as_str without it (for 0.6.1), but in the mean time using the >> method and processing the InputStream should always work.

Nathan

---

View message @ http://n2.nabble.com/Getting-exception-with-Exception-message-null-tp3956879p3959051.html
To unsubscribe from Re: Getting exception with Exception message null, click here.

--
Jack Widman

co-founder / cto,  Authoritude, Inc.

203-641-9355

[n8han]

I've narrowed down the problem a bit. It's that that page is encoded in iso-8859-1 and does in fact have some non-ascii in it. scala.io.Source chokes on this. The page doesn't have charset header, but even if it did  Dispatch is not looking for it. I'm going to fix it so that the charset is considered, if present, and so that you can override it manually. I also think that means we can continue to use Source for the time being (and I expect that class to get more capable and flexible over time, rather than the other way around).

Nathan

[n8han]

Okay, I've published Dispatch 0.6.1 with a solution to this problem. Since the resource doesn't specify a charset and it doesn't use Dispatch's default encoding, you need to do this to get it as a string:

import dispatch._
import Http._
val http = new Http
http("http://bae.cf.huffingtonpost.com/" <:< Map("User-Agent" -> "Mozilla/4.0") >\ "iso-8859-1" as_str)

The new >\ operator changes the default character encoding for the request. To use this you'll need to update the version spec in whatever project you're using to 0.6.1 and also refer to the databinder.net repository because I haven't been able to publish to scala-tools for the past few days. In your sbt project:

  val databinder_net = "databinder.net repository" at "http://databinder.net/repo"

Then run update in sbt. (Or if you're running Dispatch from a cloned repo or something you can just pull the changes.) I've also published a new Twine jar which refers to this version.

And thanks for reporting the problem!

Nathan