OAuth fails for stream.twitter.com

[bmaso]

I've been trying to use Dispatch to access Twitter's Streamed API. No luck authenticating with OAuth, though. Dispatch OAuth works great with the Twitter restful APIs.

I'm using the single-user access token key and secret provided by Twitter for my application. So, e.g., accessing Status.friends_timeline works great:

----------------------------------------------
import dispatch._
import Http._
import twitter._
import Twitter._
import oauth._

val consumer = new Consumer("my-consumer-key", "my-consumer-secret")
val token = new Token("my-access-token-key", "my-access-token-secret")

val http = new Http

val tweets = http(Status.friends_timeline(consumer, token)
---------------------------------------------

However, using similar techniques to access the streamed API I always get 401 replies:

---------------------------------------------
import dispatch._
import Http._
import oauth._
import OAuth._

val consumer = new Consumer("my-consumer-key", "my-consumer-secret")
val token = new Token("my-access-token-key", "my-access-token-secret")

val http = new Http

val samplesStream = :/("stream.twitter.com") / "statuses" / "sample.json"
val lines = http(samplesStream <@ (consumer, token) >~ {_.getLines})
----------------------------------------------

The last line always throws a dispatch.StatusCode 401 (unauthorized access)

After a lot of googling and bungling, I'm getting the sense that the problem is either:

1. The OAuth signature is not being generated in a way that the Twitter Streamed API likes (e.g., extra request parameters may be present on the request URI, as described at the end of the thread in [1])

2. The OAuth requirements for the Streamed API is different than what's required for the restful API. The second message in [1] suggests the Streamed API does not require an access_token (?) I don't think dispatch.oauth can do that, can it? Or maybe it can and I just need to know the right way to invoke it.

Can anyone get this working?

Best regards,
Brian Maso
[phone]
[e-mail]


[1] http://groups.google.com/group/twitter-development-talk/browse_thread/thread/fd71f9e8504ab36c/a2550ffc4538767c?q=oauth+streamed&lnk=ol&

[n8han]

Hi Brian, I'll test it out now.

Nathan

On 08/21/2010 01:16 PM, bmaso [via Databinder] wrote:

I've been trying to use Dispatch to access Twitter's Streamed API. No luck authenticating with OAuth, though. Dispatch OAuth works great with the Twitter restful APIs.

I'm using the single-user access token key and secret provided by Twitter for my application. So, e.g., accessing Status.friends_timeline works great:

----------------------------------------------
import dispatch._
import Http._
import twitter._
import Twitter._
import oauth._

val consumer = new Consumer("my-consumer-key", "my-consumer-secret")
val token = new Token("my-access-token-key", "my-access-token-secret")

val http = new Http

val tweets = http(Status.friends_timeline(consumer, token)
---------------------------------------------

However, using similar techniques to access the streamed API I always get 401 replies:

---------------------------------------------
import dispatch._
import Http._
import oauth._
import OAuth._

val consumer = new Consumer("my-consumer-key", "my-consumer-secret")
val token = new Token("my-access-token-key", "my-access-token-secret")

val http = new Http

val samplesStream = :/("stream.twitter.com") / "statuses" / "sample.json"
val lines = http(samplesStream <@ (consumer, token) >~ {_.getLines})
----------------------------------------------

The last line always throws a dispatch.StatusCode 401 (unauthorized access)

After a lot of googling and bungling, I'm getting the sense that the problem is either:

1. The OAuth signature is not being generated in a way that the Twitter Streamed API likes (e.g., extra request parameters may be present on the request URI, as described at the end of the thread in [1])

2. The OAuth requirements for the Streamed API is different than what's required for the restful API. The second message in [1] suggests the Streamed API does not require an access_token (?) I don't think dispatch.oauth can do that, can it? Or maybe it can and I just need to know the right way to invoke it.

Can anyone get this working?

Best regards,
Brian Maso
[phone]
[hidden email]


[1] http://groups.google.com/group/twitter-development-talk/browse_thread/thread/fd71f9e8504ab36c/a2550ffc4538767c?q=oauth+streamed&lnk=ol&

---

View message @ http://databinder.3617998.n2.nabble.com/OAuth-fails-for-stream-twitter-com-tp5448011p5448011.html
To start a new topic under Databinder, email [hidden email]
To unsubscribe from Databinder, click here.

[n8han]

No luck so far. I was able to open a statuses/sample stream with basic oauth and my twitter login, but not with oauth.

To answer your specific questions:

1. The OAuth signature is not being generated in a way that the Twitter Streamed API likes (e.g., extra request parameters may be present on the request URI, as described at the end of the thread in [1])

Dispatch doesn't do anything like that.

2. The OAuth requirements for the Streamed API is different than what's required for the restful API. The second message in [1] suggests the Streamed API does not require an access_token (?) I don't think dispatch.oauth can do that, can it? Or maybe it can and I just need to know the right way to invoke it.

Dispatch can do that. It's necessary for the first step of the token exchange process. To use it yourself, just call

    myRequest <@ (consumer)

But I tried that with statuses/sample and it didn't work either.

The instructions in the Twitter thread were inconsistent on this point. At first they said that you shouldn't sign the requests with an access token, but later the guy was instructing someone in the use of the access token. I guess what happened is that, in the mean time, Twitter had released its system for issuing access tokens on the developer site, so it was no longer necessary to permit requests signed only by the consumer.

But it's still very unclear what exactly is required, as that misunderstanding was never corrected and I couldn't find any official information outside of that thread on using OAuth with the streaming API. It's my impression that this interface is not well established at the moment.

It's possible that a consumer who should be able to access something, like statuses/sample, can not do it through OAuth for whatever reason. The only way I would have confidence that it can work for a particular set of tokens is to test them through the sample Earlybird application [1]. If it works there, at least we know it can be made to work through Dispatch.

If you're able to confirm that your tokens work through Earlybird, I'll do some more tracing to try to find out what the problem is through Dispatch. Otherwise, I would recommend using the basic auth interface for the time being as its use is better documented and, it's known to work through Dispatch. :)

[1]: http://github.com/zbowling/earlybird

Nathan

[bmaso]

Hi Nathan,

OK, so I got earlybird to work with the "/1/statuses/sample.json" stream. Took a bit of wrangling, but here's how to get it going:

* By default, earlybird uses "http://betastream.twitter.com/2b/user.json" as the target URL -- due to a bug you actually need to MODIFY earlybird.rb to hit the production Twitter Streamed API
  - Insert the following at line 316 of earlybird.rb:

      [ '-u', GetoptLong::OPTIONAL_ARGUMENT],

* Execute the following command line to access the sample.json stream:

ruby earlybird.rb -c "consumer-key" -s "consumer-secret" -a "access-token-key" \
-S "access-token-secret" -h "stream.twitter.com" -u "/1/statuses/sample.json"

  - Note: Twitter API docs actually list two separate URLs for the "sample" Stream -- the correct one is "/1/statuses/sample.json". And yes, that is the bitter voice of frustrated experience talking.
  - Note also: earlybird states in its output that is is connecting to "https://host/url" -- but earlybird is actually using "http", not "https".

If you make all those changes, earlybird is able to oauth authenticate and connect to the sample stream.

The code for oauth authentication earlybird uses is actually implemented in the "twitter" gem. So there's something the "twitter" ruby gem is doing right that Dispatch seems to be doing wrong when authenticating to stream.twitter.com.

I am trying to avoid basic auth since Twitter may be removing it at any time, and oauth seems to be working correctly (for those libs who have it working).

Best regards,
Brian Maso
[phone]
[e-mail]

[n8han]

Found it. (Wireshark *never* lets me down.) The stream.twitter.com server responds 401 until I add oauth_version to the Authorization header. Twitter is off-spec here:

       oauth_version
         OPTIONAL.  If present, MUST be set to "1.0".  Provides the
         version of the authentication process as defined in this
         specification.

http://tools.ietf.org/html/rfc5849

I'll add oauth_version -> 1.0 to the header even though it's a silly thing, as we are 1.0a. This will go out in Dispatch 0.7.6 which I am eager to release anyway.

Nathan

On 08/22/2010 11:42 PM, bmaso [via Databinder] wrote:

Hi Nathan,

OK, so I got earlybird to work with the "/1/statuses/sample.json" stream. Took a bit of wrangling, but here's how to get it going:

* By default, earlybird uses "http://betastream.twitter.com/2b/user.json" as the target URL -- due to a bug you actually need to MODIFY earlybird.rb to hit the production Twitter Streamed API
  - Insert the following at line 316 of earlybird.rb:

      [ '-u', GetoptLong::OPTIONAL_ARGUMENT],

* Execute the following command line to access the sample.json stream:

ruby earlybird.rb -c "consumer-key" -s "consumer-secret" -a "access-token-key" \
-S "access-token-secret" -h "stream.twitter.com" -u "/1/statuses/sample.json"

  - Note: Twitter API docs actually list two separate URLs for the "sample" Stream -- the correct one is "/1/statuses/sample.json". And yes, that is the bitter voice of frustrated experience talking.
  - Note also: earlybird states in its output that is is connecting to "https://host/url" -- but earlybird is actually using "http", not "https".

If you make all those changes, earlybird is able to oauth authenticate and connect to the sample stream.

The code for oauth authentication earlybird uses is actually implemented in the "twitter" gem. So there's something the "twitter" ruby gem is doing right that Dispatch seems to be doing wrong when authenticating to stream.twitter.com.

I am trying to avoid basic auth since Twitter may be removing it at any time, and oauth seems to be working correctly (for those libs who have it working).

Best regards,
Brian Maso
[phone]
[hidden email]

---

View message @ http://databinder.3617998.n2.nabble.com/OAuth-fails-for-stream-twitter-com-tp5448011p5451348.html
To start a new topic under Databinder, email [hidden email]
To unsubscribe from Databinder, click here.

[bmaso]

Great to hear!

Brian Maso