Ask questions_get_response without headers doesn't work (at least with 'yahoo' source

to fix, I put in

def _get_response(self, url, params=None, headers=None):
    """ send raw HTTP request to get requests.Response from the specified url
    url : str
        target URL
    params : dict or None
        parameters passed to the URL

    # initial attempt + retry
    if headers == None:
        headers          = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}

    pause = self.pause

Answer questions kdvolder

Also, as a thought exercise, what are some ideas to address the "brittle"-ness of this change if Yahoo were to make future changes?

There isn't really much you can do about that really. The nature of yahoo data reader is that it is essentially using yahoo internal apis. There will never be any guarantee they won't change something that breaks datareader in the future, either deliberately or simply as a consequence of them deciding to change the structure of their internal apis for whatever reason. Personally I don't actually think that yahoo broke this deliberately. But in any case, using an internal api like this will always be somewhat brittle.

So we have some confidence a change to won't affect other readers?

Would be good to check that yes. It seems unlikely though. In some sense sending requests without setting a 'user-agent' is a bit unusual and is more likely to cause problems than the other way around. So overall I would expect a change that guarantees such a header is present to be making things less brittle (for any data provider, not just for yahoo).

Kris De Volder kdvolder SpringSource a Division of VMWare Vancouver, BC, Canada
Github User Rank List