profile
viewpoint
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/galashour/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.

galashour/sss 0

Stock Scanner & Screener: A yfinance-based Stock Scanner and Screener, focusing on Fundamental Properties of scanned stocks.

issue commentpydata/pandas-datareader

data_source='yahoo': reading data fails since July 01

Finally it worked by updating yfinance ( and datareader). I first tried updating datareader, didn`t work and after updating yfinance it worked.

Both yfinance and datareader were exposed to the same issue, and thus both affected at the same time. If you used both libraries, you had to update both of them.

aeolio

comment created time in a month

push eventgalashour/sss

Asaf

commit sha eecfde3927b3b0dcb591dbebe5b0b8d7efe2ab90

Adding back sss_manual_currency_converter.py sss_manual_currency_converter.py

view details

Asaf

commit sha b6be8ae4869f028378cf91b57bf7ba69815ae254

V 0.1.118 Introducing Full support for aggregated multi-dimentional scan (Geometric + Arithmetic Values)

view details

Asaf

commit sha 6ca7152111eac25dc1e3386291c291ec93e0971c

V 0.1.119 Fix The width of the symbol column in PDF generator of sss_results

view details

Asaf

commit sha 64828a5fa867b068de8ad0e7a7a0cad2f1c9b0c6

V 0.1.120 Fix missing round values for visualization proper in PDF

view details

push time in 2 months

pull request commentpydata/pandas-datareader

resolve issue #867 by explicitly adding headers to url _get_response

Please try master to see if it is working now.

pip install git+https://github.com/pydata/pandas-datareader.git

If there are positive reports, I'll release very soon.

Looks good from my side. Thanks (!)

galashour

comment created time in 2 months

fork galashour/sss

Stock Scanner & Screener: A yfinance-based Stock Scanner and Screener, focusing on Fundamental Properties of scanned stocks.

fork in 2 months

pull request commentpydata/pandas-datareader

resolve issue #867 by explicitly adding headers to url _get_response

The API that used to work was simple and is used by many:

df1 = web.DataReader(ticker1, data_source='yahoo', start=date_start, end=date_end)

This API is now broken. It is a common API used, by many.

It is not only for me, quite a few on issue #867 (and also on issue #868 that is a duplicate) - reported the same problem, and reported it works for them with the simple fix applied.

Original API was broken due to a 'bug' in the existing that yahoo.daily as I mentioned that 'forgot' to add the self.headers that was already set internally correctly (with no explicit value from the users), while a preferred practice regrdless to yahoo is to use headers in the url-request (as can be seen also in the base class).

I will say it one last time, as nicely as I can, I didn't violate any public API (strong and unjustified claims), nor did I change any API (public or internal). Simply added a parameter that was already there unused while it should have been used. Also the base class does use the headers in the url-request, and just the yahoo-daily implementation forgot to add this 3rd parameter.

Lastly, I will add the yfinance package used the same approach as I'm suggesting to resolve the same failure they experienced last week with yahoo source.

galashour

comment created time in 2 months

pull request commentpydata/pandas-datareader

resolve issue #867 by explicitly adding headers to url _get_response

once again - wrong. Users don't provide headers, and I didn't provide any headers in my testing, and no interface was changes whatsoever.

galashour

comment created time in 2 months

pull request commentpydata/pandas-datareader

resolve issue #867 by explicitly adding headers to url _get_response

There is No HARD CODED header content in the fix, not sure if you actually read the actual fix.

It puts on line 153 of the yahoo/daily.py the 3rd parameter that was missing, with NO hardcoded string: resp = self._get_response(url, params=params, headers=self.headers)

Similarly, it also put a safeguard in the base class (again happens to be line 153) only for the case somehow we end up with headers == None: if headers == None: headers = self.headers

galashour

comment created time in 2 months

push eventgalashour/sss

Asaf

commit sha 195f50e973100324ecac92be35b1f810471018b1

Updated All results with Altman Z proper and corrected quarterly calculations on earnings, revenues, etc Updated All results with Altman Z proper and corrected quarterly calculations on earnings, revenues, etc

view details

Asaf

commit sha a7efd1eb979d056f9ff70b01efcff39d462bc618

Delete Results/All/20210606-011608_Tchnlgy3.0_FnnclSrvcs0.5_A_Bdb_nRes3195 directory

view details

Asaf

commit sha a24a2f735de87995109c23abcdbddc2a2a1f079f

V 0.1.102 Post processing with normalized values for an arythmetic average (in addition to the geometric average) of sss_value

view details

Asaf

commit sha e3acd2bba4f3b093b48c95f6dc34e14c1264e140

V 0.1.103 Fix sorting by sss_value_normalized

view details

Asaf

commit sha 0e0ffd6f8e062c5208948d081bae8a8a0a4dc83a

V 0.1.104 Fix row -> row_index in reference_db access

view details

Asaf

commit sha 2f37000d6bbf5a5b1c477b5cbc93548fb1af4056

V 0.1.105 Sort by sss_value_normalized + fix normalized columns positions

view details

Asaf

commit sha be4af47f9c6595283864648862949e7589a7672e

V 0.1.109 Use TTM only + add some dampers

view details

galashour

commit sha bb7fe69ed857b969abfe170fa3638ed65bb94c85

Merge branch 'asafravid:master' into master

view details

push time in 2 months

issue commentpydata/pandas-datareader

data_source='yahoo': reading data fails since July 01

yfinance made a similar fix as outlined in #867 , a pull request for pandas_datareader is 'in the process', hopefully merged 'soon'.

aeolio

comment created time in 2 months

pull request commentpydata/pandas-datareader

resolve issue #867 by explicitly adding headers to url _get_response

Hi galashour,

I made a similar PR a couple of days ago here but was made aware of an issue by @ralexx with whom you had a similar discussion here. As there is no reason to have multiple PRs for the same problem, I'll close mine and continue to contribute here. I hope the issue raised by @ralexx can be solved soon as I want the YDR back to work.

However, you should test if you actually get any data back and not just an empty DataFrame. I've noticed that I had to change the default interval from 'd' to '1d' in the daily.py file too in order to get actual data. See my commit #869 for further details. You might want to add another commit to fix that issue in your PR.

Thanks for the comment. I verified that I do get valid daily data once the header issue was addressed.

df1 = web.DataReader(ticker1, data_source='yahoo', start=date_start, end=date_end)

galashour

comment created time in 2 months

push eventgalashour/sss

Asaf

commit sha b2c94970a7e93e16a36713d42ec598c1b09c5036

V 0.1.100 Introducing sss_post_processing.py

view details

galashour

commit sha 4cc2281590748f3d563dce37179e7d98f1702ada

Merge branch 'asafravid:master' into master

view details

push time in 2 months

push eventgalashour/sss

ashour

commit sha 0a56b35104e436d95b54f32c87465b9690598470

Updated git.ignore

view details

push time in 2 months

pull request commentpydata/pandas-datareader

resolve issue #867 by explicitly adding headers to url _get_response

I'm relatively new, the fix is simple and addresses issue #867 , without the fix, using yahoo source fails (/crashes).

I'm not sure if / how to do some of the above, but I did test that before the fix the library crashes and after the fix it completes. Also others commenting on issue #867 confrimed the same.

I'm not sure if/how to performm the above checkboxes, and in which environment, possibly you guys can help.

galashour

comment created time in 2 months

issue commentpydata/pandas-datareader

_get_response without headers doesn't work (at least with 'yahoo' source

I just made a pull request using in the yahoo/daily: resp = self._get_response(url, params=params, headers=self.headers)

and also adding a 'safeguard' check in the base class.

hopefully the request will go through.

Thanks all for the constructive feedback.

galashour

comment created time in 2 months

PR opened pydata/pandas-datareader

resolve issue #867 by explicitly adding headers to url _get_response
  • [ ] closes #xxxx
  • [ ] tests added / passed
  • [ ] passes git diff upstream/master -u -- "*.py" | flake8 --diff
  • [ ] passes black --check pandas_datareader
  • [ ] added entry to docs/source/whatsnew/vLATEST.txt
+24 -21

0 comment

2 changed files

pr created time in 2 months

push eventgalashour/pandas-datareader

ashour

commit sha c0d5644d39e5f4ec9168bc5e48590ea0fb27d6d3

resolve issue #867 by explicitly adding headers to url _get_response

view details

push time in 2 months

fork galashour/pandas-datareader

Extract data from a wide range of Internet sources into a pandas DataFrame.

https://pydata.github.io/pandas-datareader/stable/index.html

fork in 2 months

push eventgalashour/sss

Asaf

commit sha 455ef96b6bc3d842c58f3b9896434f2452c100a1

Merge pull request #10 from galashour/master Updated currencies.json file - July 9th 2021

view details

push time in 2 months

PR opened asafravid/sss

Updated currencies.json file - July 9th 2021
+205 -205

0 comment

1 changed file

pr created time in 2 months

push eventgalashour/sss

ashour

commit sha 58e3a3dc8d1cfff00b2a9584be740d01b7c386e9

Updated currencies.json file - July 9th 2021

view details

push time in 2 months

push eventgalashour/sss

Asaf

commit sha 4b634385525cecf80ac74f783a8279e4e2ef5cd6

Update 3rd July 2021 NSR results Update 3rd July 2021 NSR results

view details

Asaf

commit sha b3cf0845625cccb191cbc57891f102fbcaca236e

V 0.1.92 Update Decimal rounding to 6 points

view details

push time in 2 months

push eventgalashour/sss

Asaf

commit sha d76f95f6ed98f30f96eb70080dd81bce33e1dad7

Updating July 2nd 2021 Results TASE Updating July 2nd 2021 Results TASE

view details

Asaf

commit sha c8a69ee14cfe8263d7398d46db278c1a2e2ee725

Remove older results Remove older results

view details

Asaf

commit sha 3c4488c2d31bef0eb12601470b8b2640b93d89b8

Remove older results Remove older results

view details

Asaf

commit sha 4041628cc5ab6795e5f5c3f3f3aa7111fb3c1967

Updating July Indices Nasdaq TASE

view details

Asaf

commit sha 81767970ab13da2cb2f6d62caa373b39d889fdd4

Update July Indices Nasdaq

view details

Asaf

commit sha bceb0a6860472726e3ea7186f10ad9a49a910410

No need for text sources

view details

Asaf

commit sha 217bab86b73f69c3c8c6afe7a3950c562faf1f27

No need for text sources

view details

Asaf

commit sha 851e3685640f65fa51922e7cbae97ff8184be896

No need for text sources

view details

Asaf

commit sha bc783fe27e90e71abb78305a2cc94bceea52a4b7

Ref results 20210620 Ref results 20210620

view details

Asaf

commit sha 13d30e33362e3ba2d2115ee53881e00755762714

Update all Nasdaq scan 20th June 2021 Update all Nasdaq scan 20th June 2021

view details

Asaf

commit sha b1de24c140d668fa32d5a8c77514ef25270c157b

Remove folder in typo-wrong path Remove folder in typo-wrong path

view details

Asaf

commit sha 901ec495c6f874180545dd5c9eca70a279691213

Update 4th of July 2021 Results All Update 4th of July 2021 Results All

view details

Asaf

commit sha ca92f1bbcc780357c33b4b5761099929b7cf8290

V 0.1.91 Fixed Calculations Quarterly Revenue, Earnings, Net Income, Cash flows (ingress money) - no weighted average - just sum (TTM)

view details

push time in 2 months

issue commentpydata/pandas-datareader

_get_response without headers doesn't work (at least with 'yahoo' source

So we have some confidence a change to base.py won't affect other readers?

Also, as a thought exercise, what are some ideas to address the "brittle"-ness of this change if Yahoo were to make future changes?

Anyone on the YOLO route and just deal with Yahoo changes as they happen? Because, it's basically what we're doing right now with this current header issue.

Indeed would be good if someone who has scripts that use also datareaders from other sources could verify that the suggested update doesn't degrade anything (it shouldn't, but worth double checking). Would be good if they can update here which other datareader source was checked after making the suggested 2 lines addition to the base.

galashour

comment created time in 2 months

issue commentpydata/pandas-datareader

_get_response without headers doesn't work (at least with 'yahoo' source

Hello, I think, we shouldn't change base.py, because this file is used also by many other reader classes and we don't know if this change in base.py can't have some side effect for other readers. I propose change directly in the file yahoo/daily.py in the class YahooDailyReader. In that class, we already have defined and filled the variable self.headers but this variable is not used anywhere. Why is this variable there without using it? How about we use the self.headers variable as a parameter in _get_response method, which is then called to base.py?

So in yahoo/daily.py in the method _read_one_data on row 153 instead of this

        resp = self._get_response(url, params=params)

can be this

        resp = self._get_response(url, params=params, headers=self.headers)

I agree with the spirit of the comment, but my perspective is slightly different: The base class provides a value as a 'safety measure' in case we end up with None as a header. Each of the derived reader classes can still override as applicable.

So, think about it as a 'last safety measure', just in case the 'sub-class' or user didn't provide a value, so that we still end up with a valid header (lack of which introduced this little drama). Whether that value is fixed or derived in other way to introduce some 'variability' in headers like suggested above - is indeed something that could be considered.

galashour

comment created time in 3 months

issue commentpydata/pandas-datareader

_get_response without headers doesn't work (at least with 'yahoo' source

No change to pandas_datareader code is required. You can instantiate your own requests.Session and update its .headers attribute with the user agent header of your choice. Then pass the Session instance to pandas_datareader.yahoo.daily.YahooDailyReader:

# from pandas_datareader.yahoo.daily import YahooDailyReader as ydr
# import requests

USER_AGENT = {
    'User-Agent': ('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)'
                   ' Chrome/91.0.4472.124 Safari/537.36')
    }
sesh = requests.Session()
sesh.headers.update(USER_AGENT)

df = ydr(**your_kwargs, session=sesh)

This works for me with e.g. the current Chrome user agent as above. See for example HSBC's HK listing 0005.HK (which has closed at the time of writing and should match your results if you check the "Historical Data" tab on finance.yahoo.com).

                 High        Low       Open      Close    Volume  Adj Close
Date                                                                       
2021-06-15  48.000000  47.000000  47.849998  47.349998  22382507  47.349998
2021-06-16  48.349998  47.400002  47.650002  48.099998  18503780  48.099998
2021-06-17  48.500000  47.650002  48.049999  48.299999  20179603  48.299999
2021-06-18  47.650002  47.099998  47.599998  47.200001  26316274  47.200001
2021-06-21  46.200001  45.099998  46.049999  45.549999  48269764  45.549999
2021-06-22  46.000000  45.650002  45.849998  45.750000  18034452  45.750000
2021-06-23  46.450001  45.299999  45.650002  46.049999  15193950  46.049999
2021-06-24  46.200001  45.650002  45.650002  45.900002  11924309  45.900002
2021-06-25  46.299999  45.650002  45.650002  46.099998  14515655  46.099998
2021-06-28  46.200001  45.500000  46.099998  45.900002   9692425  45.900002
2021-06-29  45.599998  45.000000  45.549999  45.250000  27624318  45.250000
2021-06-30  45.250000  44.799999  45.000000  44.849998  18489180  44.849998
2021-07-02  45.049999  44.400002  45.000000  44.849998  19693352  44.849998

True, but in my opinion it is more elegant to abstract these details from the user, and have the library handle the various 'subtleties' internally.

galashour

comment created time in 3 months

issue commentpydata/pandas-datareader

_get_response without headers doesn't work (at least with 'yahoo' source

I'm getting same error as well. Thanks for creating this fix.

Did yahoo change their HTML in a recent release?

My guess (speculation) is that yahoo tightened the requirements on url queries to verify these are legitimate browser requests, thus resulting in the breakage in case the header section of the query was missing (None).

The suggested 2 lines addition to base.py (line #152) of pandas_datareader seems to resolve the issue by explicitly forcing a valid header when one is missing.

galashour

comment created time in 3 months

issue commentpydata/pandas-datareader

_get_response without headers doesn't work (at least with 'yahoo' source

Hi everyone, I am getting an error as well when I try to retrieve stock data from yahoo finance. Is this error related to the same issue as described above?

RemoteDataError: Unable to read URL: https://finance.yahoo.com/quote/F/history?period1=1264996800&period2=1517543999&interval=1d&frequency=1d&filter=history
Response Text:
b'<!DOCTYPE html>\n  <html lang="en-us"><head>\n  <meta http-equiv="content-type" content="text/html; charset=UTF-8">\n      <meta charset="utf-8">\n      <title>Yahoo</title>\n      <meta name="viewport" content="width=device-width,initial-scale=1,minimal-ui">\n      <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">\n      <style>\n  html {\n      height: 100%;\n  }\n  body {\n      background: #fafafc url(https://s.yimg.com/nn/img/sad-panda-201402200631.png) 50% 50%;\n      background-size: cover;\n      height: 100%;\n      text-align: center;\n      font: 300 18px "helvetica neue", helvetica, verdana, tahoma, arial, sans-serif;\n  }\n  table {\n      height: 100%;\n      width: 100%;\n      table-layout: fixed;\n      border-collapse: collapse;\n      border-spacing: 0;\n      border: none;\n  }\n  h1 {\n      font-size: 42px;\n      font-weight: 400;\n      color: #400090;\n  }\n  p {\n      color: #1A1A1A;\n  }\n  #message-1 {\n      font-weight: bold;\n      margin: 0;\n  }\n  #message-2 {\n      display: inline-block;\n      *display: inline;\n      zoom: 1;\n      max-width: 17em;\n      _width: 17em;\n  }\n      </style>\n  <script>\n    document.write(\'Yahoo Logo\n      <h1 style="margin-top:20px;">Will be right back...</h1>\n      <p id="message-1">Thank you for your patience.</p>\n      <p id="message-2">Our engineers are working quickly to resolve the issue.</p>\n      </td>\n  </tr>\n  </tbody></table>\n  </body></html>

yes - looks to me the same

galashour

comment created time in 3 months

issue openedpydata/pandas-datareader

_get_response without headers doesn't work (at least with 'yahoo' source

to fix, I put in base.py:

def _get_response(self, url, params=None, headers=None):
    """ send raw HTTP request to get requests.Response from the specified url
    Parameters
    ----------
    url : str
        target URL
    params : dict or None
        parameters passed to the URL
    """

    # initial attempt + retry
    **if headers == None:
        headers          = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}**

    pause = self.pause

created time in 3 months

push eventgalashour/sss

Asaf

commit sha 7165cb8077eed9d716d216f086754590cc0e42d4

V 0.1.89 2 small bug fixes for research_mode

view details

Asaf

commit sha c09b451c11ae305ee5cfbce808fae34ebcbc61bf

Merge pull request #9 from galashour/master updating currencies.json as of July 2nd

view details

push time in 3 months

PR opened asafravid/sss

updating currencies.json as of July 2nd
+204 -204

0 comment

1 changed file

pr created time in 3 months