Yahoo Finance Stock data (FTSEMIB.MI) Issue

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Yahoo Finance Stock data (FTSEMIB.MI) Issue

Rmetrics mailing list
To whom it may concern,

I am interested in the FTSE MIB (or Milano Indice di Borsa) index. I am
trying to download the data up to the present date (18 June 2020). I am
using the get.hist.quote command in R from the tseries package.

I noticed the FTSE MIB  from yahoo has no pricing of any kind (open,
closed, adjusted, etc.) from 12 February 2018 to present. It appears from
the Yahoo site that the prices exist to the present day based on the
summary charts, but the other charts don't seem to show data. I am assuming
this stock is still used, which is why I am a bit confused by all the
missing data.

Is it possible to obtain this FTSE MIB data without all the missing data
from Yahoo Finance?

If this is possible, how?

If I cannot get the data from Yahoo Finance, are there alternative
suggested sources for an R user to get this data?

Thank you.

Here is the url and the Yahoo Finance data symbol.

https://finance.yahoo.com/quote/FTSEMIB.MI?p=FTSEMIB.MI

Symbol: FTSEMIB.MI

        [[alternative HTML version deleted]]

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.
Reply | Threaded
Open this post in threaded view
|

Re: Yahoo Finance Stock data (FTSEMIB.MI) Issue

Daniel Cegiełka

I did it quickly, but you can write some script in R.

curl and json (5 years):

curl 'https://charts.borsaitaliana.it/charts/services/ChartWService.asmx/GetPrices' -H 'Connection: keep-alive' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'DNT: 1' -H 'X-Requested-With: XMLHttpRequest' -H 'Content-Type: application/json; charset=UTF-8' -H 'Origin: https://charts.borsaitaliana.it' -H 'Sec-Fetch-Site: same-origin' -H 'Sec-Fetch-Mode: cors' -H 'Referer: https://charts.borsaitaliana.it/charts/Bit/SummaryChart.aspx?code=CB.FTSEMIB&lang=en' -H 'Accept-Encoding: gzip, deflate, br' --data-binary '{"request":{"SampleTime":"1d","TimeFrame":"5y","RequestedDataSetType":"ohlc","ChartPriceType":"price","Key":"FTSEMIB.IDX","OffSet":0,"FromDate":null,"ToDate":null,"UseDelay":true,"KeyType":"Topic","KeyType2":"Topic","Language":"en-US"}}' --compressed > FTSEMIB.json


And try fromJSON() etc.

btw:

[1592352000000,19585.89,19625.63,19876.12,19487.39,19585.89]

1592352000000 <--- it looks like POSIX time in milliseconds :)

so you need an extra trick:

> x <- 1592352000000
> z <- x/1000
> z
[1] 1592352000
> class(z) <- 'POSIXct'
> z
[1] "2020-06-17 02:00:00 CEST"


Best regards,
Daniel



> On 18 Jun 2020, at 16:58, Christopher Kromm via R-SIG-Finance <[hidden email]> wrote:
>
> To whom it may concern,
>
> I am interested in the FTSE MIB (or Milano Indice di Borsa) index. I am
> trying to download the data up to the present date (18 June 2020). I am
> using the get.hist.quote command in R from the tseries package.
>
> I noticed the FTSE MIB  from yahoo has no pricing of any kind (open,
> closed, adjusted, etc.) from 12 February 2018 to present. It appears from
> the Yahoo site that the prices exist to the present day based on the
> summary charts, but the other charts don't seem to show data. I am assuming
> this stock is still used, which is why I am a bit confused by all the
> missing data.
>
> Is it possible to obtain this FTSE MIB data without all the missing data
> from Yahoo Finance?
>
> If this is possible, how?
>
> If I cannot get the data from Yahoo Finance, are there alternative
> suggested sources for an R user to get this data?
>
> Thank you.
>
> Here is the url and the Yahoo Finance data symbol.
>
> https://finance.yahoo.com/quote/FTSEMIB.MI?p=FTSEMIB.MI
>
> Symbol: FTSEMIB.MI
>
> [[alternative HTML version deleted]]
>
> _______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-finance
> -- Subscriber-posting only. If you want to post, subscribe first.
> -- Also note that this is not the r-help list where general R questions should go.

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.
Reply | Threaded
Open this post in threaded view
|

Re: Yahoo Finance Stock data (FTSEMIB.MI) Issue

Rasmus Liland-3
On 2020-06-18 18:48 +0200, Daniel Cegiełka wrote:
> On 18 Jun 2020, at 16:58, Christopher Kromm wrote:
> >
> > If I cannot get the data from Yahoo
> > Finance, are there alternative suggested
> > sources for an R user to get this data?
>
> curl 'https://charts.borsaitaliana.it/charts/services/ChartWService.asmx/GetPrices' -H 'Connection: keep-alive' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'DNT: 1' -H 'X-Requested-With: XMLHttpRequest' -H 'Content-Type: application/json; charset=UTF-8' -H 'Origin: https://charts.borsaitaliana.it' -H 'Sec-Fetch-Site: same-origin' -H 'Sec-Fetch-Mode: cors' -H 'Referer: https://charts.borsaitaliana.it/charts/Bit/SummaryChart.aspx?code=CB.FTSEMIB&lang=en' -H 'Accept-Encoding: gzip, deflate, br' --data-binary '{"request":{"SampleTime":"1d","TimeFrame":"5y","RequestedDataSetType":"ohlc","ChartPriceType":"price","Key":"FTSEMIB.IDX","OffSet":0,"FromDate":null,"ToDate":null,"UseDelay":true,"KeyType":"Topic","KeyType2":"Topic","Language":"en-US"}}' --compressed > FTSEMIB.json

Dear Daniel Cegiełka,

that's a great answer!  Are you (or others)
able to elaborate more on how you determined
the headers to send?

Best,
Rasmus

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Yahoo Finance Stock data (FTSEMIB.MI) Issue

Daniel Cegiełka


> On 18 Jun 2020, at 19:48, Rasmus Liland <[hidden email]> wrote:
>
> On 2020-06-18 18:48 +0200, Daniel Cegiełka wrote:
>> On 18 Jun 2020, at 16:58, Christopher Kromm wrote:
>>>
>>> If I cannot get the data from Yahoo
>>> Finance, are there alternative suggested
>>> sources for an R user to get this data?
>>
>> curl 'https://charts.borsaitaliana.it/charts/services/ChartWService.asmx/GetPrices' -H 'Connection: keep-alive' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'DNT: 1' -H 'X-Requested-With: XMLHttpRequest' -H 'Content-Type: application/json; charset=UTF-8' -H 'Origin: https://charts.borsaitaliana.it' -H 'Sec-Fetch-Site: same-origin' -H 'Sec-Fetch-Mode: cors' -H 'Referer: https://charts.borsaitaliana.it/charts/Bit/SummaryChart.aspx?code=CB.FTSEMIB&lang=en' -H 'Accept-Encoding: gzip, deflate, br' --data-binary '{"request":{"SampleTime":"1d","TimeFrame":"5y","RequestedDataSetType":"ohlc","ChartPriceType":"price","Key":"FTSEMIB.IDX","OffSet":0,"FromDate":null,"ToDate":null,"UseDelay":true,"KeyType":"Topic","KeyType2":"Topic","Language":"en-US"}}' --compressed > FTSEMIB.json
>
> Dear Daniel Cegiełka,

Daniel :)

> that's a great answer!  Are you (or others)
> able to elaborate more on how you determined
> the headers to send?
>

Let's start that web scraping is probably legal now (in US):

https://towardsdatascience.com/web-scraping-is-now-legal-6bf0e5730a78 <https://towardsdatascience.com/web-scraping-is-now-legal-6bf0e5730a78>

Scraping requires some experience and intuition. There is no other way than web debugging - and each one is different. However, current web browsers are very powerful and have many useful add-ons. Sometimes it takes a minute, and sometimes it requires a lot of work. A good source for learning is getSymbols:

https://github.com/joshuaulrich/quantmod/blob/master/R/getSymbols.R <https://github.com/joshuaulrich/quantmod/blob/master/R/getSymbols.R>

And

https://stackoverflow.com/questions/44030983/yahoo-finance-url-not-working <https://stackoverflow.com/questions/44030983/yahoo-finance-url-not-working>

Best


> Best,
> Rasmus


        [[alternative HTML version deleted]]

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.
Reply | Threaded
Open this post in threaded view
|

Re: Yahoo Finance Stock data (FTSEMIB.MI) Issue

Mark McClellan
In reply to this post by Rmetrics mailing list
Take a look at the Y! json url. It's there. Here's a URL that will get you
1min bars.
https://l1-query.finance.yahoo.com/v8/finance/chart/FTSEMIB.MI?interval=1m&range=1d&indicators=quote&includeTimestamps=true

Cheers,

Mark

On Thu, Jun 18, 2020 at 9:58 AM Christopher Kromm via R-SIG-Finance <
[hidden email]> wrote:

> To whom it may concern,
>
> I am interested in the FTSE MIB (or Milano Indice di Borsa) index. I am
> trying to download the data up to the present date (18 June 2020). I am
> using the get.hist.quote command in R from the tseries package.
>
> I noticed the FTSE MIB  from yahoo has no pricing of any kind (open,
> closed, adjusted, etc.) from 12 February 2018 to present. It appears from
> the Yahoo site that the prices exist to the present day based on the
> summary charts, but the other charts don't seem to show data. I am assuming
> this stock is still used, which is why I am a bit confused by all the
> missing data.
>
> Is it possible to obtain this FTSE MIB data without all the missing data
> from Yahoo Finance?
>
> If this is possible, how?
>
> If I cannot get the data from Yahoo Finance, are there alternative
> suggested sources for an R user to get this data?
>
> Thank you.
>
> Here is the url and the Yahoo Finance data symbol.
>
> https://finance.yahoo.com/quote/FTSEMIB.MI?p=FTSEMIB.MI
>
> Symbol: FTSEMIB.MI
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-finance
> -- Subscriber-posting only. If you want to post, subscribe first.
> -- Also note that this is not the r-help list where general R questions
> should go.
>


--
2.72

        [[alternative HTML version deleted]]

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.
Reply | Threaded
Open this post in threaded view
|

Re: Yahoo Finance Stock data (FTSEMIB.MI) Issue

Rmetrics mailing list
Thank you very much. That was very helpful.

On Thu, 18 Jun 2020 at 21:49, Mark McClellan <[hidden email]> wrote:

>
> Take a look at the Y! json url. It's there. Here's a URL that will get you
> 1min bars.
>
> https://l1-query.finance.yahoo.com/v8/finance/chart/FTSEMIB.MI?interval=1m&range=1d&indicators=quote&includeTimestamps=true
>
> Cheers,
>
> Mark
>
> On Thu, Jun 18, 2020 at 9:58 AM Christopher Kromm via R-SIG-Finance <
> [hidden email]> wrote:
>
>> To whom it may concern,
>>
>> I am interested in the FTSE MIB (or Milano Indice di Borsa) index. I am
>> trying to download the data up to the present date (18 June 2020). I am
>> using the get.hist.quote command in R from the tseries package.
>>
>> I noticed the FTSE MIB  from yahoo has no pricing of any kind (open,
>> closed, adjusted, etc.) from 12 February 2018 to present. It appears from
>> the Yahoo site that the prices exist to the present day based on the
>> summary charts, but the other charts don't seem to show data. I am
>> assuming
>> this stock is still used, which is why I am a bit confused by all the
>> missing data.
>>
>> Is it possible to obtain this FTSE MIB data without all the missing data
>> from Yahoo Finance?
>>
>> If this is possible, how?
>>
>> If I cannot get the data from Yahoo Finance, are there alternative
>> suggested sources for an R user to get this data?
>>
>> Thank you.
>>
>> Here is the url and the Yahoo Finance data symbol.
>>
>> https://finance.yahoo.com/quote/FTSEMIB.MI?p=FTSEMIB.MI
>>
>> Symbol: FTSEMIB.MI
>>
>>         [[alternative HTML version deleted]]
>>
>> _______________________________________________
>> [hidden email] mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-sig-finance
>> -- Subscriber-posting only. If you want to post, subscribe first.
>> -- Also note that this is not the r-help list where general R questions
>> should go.
>>
>
>
> --
> 2.72
>

        [[alternative HTML version deleted]]

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.
Reply | Threaded
Open this post in threaded view
|

Re: Yahoo Finance Stock data (FTSEMIB.MI) Issue

Rasmus Liland-3
In reply to this post by Daniel Cegiełka
On 2020-06-18 21:08 +0200, Daniel Cegiełka wrote:
> On 18 Jun 2020, at 19:48, Rasmus Liland wrote:
> >
> > Dear Daniel Cegiełka,
>
> Daniel :)

Ok, Daniel it is then :)

> > Are you (or others) able to elaborate
> > more on how you determined the headers to
> > send?
>
> Let's start that web scraping is probably
> legal now (in US):
>
> https://towardsdatascience.com/web-scraping-is-now-legal-6bf0e5730a78

So as long as you don't log in to Facebook
through your bot, it seems to be legal at
least :/

> Scraping requires some experience and
> intuition. There is no other way than web
> debugging - and each one is different.  
> However, current web browsers are very
> powerful and have many useful add-ons.  
> Sometimes it takes a minute, and sometimes
> it requires a lot of work. A good source
> for learning is getSymbols:
>
> https://github.com/joshuaulrich/quantmod/blob/master/R/getSymbols.R
>
> And
>
> https://stackoverflow.com/questions/44030983/yahoo-finance-url-not-working
Such a well balanced answer, every website
being different, right, so it depends on
whether you meet an interface like Angular
like Yahoo changed to.

Thanks for good pointers on this.

Best,
Rasmus

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.

signature.asc (849 bytes) Download Attachment