Download data

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Download data

jcrosbie
Hi, I'm trying to download data from:
http://www.ngx.com/settlehistory.html

Is it possible to fetch the data with R?

Thank you
Reply | Threaded
Open this post in threaded view
|

Re: Download data

Jim Lemon
On 05/29/2013 02:02 AM, jcrosbie wrote:
> Hi, I'm trying to download data from:
> http://www.ngx.com/settlehistory.html
>
> Is it possible to fetch the data with R?
>
Hi jcrosbie,
The simplest way seems to be to highlight the desired spreadsheet (less
the title row), copy (Ctrl-C) and paste (Ctrl-V) it into a text editor
and save it (e.g. ss1.tab). This produces a TAB delimited file that can
be read into a data frame in R with:

ss1<-read.table("ss1.tab")

Jim

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Download data

Pascal Oettli-2
Hi,

The combination "read.table" (and its arguments) + "stdin" also can be
used, directly in R.

 > read.table(stdin(), ...)

Regards,
Pascal

On 29/05/2013 21:35, Jim Lemon wrote:

> On 05/29/2013 02:02 AM, jcrosbie wrote:
>> Hi, I'm trying to download data from:
>> http://www.ngx.com/settlehistory.html
>>
>> Is it possible to fetch the data with R?
>>
> Hi jcrosbie,
> The simplest way seems to be to highlight the desired spreadsheet (less
> the title row), copy (Ctrl-C) and paste (Ctrl-V) it into a text editor
> and save it (e.g. ss1.tab). This produces a TAB delimited file that can
> be read into a data frame in R with:
>
> ss1<-read.table("ss1.tab")
>
> Jim
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Download data

Adams, Jean
In reply to this post by jcrosbie
I tried reading in the data using the XML package, but I can't figure out
how to read either ALL of the tables or a particular table.  The code below
just reads the first table.  Maybe someone else will know how.

Jean


library(XML)
look <- readHTMLTable("http://www.ngx.com/settlehistory.html")
head(look[[1]])
                                          V1         V2         V3 V4 V5 V6
V7 V8       V9 V10
1 Crude Firm Phys AHS, ID, WTI, Edm-Enbridge 2013-05-01 2013-05-31  0
      -16.0000
2 Crude Firm Phys AHS, ID, WTI, Edm-Enbridge 2013-06-01 2013-06-30  0
      -18.2500
3 Crude Firm Phys AHS, ID, WTI, Edm-Enbridge 2013-07-01 2013-07-31  0
      -19.7500
4 Crude Firm Phys AHS, ID, WTI, Edm-Enbridge 2013-08-01 2013-08-31  0
      -21.2500
5 Crude Firm Phys AHS, ID, WTI, Edm-Enbridge 2013-09-01 2013-09-30  0
      -22.7500
6 Crude Firm Phys AHS, ID, WTI, Edm-Enbridge 2013-10-01 2013-10-31  0
      -23.0000



On Tue, May 28, 2013 at 11:02 AM, jcrosbie <[hidden email]> wrote:

> Hi, I'm trying to download data from:
> http://www.ngx.com/settlehistory.html
>
> Is it possible to fetch the data with R?
>
> Thank you
>
>
>
> --
> View this message in context:
> http://r.789695.n4.nabble.com/Download-data-tp4668138.html
> Sent from the R help mailing list archive at Nabble.com.
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Download data

jcrosbie
Thank you, I will have to wait until I get home form work to test the XML. But I'm looking to do something more along the lines of an automated process over copy and pasting. 

James


________________________________
 From: "Adams, Jean" <[hidden email]>
To: jcrosbie <[hidden email]>
Cc: R help <[hidden email]>
Sent: Wednesday, May 29, 2013 6:50 AM
Subject: Re: [R] Download data



I tried reading in the data using the XML package, but I can't figure out how to read either ALL of the tables or a particular table.  The code below just reads the first table.  Maybe someone else will know how.

Jean



library(XML)
look <- readHTMLTable("http://www.ngx.com/settlehistory.html")
head(look[[1]])
                                          V1         V2         V3 V4 V5 V6 V7 V8       V9 V10
1 Crude Firm Phys AHS, ID, WTI, Edm-Enbridge 2013-05-01 2013-05-31  0             -16.0000    
2 Crude Firm Phys AHS, ID, WTI, Edm-Enbridge 2013-06-01 2013-06-30  0             -18.2500    
3 Crude Firm Phys AHS, ID, WTI, Edm-Enbridge 2013-07-01 2013-07-31  0             -19.7500    
4 Crude Firm Phys AHS, ID, WTI, Edm-Enbridge 2013-08-01 2013-08-31  0             -21.2500    
5 Crude Firm Phys AHS, ID, WTI, Edm-Enbridge 2013-09-01 2013-09-30  0             -22.7500    
6 Crude Firm Phys AHS, ID, WTI, Edm-Enbridge 2013-10-01 2013-10-31  0             -23.0000    




On Tue, May 28, 2013 at 11:02 AM, jcrosbie <[hidden email]> wrote:

Hi, I'm trying to download data from:

>http://www.ngx.com/settlehistory.html
>
>Is it possible to fetch the data with R?
>
>Thank you
>
>
>
>--
>View this message in context: http://r.789695.n4.nabble.com/Download-data-tp4668138.html
>Sent from the R help mailing list archive at Nabble.com.
>
>______________________________________________
>[hidden email] mailing list
>https://stat.ethz.ch/mailman/listinfo/r-help
>PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
>and provide commented, minimal, self-contained, reproducible code.
>
        [[alternative HTML version deleted]]


______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Download data

jcrosbie
Thank you for all your help. I'm still not able to figure out how automate downloads from online websites.

This is a daily function to download the needed data. I would also like to be able to do this on other websites such as:

http://ets.aeso.ca/ets_web/docroot/Market/Reports/HistoricalReportsStart.html

and

http://www.ngx.com/?page_id=561
Reply | Threaded
Open this post in threaded view
|

Re: Download data

Jeff Newmiller
I am sorry to hear that you are having difficulty, but your automation task is one that requires operating-system-specific knowledge that would be off-topic for this list, and web-scraping of forms really requires knowledge of web protocols and (in this case) Java and JavaScript that are also off-topic here. There exist packages in CRAN that may be helpful in your endeavor, but likely only if you study the appropriate subject areas outside of R first so that you know what you need to accomplish in detail. My quick estimation is that the aeso web site will be unusually difficult to extract data from, so you may need to pay a consultant to help you with this and/or ask the website developers if they support an automation mechanism that you can use.
---------------------------------------------------------------------------
Jeff Newmiller                        The     .....       .....  Go Live...
DCN:<[hidden email]>        Basics: ##.#.       ##.#.  Live Go...
                                      Live:   OO#.. Dead: OO#..  Playing
Research Engineer (Solar/Batteries            O.O#.       #.O#.  with
/Software/Embedded Controllers)               .OO#.       .OO#.  rocks...1k
---------------------------------------------------------------------------
Sent from my phone. Please excuse my brevity.

jcrosbie <[hidden email]> wrote:

>Thank you for all your help. I'm still not able to figure out how
>automate
>downloads from online websites.
>
>This is a daily function to download the needed data. I would also like
>to
>be able to do this on other websites such as:
>
>http://ets.aeso.ca/ets_web/docroot/Market/Reports/HistoricalReportsStart.html
>
>and
>
>http://www.ngx.com/?page_id=561
>
>
>
>
>--
>View this message in context:
>http://r.789695.n4.nabble.com/Download-data-tp4668138p4676465.html
>Sent from the R help mailing list archive at Nabble.com.
>
>______________________________________________
>[hidden email] mailing list
>https://stat.ethz.ch/mailman/listinfo/r-help
>PLEASE do read the posting guide
>http://www.R-project.org/posting-guide.html
>and provide commented, minimal, self-contained, reproducible code.

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.