Importing Big data to R

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Importing Big data to R

Mangalani Peter Makananisa
Dear R-Gurus,

I am trying to read in data with 24,349,113 rows to R-3.3.3 (64 bit) and have used  the library   "data.table"  and It managed to read 23,347,070 rows  and the remainder was 2,043 rows only.

Could you please advise me as to which library/R-commands is suitable to read the full data in to R?

Kind regards,

Mangalani Peter Makananisa (5786)
South African Revenue Service (SARS)
+2782 456 4669 / +2712 422 7357

Please Note: This email and its contents are subject to our email legal notice which can be viewed at http://www.sars.gov.za/Pages/Email-disclaimer.aspx

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Importing Big data to R

jholtman
A little more information would be useful.  Why did it stop? Was there an
error message?  Can you show the commands/console log of what you did.
Provide information on how much memory your computer has on it.  When the
operation completed, how much memory was used.  An important aspect is how
many columns did the data have.  How big was the file on disk.  What other
objects were in memory at the same time.  The list can go on and on, so
more information would be useful to understand the problem.


Jim Holtman
Data Munger Guru

What is the problem that you are trying to solve?
Tell me what you want to do, not how you want to do it.

On Wed, Jul 12, 2017 at 2:44 AM, Mangalani Peter Makananisa <
[hidden email]> wrote:

> Dear R-Gurus,
>
> I am trying to read in data with 24,349,113 rows to R-3.3.3 (64 bit) and
> have used  the library   "data.table"  and It managed to read 23,347,070
> rows  and the remainder was 2,043 rows only.
>
> Could you please advise me as to which library/R-commands is suitable to
> read the full data in to R?
>
> Kind regards,
>
> Mangalani Peter Makananisa (5786)
> South African Revenue Service (SARS)
> +2782 456 4669 / +2712 422 7357
>
> Please Note: This email and its contents are subject to our email legal
> notice which can be viewed at http://www.sars.gov.za/Pages/
> Email-disclaimer.aspx
>
>         [[alternative HTML version deleted]]
>
> ______________________________________________
> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/
> posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Importing Big data to R

Peter Dalgaard-2
In reply to this post by Mangalani Peter Makananisa

> On 12 Jul 2017, at 08:44 , Mangalani Peter Makananisa <[hidden email]> wrote:
>
> Dear R-Gurus,
>
> I am trying to read in data with 24,349,113 rows to R-3.3.3 (64 bit) and have used  the library   "data.table"  and It managed to read 23,347,070 rows  and the remainder was 2,043 rows only.

I make that 1002043 rows... (typo?). One possibility is that you are skipping lines because of something like quote characters in data or some input lines being one field short. You might also simply be running out of memory.

However,

1) data.table is a contributed package. Its maintainer may be a better point of contact.

2) if you hope to get advice from the R community at large, it would be useful to tell us what you did, what kind of data you try to read using which command, etc. And, as Jim Holtman already said: error messages, if any.

-pd

>
> Could you please advise me as to which library/R-commands is suitable to read the full data in to R?
>
> Kind regards,
>
> Mangalani Peter Makananisa (5786)
> South African Revenue Service (SARS)
> +2782 456 4669 / +2712 422 7357
>
> Please Note: This email and its contents are subject to our email legal notice which can be viewed at http://www.sars.gov.za/Pages/Email-disclaimer.aspx
>
> [[alternative HTML version deleted]]
>
> ______________________________________________
> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

--
Peter Dalgaard, Professor,
Center for Statistics, Copenhagen Business School
Solbjerg Plads 3, 2000 Frederiksberg, Denmark
Phone: (+45)38153501
Office: A 4.23
Email: [hidden email]  Priv: [hidden email]

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Loading...