I am trying to read a 6.9 million row text file with 26 columns separated by
spaces into R using ff. When I specify a small number for first.rows,
next.rows and nrows it is read with no issue. However, when I try to specify
larger next.rows values and no nrows parameter to read the entire file, I
keep getting errors. Please see code below.
I am trying to this on a m1.large EC2 machine running R with 14.8 GB of
memory. I haven't been able to read the entire dataset into memory using
I guess I am not sure given the error message if I need to specify further
A little late, but perhaps this will help someone in the future. I am guessing that some of your "integer" fields contain scientific notation, and for some reason read.table is not interpreting those as integers. Consider changing the affected column classes from "integer" to "numeric" and I bet you'll see better results.