add data to a file while doing a loop

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

add data to a file while doing a loop

Joao Fadista
Hi,

I would like to know how can I keep adding data to a file while doing a loop and without deleting the data of the previous iteration. Thanks.

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: add data to a file while doing a loop

stephen sefick-2
Without context, read reproducible code, it is hard to answer this
question.  What system are you on?  Do you need to write one line of
data or a data frame or a list out?  Are trying to write a graphic out?
 It will be easier to answer your question with some context.  Good
luck!

Stephen

On Fri 06 Jan 2012 05:49:01 AM CST, Joao Fadista wrote:
> Hi,
>
> I would like to know how can I keep adding data to a file while doing a loop and without deleting the data of the previous iteration. Thanks.
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: add data to a file while doing a loop

Joao Fadista
Thanks for the reply. Every iteration produces a text data.frame directly into a file without saving it in R. And every iteration overwrites the data produced from the previous iteration. So what I would like to do is being able not to overwrite the data of any iteration but adding it to the same file with maybe the functions paste or rbind.


> sessionInfo()
R version 2.13.2 (2011-09-30)
Platform: x86_64-pc-mingw32/x64 (64-bit)

locale:
[1] LC_COLLATE=Swedish_Sweden.1252  LC_CTYPE=Swedish_Sweden.1252    LC_MONETARY=Swedish_Sweden.1252 LC_NUMERIC=C                    LC_TIME=Swedish_Sweden.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base    

other attached packages:
[1] MatrixEQTL_1.2.0

loaded via a namespace (and not attached):
[1] tools_2.13.2


Best regards,
João Fadista, Ph.D.
Post Doc

Lund University Diabetes Centre
CRC, Malmö University Hospital
Entrance 72, building 60, level 13
SE-205 02 Malmö, Sweden
Tel: +46 (0)40 391237
e-mail: [hidden email]


-----Original Message-----
From: stephen sefick [mailto:[hidden email]] On Behalf Of Stephen Sefick
Sent: den 6 januari 2012 14:14
To: Joao Fadista
Cc: [hidden email]
Subject: Re: [R] add data to a file while doing a loop

Without context, read reproducible code, it is hard to answer this question.  What system are you on?  Do you need to write one line of data or a data frame or a list out?  Are trying to write a graphic out?
 It will be easier to answer your question with some context.  Good luck!

Stephen

On Fri 06 Jan 2012 05:49:01 AM CST, Joao Fadista wrote:

> Hi,
>
> I would like to know how can I keep adding data to a file while doing a loop and without deleting the data of the previous iteration. Thanks.
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: add data to a file while doing a loop

Rui Barradas
Hello,

See

?open and ?capture.output

or

?textConnection

To open a connection (with 'open') then write to it is probably the solution.

Rui Barradas
Reply | Threaded
Open this post in threaded view
|

Re: add data to a file while doing a loop

Sander Timmer-2
In reply to this post by Joao Fadista
So you're looking for:

write.table(dataset, file="x.txt", append=TRUE)

Or do I understand your question wrongly?



______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: add data to a file while doing a loop

David Winsemius
In reply to this post by Joao Fadista

On Jan 6, 2012, at 8:27 AM, Joao Fadista wrote:

> Thanks for the reply. Every iteration produces a text data.frame  
> directly into a file without saving it in R. And every iteration  
> overwrites the data produced from the previous iteration. So what I  
> would like to do is being able not to overwrite the data of any  
> iteration but adding it to the same file with maybe the functions  
> paste or rbind.

If you are using write.table, then pay more attention to the append  
parameter documented in the help file.
--
David.

>
>
>> sessionInfo()
> R version 2.13.2 (2011-09-30)
> Platform: x86_64-pc-mingw32/x64 (64-bit)
>
> locale:
> [1] LC_COLLATE=Swedish_Sweden.1252  LC_CTYPE=Swedish_Sweden.1252    
> LC_MONETARY=Swedish_Sweden.1252 LC_NUMERIC=C                    
> LC_TIME=Swedish_Sweden.1252
>
> attached base packages:
> [1] stats     graphics  grDevices utils     datasets  methods   base
>
> other attached packages:
> [1] MatrixEQTL_1.2.0
>
> loaded via a namespace (and not attached):
> [1] tools_2.13.2
>
>
> Best regards,
> João Fadista, Ph.D.
> Post Doc
>
> Lund University Diabetes Centre
> CRC, Malmö University Hospital
> Entrance 72, building 60, level 13
> SE-205 02 Malmö, Sweden
> Tel: +46 (0)40 391237
> e-mail: [hidden email]
>
>
> -----Original Message-----
> From: stephen sefick [mailto:[hidden email]] On Behalf Of Stephen  
> Sefick
> Sent: den 6 januari 2012 14:14
> To: Joao Fadista
> Cc: [hidden email]
> Subject: Re: [R] add data to a file while doing a loop
>
> Without context, read reproducible code, it is hard to answer this  
> question.  What system are you on?  Do you need to write one line of  
> data or a data frame or a list out?  Are trying to write a graphic  
> out?
> It will be easier to answer your question with some context.  Good  
> luck!
>
> Stephen
>
> On Fri 06 Jan 2012 05:49:01 AM CST, Joao Fadista wrote:
>> Hi,
>>
>> I would like to know how can I keep adding data to a file while  
>> doing a loop and without deleting the data of the previous  
>> iteration. Thanks.
>>
>> ______________________________________________
>> [hidden email] mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

David Winsemius, MD
West Hartford, CT

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: add data to a file while doing a loop

William Dunlap
In reply to this post by Joao Fadista
Is this an example of what you want to do?

  fileName <- tempfile(fileext=".txt")
  fileCon <- file(fileName, "wt") # a file connection, opened for writing text
  for(i in 1:5) {
      cat(file=fileCon, "Line", i, "\n") # write to connection, not file name
  }
  close(fileCon)

Then look to see what the file contains:

  > readLines(fileName)
  [1] "Line 1 " "Line 2 " "Line 3 " "Line 4 " "Line 5 "

Bill Dunlap
Spotfire, TIBCO Software
wdunlap tibco.com

> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On Behalf Of Joao Fadista
> Sent: Friday, January 06, 2012 5:27 AM
> To: [hidden email]
> Cc: [hidden email]
> Subject: Re: [R] add data to a file while doing a loop
>
> Thanks for the reply. Every iteration produces a text data.frame directly into a file without saving
> it in R. And every iteration overwrites the data produced from the previous iteration. So what I would
> like to do is being able not to overwrite the data of any iteration but adding it to the same file
> with maybe the functions paste or rbind.
>
>
> > sessionInfo()
> R version 2.13.2 (2011-09-30)
> Platform: x86_64-pc-mingw32/x64 (64-bit)
>
> locale:
> [1] LC_COLLATE=Swedish_Sweden.1252  LC_CTYPE=Swedish_Sweden.1252    LC_MONETARY=Swedish_Sweden.1252
> LC_NUMERIC=C                    LC_TIME=Swedish_Sweden.1252
>
> attached base packages:
> [1] stats     graphics  grDevices utils     datasets  methods   base
>
> other attached packages:
> [1] MatrixEQTL_1.2.0
>
> loaded via a namespace (and not attached):
> [1] tools_2.13.2
>
>
> Best regards,
> João Fadista, Ph.D.
> Post Doc
>
> Lund University Diabetes Centre
> CRC, Malmö University Hospital
> Entrance 72, building 60, level 13
> SE-205 02 Malmö, Sweden
> Tel: +46 (0)40 391237
> e-mail: [hidden email]
>
>
> -----Original Message-----
> From: stephen sefick [mailto:[hidden email]] On Behalf Of Stephen Sefick
> Sent: den 6 januari 2012 14:14
> To: Joao Fadista
> Cc: [hidden email]
> Subject: Re: [R] add data to a file while doing a loop
>
> Without context, read reproducible code, it is hard to answer this question.  What system are you on?
> Do you need to write one line of data or a data frame or a list out?  Are trying to write a graphic
> out?
>  It will be easier to answer your question with some context.  Good luck!
>
> Stephen
>
> On Fri 06 Jan 2012 05:49:01 AM CST, Joao Fadista wrote:
> > Hi,
> >
> > I would like to know how can I keep adding data to a file while doing a loop and without deleting
> the data of the previous iteration. Thanks.
> >
> > ______________________________________________
> > [hidden email] mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide
> > http://www.R-project.org/posting-guide.html
> > and provide commented, minimal, self-contained, reproducible code.
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: add data to a file while doing a loop

MacQueen, Don
In reply to this post by Joao Fadista
Look at the documentation for whatever function you are using to write
data to the file.
It should be pretty obvious (look for an "append" argument).

Otherwise you'll have to provide more information, such as a short simple
example of what you have tried.

-Don

--
Don MacQueen

Lawrence Livermore National Laboratory
7000 East Ave., L-627
Livermore, CA 94550
925-423-1062





On 1/6/12 3:49 AM, "Joao Fadista" <[hidden email]> wrote:

>Hi,
>
>I would like to know how can I keep adding data to a file while doing a
>loop and without deleting the data of the previous iteration. Thanks.
>
>______________________________________________
>[hidden email] mailing list
>https://stat.ethz.ch/mailman/listinfo/r-help
>PLEASE do read the posting guide
>http://www.R-project.org/posting-guide.html
>and provide commented, minimal, self-contained, reproducible code.

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: add data to a file while doing a loop

William Dunlap
Using append=TRUE in many functions will work, but
more slowly than opening a connection once, writing
to the connection many times, then closing it.
(Opening a file is a pretty expensive operation,
while writing to it is much cheaper.)  Some more
recently written functions do not have an append=
argument because you can achieve the same with connections.

E.g., I wrote functions that used
cat(file=fileName, append=TRUE) repeatedly
and the open/cat-repeatedly/close method:

  f0 <- function (n, fileName)
  {
      unlink(fileName)
      system.time(for (i in seq_len(n)) cat("Line", i, "\n", file = fileName,
          append = TRUE))
  }
  f1 <- function (n, fileName) {
      unlink(fileName)
      system.time({
          fileConn <- file(fileName, "wt")
          on.exit(close(fileConn))
          for (i in seq_len(n)) cat("Line", i, "\n", file = fileConn)
      })
  }

and recorded the time they took to write
1000, 2000, and 20000 lines on my Window XP
laptop:

  > tf0 <- tempfile()
  > f0(1*10^3, tf0)
     user  system elapsed
     0.16    0.45    8.25
  > f0(2*10^3, tf0)
     user  system elapsed
     0.36    0.98   17.86
  > f0(20*10^3, tf0)
     user  system elapsed
     5.03   10.64  393.95
  >
  > tf1 <- tempfile()
  > f1(1*10^3, tf1)
     user  system elapsed
     0.05    0.09    0.15
  > f1(2*10^3, tf1)
     user  system elapsed
     0.02    0.08    0.11
  > f1(20*10^3, tf1)
     user  system elapsed
     0.30    0.70    0.98
 
Note that they produced identical output files:
  > identical(readLines(tf0), readLines(tf1))
  [1] TRUE
and the connection-oriented version is still usable
for a million or two iterations:
  > f1(1e6, tf1)
     user  system elapsed
    15.40   30.05   45.19
  > f1(2e6, tf1)
     user  system elapsed
    31.95   60.29   91.42

Any of the standard functions with a file= (or con=)
argument will accept a connction object instead of
a file name.  If you use the connection object you
don't need to restrict yourself to functions with
an append= argument to append to a file.

Bill Dunlap
Spotfire, TIBCO Software
wdunlap tibco.com

> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On Behalf Of MacQueen, Don
> Sent: Friday, January 06, 2012 9:32 AM
> To: Joao Fadista; [hidden email]
> Subject: Re: [R] add data to a file while doing a loop
>
> Look at the documentation for whatever function you are using to write
> data to the file.
> It should be pretty obvious (look for an "append" argument).
>
> Otherwise you'll have to provide more information, such as a short simple
> example of what you have tried.
>
> -Don
>
> --
> Don MacQueen
>
> Lawrence Livermore National Laboratory
> 7000 East Ave., L-627
> Livermore, CA 94550
> 925-423-1062
>
>
>
>
>
> On 1/6/12 3:49 AM, "Joao Fadista" <[hidden email]> wrote:
>
> >Hi,
> >
> >I would like to know how can I keep adding data to a file while doing a
> >loop and without deleting the data of the previous iteration. Thanks.
> >
> >______________________________________________
> >[hidden email] mailing list
> >https://stat.ethz.ch/mailman/listinfo/r-help
> >PLEASE do read the posting guide
> >http://www.R-project.org/posting-guide.html
> >and provide commented, minimal, self-contained, reproducible code.
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: add data to a file while doing a loop

Joao Fadista
Thank you all for your useful replies. I got it to work!

Best regards,
João Fadista, Ph.D.
Post Doc

Lund University Diabetes Centre
CRC, Malmö University Hospital
Entrance 72, building 60, level 13
SE-205 02 Malmö, Sweden
Tel: +46 (0)40 391237
e-mail: [hidden email]


-----Original Message-----
From: William Dunlap [mailto:[hidden email]]
Sent: den 6 januari 2012 19:27
To: MacQueen, Don; Joao Fadista; [hidden email]
Subject: RE: [R] add data to a file while doing a loop

Using append=TRUE in many functions will work, but more slowly than opening a connection once, writing to the connection many times, then closing it.
(Opening a file is a pretty expensive operation, while writing to it is much cheaper.)  Some more recently written functions do not have an append= argument because you can achieve the same with connections.

E.g., I wrote functions that used
cat(file=fileName, append=TRUE) repeatedly and the open/cat-repeatedly/close method:

  f0 <- function (n, fileName)
  {
      unlink(fileName)
      system.time(for (i in seq_len(n)) cat("Line", i, "\n", file = fileName,
          append = TRUE))
  }
  f1 <- function (n, fileName) {
      unlink(fileName)
      system.time({
          fileConn <- file(fileName, "wt")
          on.exit(close(fileConn))
          for (i in seq_len(n)) cat("Line", i, "\n", file = fileConn)
      })
  }

and recorded the time they took to write 1000, 2000, and 20000 lines on my Window XP
laptop:

  > tf0 <- tempfile()
  > f0(1*10^3, tf0)
     user  system elapsed
     0.16    0.45    8.25
  > f0(2*10^3, tf0)
     user  system elapsed
     0.36    0.98   17.86
  > f0(20*10^3, tf0)
     user  system elapsed
     5.03   10.64  393.95
  >
  > tf1 <- tempfile()
  > f1(1*10^3, tf1)
     user  system elapsed
     0.05    0.09    0.15
  > f1(2*10^3, tf1)
     user  system elapsed
     0.02    0.08    0.11
  > f1(20*10^3, tf1)
     user  system elapsed
     0.30    0.70    0.98
 
Note that they produced identical output files:
  > identical(readLines(tf0), readLines(tf1))
  [1] TRUE
and the connection-oriented version is still usable for a million or two iterations:
  > f1(1e6, tf1)
     user  system elapsed
    15.40   30.05   45.19
  > f1(2e6, tf1)
     user  system elapsed
    31.95   60.29   91.42

Any of the standard functions with a file= (or con=) argument will accept a connction object instead of a file name.  If you use the connection object you don't need to restrict yourself to functions with an append= argument to append to a file.

Bill Dunlap
Spotfire, TIBCO Software
wdunlap tibco.com

> -----Original Message-----
> From: [hidden email]
> [mailto:[hidden email]] On Behalf Of MacQueen, Don
> Sent: Friday, January 06, 2012 9:32 AM
> To: Joao Fadista; [hidden email]
> Subject: Re: [R] add data to a file while doing a loop
>
> Look at the documentation for whatever function you are using to write
> data to the file.
> It should be pretty obvious (look for an "append" argument).
>
> Otherwise you'll have to provide more information, such as a short
> simple example of what you have tried.
>
> -Don
>
> --
> Don MacQueen
>
> Lawrence Livermore National Laboratory
> 7000 East Ave., L-627
> Livermore, CA 94550
> 925-423-1062
>
>
>
>
>
> On 1/6/12 3:49 AM, "Joao Fadista" <[hidden email]> wrote:
>
> >Hi,
> >
> >I would like to know how can I keep adding data to a file while doing
> >a loop and without deleting the data of the previous iteration. Thanks.
> >
> >______________________________________________
> >[hidden email] mailing list
> >https://stat.ethz.ch/mailman/listinfo/r-help
> >PLEASE do read the posting guide
> >http://www.R-project.org/posting-guide.html
> >and provide commented, minimal, self-contained, reproducible code.
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.