capturing stderr/stdout

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

capturing stderr/stdout

Sundar Dorai-Raj
Hi,

I have an application in perl that prints some output to either stderr
or stdout.

Here's an example:

# tmp.pl
print STDERR "starting iterator\n";
for(my $i = 0; $i < 1000000; $i++) {
   print $i . "\n";
}

# tmp.R
con <- pipe("perl tmp.pl")
r <- readLines(con, n = -1)
close(con)

However, the second line stalls until the perl for-loop finishes. What I
would like is to process each line as it comes. E.g. something like:

while(TRUE) {
   r <- readLines(con, n = 1) # read one line
   if(r == "10000") print(r)
   if(length(r) == 0) break
}

Of course, this won't work since I'm not calling readLines
appropriately. Answers must work on Windows but may include cygwin
utilities if necessary. Any advice would be appreciated. Version info at
the end if it matters.

Thanks, --sundar


 > version
                _
platform       i386-pc-mingw32
arch           i386
os             mingw32
system         i386, mingw32
status
major          2
minor          8.0
year           2008
month          10
day            20
svn rev        46754
language       R
version.string R version 2.8.0 (2008-10-20)

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: capturing stderr/stdout

Prof Brian Ripley
I am not sure what the issue is here. Do you want to capture both stderr
and stdout (use 2>1 in the command with an sh-like shell), or is the
problem that you don't get immediate output?

The latter is a Perl issue: you need to circumvent output buffering.
See e.g

http://perl.plover.com/FAQs/Buffering.html

Sundar Dorai-Raj wrote:

> Hi,
>
> I have an application in perl that prints some output to either stderr
> or stdout.
>
> Here's an example:
>
> # tmp.pl
> print STDERR "starting iterator\n";
> for(my $i = 0; $i < 1000000; $i++) {
>   print $i . "\n";
> }
>
> # tmp.R
> con <- pipe("perl tmp.pl")
> r <- readLines(con, n = -1)
> close(con)
>
> However, the second line stalls until the perl for-loop finishes. What I
> would like is to process each line as it comes. E.g. something like:
>
> while(TRUE) {
>   r <- readLines(con, n = 1) # read one line
>   if(r == "10000") print(r)
>   if(length(r) == 0) break
> }
>
> Of course, this won't work since I'm not calling readLines
> appropriately. Answers must work on Windows but may include cygwin
> utilities if necessary. Any advice would be appreciated. Version info at
> the end if it matters.
>
> Thanks, --sundar
>
>
>  > version
>                _
> platform       i386-pc-mingw32
> arch           i386
> os             mingw32
> system         i386, mingw32
> status
> major          2
> minor          8.0
> year           2008
> month          10
> day            20
> svn rev        46754
> language       R
> version.string R version 2.8.0 (2008-10-20)
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.


--
Brian D. Ripley,                  [hidden email]
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: capturing stderr/stdout

Sundar Dorai-Raj
Hi, Prof. Ripley,

Thanks for the reply. Mostly I want to capture output as it is written
to the stream. For example, I quite often do the following to view the
progress of a log file from a computationally intensive script.

1. Open a console and type:

Rscript script.R >& script.log

which directs both stdout and stderr to "script.log"

2. Open another console and type:

tail -f script.log

This way I get both the script's log file and its current progress.

I guess my question is: Is there a way to accomplish the "tail -f"
command in R?

Thanks,

--sundar

Prof Brian Ripley said the following on 11/20/2008 11:43 PM:

> I am not sure what the issue is here. Do you want to capture both stderr
> and stdout (use 2>1 in the command with an sh-like shell), or is the
> problem that you don't get immediate output?
>
> The latter is a Perl issue: you need to circumvent output buffering.
> See e.g
>
> http://perl.plover.com/FAQs/Buffering.html
>
> Sundar Dorai-Raj wrote:
>> Hi,
>>
>> I have an application in perl that prints some output to either stderr
>> or stdout.
>>
>> Here's an example:
>>
>> # tmp.pl
>> print STDERR "starting iterator\n";
>> for(my $i = 0; $i < 1000000; $i++) {
>>   print $i . "\n";
>> }
>>
>> # tmp.R
>> con <- pipe("perl tmp.pl")
>> r <- readLines(con, n = -1)
>> close(con)
>>
>> However, the second line stalls until the perl for-loop finishes. What
>> I would like is to process each line as it comes. E.g. something like:
>>
>> while(TRUE) {
>>   r <- readLines(con, n = 1) # read one line
>>   if(r == "10000") print(r)
>>   if(length(r) == 0) break
>> }
>>
>> Of course, this won't work since I'm not calling readLines
>> appropriately. Answers must work on Windows but may include cygwin
>> utilities if necessary. Any advice would be appreciated. Version info
>> at the end if it matters.
>>
>> Thanks, --sundar
>>
>>
>>  > version
>>                _
>> platform       i386-pc-mingw32
>> arch           i386
>> os             mingw32
>> system         i386, mingw32
>> status
>> major          2
>> minor          8.0
>> year           2008
>> month          10
>> day            20
>> svn rev        46754
>> language       R
>> version.string R version 2.8.0 (2008-10-20)
>>
>> ______________________________________________
>> [hidden email] mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>
>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: capturing stderr/stdout

Prof Brian Ripley
Sundar Dorai-Raj wrote:

> Hi, Prof. Ripley,
>
> Thanks for the reply. Mostly I want to capture output as it is written
> to the stream. For example, I quite often do the following to view the
> progress of a log file from a computationally intensive script.
>
> 1. Open a console and type:
>
> Rscript script.R >& script.log
>
> which directs both stdout and stderr to "script.log"
>
> 2. Open another console and type:
>
> tail -f script.log
>
> This way I get both the script's log file and its current progress.

The usual way to so this is via 'tee' .

> I guess my question is: Is there a way to accomplish the "tail -f"
> command in R?

Yes, but why would you want to?  Note that sink() has a 'split' option,
so you could in principle do all this in your script.R.

Otherwise you can write a simple R loop to do readLines(n=1);
writeLines() to emulate tail -F.


> Thanks,
>
> --sundar
>
> Prof Brian Ripley said the following on 11/20/2008 11:43 PM:
>> I am not sure what the issue is here. Do you want to capture both
>> stderr and stdout (use 2>1 in the command with an sh-like shell), or
>> is the
>> problem that you don't get immediate output?
>>
>> The latter is a Perl issue: you need to circumvent output buffering.
>> See e.g
>>
>> http://perl.plover.com/FAQs/Buffering.html
>>
>> Sundar Dorai-Raj wrote:
>>> Hi,
>>>
>>> I have an application in perl that prints some output to either
>>> stderr or stdout.
>>>
>>> Here's an example:
>>>
>>> # tmp.pl
>>> print STDERR "starting iterator\n";
>>> for(my $i = 0; $i < 1000000; $i++) {
>>>   print $i . "\n";
>>> }
>>>
>>> # tmp.R
>>> con <- pipe("perl tmp.pl")
>>> r <- readLines(con, n = -1)
>>> close(con)
>>>
>>> However, the second line stalls until the perl for-loop finishes.
>>> What I would like is to process each line as it comes. E.g. something
>>> like:
>>>
>>> while(TRUE) {
>>>   r <- readLines(con, n = 1) # read one line
>>>   if(r == "10000") print(r)
>>>   if(length(r) == 0) break
>>> }
>>>
>>> Of course, this won't work since I'm not calling readLines
>>> appropriately. Answers must work on Windows but may include cygwin
>>> utilities if necessary. Any advice would be appreciated. Version info
>>> at the end if it matters.
>>>
>>> Thanks, --sundar
>>>
>>>
>>>  > version
>>>                _
>>> platform       i386-pc-mingw32
>>> arch           i386
>>> os             mingw32
>>> system         i386, mingw32
>>> status
>>> major          2
>>> minor          8.0
>>> year           2008
>>> month          10
>>> day            20
>>> svn rev        46754
>>> language       R
>>> version.string R version 2.8.0 (2008-10-20)
>>>
>>> ______________________________________________
>>> [hidden email] mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>> PLEASE do read the posting guide
>>> http://www.R-project.org/posting-guide.html
>>> and provide commented, minimal, self-contained, reproducible code.
>>
>>


--
Brian D. Ripley,                  [hidden email]
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.