Error when calling R from Python

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

Error when calling R from Python

Marcel Baumgartner
Dear all,

my colleague posted our issue on stackoverflow:

Calling R script from Python does not save log file in version 4 -
Stack Overflow
[https://stackoverflow.com/questions/65887485/calling-r-script-from-python-does-not-save-log-file-in-version-4]

It is about this kind of call to R:

R.exe -f code.R --args "~/file.txt" 1> "~/log.txt" 2>&1".

The issue is that the log.txt file is not created when running R
4.x.x. The same code works perfectly fine with R 3.6.x.

Any idea what's going wrong as of version 4? Regards Marcel 



        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Error when calling R from Python

Martin Maechler
>>>>> Marcel Baumgartner
>>>>>     on Tue, 26 Jan 2021 08:55:48 +0100 writes:

    > Dear all, my colleague posted our issue on stackoverflow:

    > Calling R script from Python does not save log file in
    > version 4 - Stack Overflow
    > [https://stackoverflow.com/questions/65887485/calling-r-script-from-python-does-not-save-log-file-in-version-4]

    > It is about this kind of call to R:

    > R.exe -f code.R --args "~/file.txt" 1> "~/log.txt" 2>&1".

    > The issue is that the log.txt file is not created when
    > running R 4.x.x. The same code works perfectly fine with R
    > 3.6.x.

    > Any idea what's going wrong as of version 4? Regards
    > Marcel 

Dear Marcel,
I think the solution is embarrassingly simple:

From the SO post, where she showed a bit more detail than you
show here, it's clear  you have confused
'R.exe' and 'Rscript.exe' and what you say above is not true:

'R.exe' was used for R 3.6.0  but for R 4.0.3, you/she used
'Rscript.exe' instead.

... as you've noticed now, they do behave differently, indeed!

Avec meilleures salutations,
Martin

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Error when calling (R 4.0.x on Windows) from Python

Martin Maechler
>>>>> Martin Maechler
>>>>>     on Tue, 26 Jan 2021 12:37:58 +0100 writes:

>>>>> Marcel Baumgartner
>>>>>     on Tue, 26 Jan 2021 08:55:48 +0100 writes:

    >> Dear all, my colleague posted our issue on stackoverflow:

    >> Calling R script from Python does not save log file in
    >> version 4 - Stack Overflow
    >> [https://stackoverflow.com/questions/65887485/calling-r-script-from-python-does-not-save-log-file-in-version-4]

    >> It is about this kind of call to R:

    >> R.exe -f code.R --args "~/file.txt" 1> "~/log.txt" 2>&1".

    >> The issue is that the log.txt file is not created when
    >> running R 4.x.x. The same code works perfectly fine with
    >> R 3.6.x.

    >> Any idea what's going wrong as of version 4? Regards
    >> Marcel 

    > Dear Marcel, I think the solution is embarrassingly
    > simple:

    >> From the SO post, where she showed a bit more detail than you
    > show here, it's clear you have confused 'R.exe' and
    > 'Rscript.exe' and what you say above is not true:

    > 'R.exe' was used for R 3.6.0 but for R 4.0.3, you/she used
    > 'Rscript.exe' instead.


    > ... as you've noticed now, they do behave differently,
    > indeed!

Well, this was not the solution to their -- Windows-only -- problem.
The problem *is* indeed visible if they only use  R.exe  (also
for R 4.0.3).

I've commented more on the SO issue (see above),
notably asking for a *minimal* repr.ex. (reproducible example),
and one *not* using "<YOUR PATH>" and setwd() ..

Martin

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Error when calling (R 4.0.x on Windows) from Python

Bill Dunlap-2
Note that in R-3.6.3 commandArgs() does not include the arguments
intended to be processed by the shell, "1>", "arguments.txt", etc.,
but in R-4.0.3 it does include them.  It is as though an R shell()
command was replaced by a system() command so cmd.exe didn't get a
chance to process the command line.

-Bill

On Wed, Jan 27, 2021 at 12:39 AM Martin Maechler
<[hidden email]> wrote:

>
> >>>>> Martin Maechler
> >>>>>     on Tue, 26 Jan 2021 12:37:58 +0100 writes:
>
> >>>>> Marcel Baumgartner
> >>>>>     on Tue, 26 Jan 2021 08:55:48 +0100 writes:
>
>     >> Dear all, my colleague posted our issue on stackoverflow:
>
>     >> Calling R script from Python does not save log file in
>     >> version 4 - Stack Overflow
>     >> [https://stackoverflow.com/questions/65887485/calling-r-script-from-python-does-not-save-log-file-in-version-4]
>
>     >> It is about this kind of call to R:
>
>     >> R.exe -f code.R --args "~/file.txt" 1> "~/log.txt" 2>&1".
>
>     >> The issue is that the log.txt file is not created when
>     >> running R 4.x.x. The same code works perfectly fine with
>     >> R 3.6.x.
>
>     >> Any idea what's going wrong as of version 4? Regards
>     >> Marcel
>
>     > Dear Marcel, I think the solution is embarrassingly
>     > simple:
>
>     >> From the SO post, where she showed a bit more detail than you
>     > show here, it's clear you have confused 'R.exe' and
>     > 'Rscript.exe' and what you say above is not true:
>
>     > 'R.exe' was used for R 3.6.0 but for R 4.0.3, you/she used
>     > 'Rscript.exe' instead.
>
>
>     > ... as you've noticed now, they do behave differently,
>     > indeed!
>
> Well, this was not the solution to their -- Windows-only -- problem.
> The problem *is* indeed visible if they only use  R.exe  (also
> for R 4.0.3).
>
> I've commented more on the SO issue (see above),
> notably asking for a *minimal* repr.ex. (reproducible example),
> and one *not* using "<YOUR PATH>" and setwd() ..
>
> Martin
>
> ______________________________________________
> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Error when calling (R 4.0.x on Windows) from Python

Bill Dunlap-2
I mistyped, 3.6.3 omits "1>", "log.txt" (not arguments.txt"), etc.  My
test was run in C:\tmp and used -e instead of -f and bin/R.exe instead
of bin/x64/R.exe, using the following python script:

import subprocess

cmd363 = " ".join(["C:/R/R-3.6.3/bin/R.exe", "--vanilla", "--quiet",
         # "-f", '"code.R"',
         "-e", '"commandArgs()"',
         "--args", '"' + 'arguments.txt"',"1>", '"' + 'log363.txt"', "2>&1"])
print(cmd363)
status363 = subprocess.call(cmd363)

cmd403 = " ".join(["C:/R/R-4.0.3/bin/R.exe", "--vanilla", "--quiet",
         # "-f", '"code.R"',
         "-e", '"commandArgs()"',
         "--args", '"' + 'arguments.txt"',"1>", '"' + 'log403.txt"', "2>&1"])
print(cmd403)
status403 = subprocess.call(cmd403)

This gave

>>> import subprocess
>>>
>>> cmd363 = " ".join(["C:/R/R-3.6.3/bin/R.exe", "--vanilla", "--quiet",
...          # "-f", '"code.R"',
...          "-e", '"commandArgs()"',
...          "--args", '"' + 'arguments.txt"',"1>", '"' +
'log363.txt"', "2>&1"])
>>> print(cmd363)
C:/R/R-3.6.3/bin/R.exe --vanilla --quiet -e "commandArgs()" --args
"arguments.txt" 1> "log363.txt" 2>&1
>>> status363 = subprocess.call(cmd363)
>>>
>>> cmd403 = " ".join(["C:/R/R-4.0.3/bin/R.exe", "--vanilla", "--quiet",
...          # "-f", '"code.R"',
...          "-e", '"commandArgs()"',
...          "--args", '"' + 'arguments.txt"',"1>", '"' +
'log403.txt"', "2>&1"])
>>> print(cmd403)
C:/R/R-4.0.3/bin/R.exe --vanilla --quiet -e "commandArgs()" --args
"arguments.txt" 1> "log403.txt" 2>&1
>>> status403 = subprocess.call(cmd403)
> commandArgs()
 [1] "C:\\R\\R-4.0.3/bin/x64/Rterm.exe" "--vanilla"
 [3] "--quiet"                          "-e"
 [5] "commandArgs()"                    "--args"
 [7] "arguments.txt"                    "1>"
 [9] "log403.txt"                       "2>&1"

After exiting python I log403.txt does not exist and log363.txt shows
the command arguments that R got:

C:\R\R-3.6.3>cat log403.txt
cat: log403.txt: No such file or directory

C:\R\R-3.6.3>cat log363.txt
> commandArgs()
[1] "C:\\R\\R-3.6.3/bin/x64/Rterm.exe" "--vanilla"
[3] "--quiet"                          "-e"
[5] "commandArgs()"                    "--args"
[7] "arguments.txt"

On Wed, Jan 27, 2021 at 10:34 AM Bill Dunlap <[hidden email]> wrote:

>
> Note that in R-3.6.3 commandArgs() does not include the arguments
> intended to be processed by the shell, "1>", "arguments.txt", etc.,
> but in R-4.0.3 it does include them.  It is as though an R shell()
> command was replaced by a system() command so cmd.exe didn't get a
> chance to process the command line.
>
> -Bill
>
> On Wed, Jan 27, 2021 at 12:39 AM Martin Maechler
> <[hidden email]> wrote:
> >
> > >>>>> Martin Maechler
> > >>>>>     on Tue, 26 Jan 2021 12:37:58 +0100 writes:
> >
> > >>>>> Marcel Baumgartner
> > >>>>>     on Tue, 26 Jan 2021 08:55:48 +0100 writes:
> >
> >     >> Dear all, my colleague posted our issue on stackoverflow:
> >
> >     >> Calling R script from Python does not save log file in
> >     >> version 4 - Stack Overflow
> >     >> [https://stackoverflow.com/questions/65887485/calling-r-script-from-python-does-not-save-log-file-in-version-4]
> >
> >     >> It is about this kind of call to R:
> >
> >     >> R.exe -f code.R --args "~/file.txt" 1> "~/log.txt" 2>&1".
> >
> >     >> The issue is that the log.txt file is not created when
> >     >> running R 4.x.x. The same code works perfectly fine with
> >     >> R 3.6.x.
> >
> >     >> Any idea what's going wrong as of version 4? Regards
> >     >> Marcel
> >
> >     > Dear Marcel, I think the solution is embarrassingly
> >     > simple:
> >
> >     >> From the SO post, where she showed a bit more detail than you
> >     > show here, it's clear you have confused 'R.exe' and
> >     > 'Rscript.exe' and what you say above is not true:
> >
> >     > 'R.exe' was used for R 3.6.0 but for R 4.0.3, you/she used
> >     > 'Rscript.exe' instead.
> >
> >
> >     > ... as you've noticed now, they do behave differently,
> >     > indeed!
> >
> > Well, this was not the solution to their -- Windows-only -- problem.
> > The problem *is* indeed visible if they only use  R.exe  (also
> > for R 4.0.3).
> >
> > I've commented more on the SO issue (see above),
> > notably asking for a *minimal* repr.ex. (reproducible example),
> > and one *not* using "<YOUR PATH>" and setwd() ..
> >
> > Martin
> >
> > ______________________________________________
> > [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> > and provide commented, minimal, self-contained, reproducible code.

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Error when calling (R 4.0.x on Windows) from Python

Duncan Murdoch-2
In reply to this post by Martin Maechler
On 27/01/2021 3:38 a.m., Martin Maechler wrote:

>>>>>> Martin Maechler
>>>>>>      on Tue, 26 Jan 2021 12:37:58 +0100 writes:
>
>>>>>> Marcel Baumgartner
>>>>>>      on Tue, 26 Jan 2021 08:55:48 +0100 writes:
>
>      >> Dear all, my colleague posted our issue on stackoverflow:
>
>      >> Calling R script from Python does not save log file in
>      >> version 4 - Stack Overflow
>      >> [https://stackoverflow.com/questions/65887485/calling-r-script-from-python-does-not-save-log-file-in-version-4]
>
>      >> It is about this kind of call to R:
>
>      >> R.exe -f code.R --args "~/file.txt" 1> "~/log.txt" 2>&1".
>
>      >> The issue is that the log.txt file is not created when
>      >> running R 4.x.x. The same code works perfectly fine with
>      >> R 3.6.x.
>
>      >> Any idea what's going wrong as of version 4? Regards
>      >> Marcel
>
>      > Dear Marcel, I think the solution is embarrassingly
>      > simple:
>
>      >> From the SO post, where she showed a bit more detail than you
>      > show here, it's clear you have confused 'R.exe' and
>      > 'Rscript.exe' and what you say above is not true:
>
>      > 'R.exe' was used for R 3.6.0 but for R 4.0.3, you/she used
>      > 'Rscript.exe' instead.
>
>
>      > ... as you've noticed now, they do behave differently,
>      > indeed!
>
> Well, this was not the solution to their -- Windows-only -- problem.
> The problem *is* indeed visible if they only use  R.exe  (also
> for R 4.0.3).
>
> I've commented more on the SO issue (see above),
> notably asking for a *minimal* repr.ex. (reproducible example),
> and one *not* using "<YOUR PATH>" and setwd() ..
>

Isn't this purely a Python or user problem?  R shouldn't process
redirection directives like

   1> "~/log.txt" 2>&1

because it's the shell's job to process those. If Python is acting as
the shell, it needs to handle those things.  If R was handling the
command via

Duncan Murdoch

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Error when calling (R 4.0.x on Windows) from Python

Duncan Murdoch-2
On 27/01/2021 3:17 p.m., Duncan Murdoch wrote:

> On 27/01/2021 3:38 a.m., Martin Maechler wrote:
>>>>>>> Martin Maechler
>>>>>>>       on Tue, 26 Jan 2021 12:37:58 +0100 writes:
>>
>>>>>>> Marcel Baumgartner
>>>>>>>       on Tue, 26 Jan 2021 08:55:48 +0100 writes:
>>
>>       >> Dear all, my colleague posted our issue on stackoverflow:
>>
>>       >> Calling R script from Python does not save log file in
>>       >> version 4 - Stack Overflow
>>       >> [https://stackoverflow.com/questions/65887485/calling-r-script-from-python-does-not-save-log-file-in-version-4]
>>
>>       >> It is about this kind of call to R:
>>
>>       >> R.exe -f code.R --args "~/file.txt" 1> "~/log.txt" 2>&1".
>>
>>       >> The issue is that the log.txt file is not created when
>>       >> running R 4.x.x. The same code works perfectly fine with
>>       >> R 3.6.x.
>>
>>       >> Any idea what's going wrong as of version 4? Regards
>>       >> Marcel
>>
>>       > Dear Marcel, I think the solution is embarrassingly
>>       > simple:
>>
>>       >> From the SO post, where she showed a bit more detail than you
>>       > show here, it's clear you have confused 'R.exe' and
>>       > 'Rscript.exe' and what you say above is not true:
>>
>>       > 'R.exe' was used for R 3.6.0 but for R 4.0.3, you/she used
>>       > 'Rscript.exe' instead.
>>
>>
>>       > ... as you've noticed now, they do behave differently,
>>       > indeed!
>>
>> Well, this was not the solution to their -- Windows-only -- problem.
>> The problem *is* indeed visible if they only use  R.exe  (also
>> for R 4.0.3).
>>
>> I've commented more on the SO issue (see above),
>> notably asking for a *minimal* repr.ex. (reproducible example),
>> and one *not* using "<YOUR PATH>" and setwd() ..
>>
>
> Isn't this purely a Python or user problem?  R shouldn't process
> redirection directives like
>
>     1> "~/log.txt" 2>&1
>
> because it's the shell's job to process those. If Python is acting as
> the shell, it needs to handle those things.  If R was handling the
> command via

Oops, sent before finishing:

If R was handling the command via system() or system2(), it would handle
redirection itself. If it was using the Windows-only shell(), it would
call cmd.exe (by default) to handle redirection.  (This is a difference
between R on Windows and R in Unix:  in Unix a shell is always used.)

Duncan Murdoch

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Error when calling (R 4.0.x on Windows) from Python

Bill Dunlap-2
I believe the problem is from svn 77925 in gnuwin/front-ends/rcmdfn.c,
which was committed a few days after 3.6.3 was released.  Rterm used
to put double quotes around a command line argument only if it
contained a space, now it double quotes all arguments.  It sees shell
constructs like "1>" and the following file name as arguments and
double quoting them hides them from the shell, leading to this
problem.  I think we may have to rely on the user supplying quotes as
needed instead of blindly adding them.

-Bill

On Wed, Jan 27, 2021 at 12:28 PM Duncan Murdoch
<[hidden email]> wrote:

>
> On 27/01/2021 3:17 p.m., Duncan Murdoch wrote:
> > On 27/01/2021 3:38 a.m., Martin Maechler wrote:
> >>>>>>> Martin Maechler
> >>>>>>>       on Tue, 26 Jan 2021 12:37:58 +0100 writes:
> >>
> >>>>>>> Marcel Baumgartner
> >>>>>>>       on Tue, 26 Jan 2021 08:55:48 +0100 writes:
> >>
> >>       >> Dear all, my colleague posted our issue on stackoverflow:
> >>
> >>       >> Calling R script from Python does not save log file in
> >>       >> version 4 - Stack Overflow
> >>       >> [https://stackoverflow.com/questions/65887485/calling-r-script-from-python-does-not-save-log-file-in-version-4]
> >>
> >>       >> It is about this kind of call to R:
> >>
> >>       >> R.exe -f code.R --args "~/file.txt" 1> "~/log.txt" 2>&1".
> >>
> >>       >> The issue is that the log.txt file is not created when
> >>       >> running R 4.x.x. The same code works perfectly fine with
> >>       >> R 3.6.x.
> >>
> >>       >> Any idea what's going wrong as of version 4? Regards
> >>       >> Marcel
> >>
> >>       > Dear Marcel, I think the solution is embarrassingly
> >>       > simple:
> >>
> >>       >> From the SO post, where she showed a bit more detail than you
> >>       > show here, it's clear you have confused 'R.exe' and
> >>       > 'Rscript.exe' and what you say above is not true:
> >>
> >>       > 'R.exe' was used for R 3.6.0 but for R 4.0.3, you/she used
> >>       > 'Rscript.exe' instead.
> >>
> >>
> >>       > ... as you've noticed now, they do behave differently,
> >>       > indeed!
> >>
> >> Well, this was not the solution to their -- Windows-only -- problem.
> >> The problem *is* indeed visible if they only use  R.exe  (also
> >> for R 4.0.3).
> >>
> >> I've commented more on the SO issue (see above),
> >> notably asking for a *minimal* repr.ex. (reproducible example),
> >> and one *not* using "<YOUR PATH>" and setwd() ..
> >>
> >
> > Isn't this purely a Python or user problem?  R shouldn't process
> > redirection directives like
> >
> >     1> "~/log.txt" 2>&1
> >
> > because it's the shell's job to process those. If Python is acting as
> > the shell, it needs to handle those things.  If R was handling the
> > command via
>
> Oops, sent before finishing:
>
> If R was handling the command via system() or system2(), it would handle
> redirection itself. If it was using the Windows-only shell(), it would
> call cmd.exe (by default) to handle redirection.  (This is a difference
> between R on Windows and R in Unix:  in Unix a shell is always used.)
>
> Duncan Murdoch
>
> ______________________________________________
> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Error when calling (R 4.0.x on Windows) from Python

Duncan Murdoch-2
On 27/01/2021 3:40 p.m., Bill Dunlap wrote:
> I believe the problem is from svn 77925 in gnuwin/front-ends/rcmdfn.c,
> which was committed a few days after 3.6.3 was released.  Rterm used
> to put double quotes around a command line argument only if it
> contained a space, now it double quotes all arguments.  It sees shell
> constructs like "1>" and the following file name as arguments and
> double quoting them hides them from the shell, leading to this
> problem.  I think we may have to rely on the user supplying quotes as
> needed instead of blindly adding them.

Okay, now I see what you mean.

If you invoke R using R.exe, it asks cmd.exe to run Rterm.exe, so it is
possible that redirection would be handled.

If you invoke R directly using Rterm.exe, then my description down below
would be correct.

Duncan Murdoch


>
> -Bill
>
> On Wed, Jan 27, 2021 at 12:28 PM Duncan Murdoch
> <[hidden email]> wrote:
>>
>> On 27/01/2021 3:17 p.m., Duncan Murdoch wrote:
>>> On 27/01/2021 3:38 a.m., Martin Maechler wrote:
>>>>>>>>> Martin Maechler
>>>>>>>>>        on Tue, 26 Jan 2021 12:37:58 +0100 writes:
>>>>
>>>>>>>>> Marcel Baumgartner
>>>>>>>>>        on Tue, 26 Jan 2021 08:55:48 +0100 writes:
>>>>
>>>>        >> Dear all, my colleague posted our issue on stackoverflow:
>>>>
>>>>        >> Calling R script from Python does not save log file in
>>>>        >> version 4 - Stack Overflow
>>>>        >> [https://stackoverflow.com/questions/65887485/calling-r-script-from-python-does-not-save-log-file-in-version-4]
>>>>
>>>>        >> It is about this kind of call to R:
>>>>
>>>>        >> R.exe -f code.R --args "~/file.txt" 1> "~/log.txt" 2>&1".
>>>>
>>>>        >> The issue is that the log.txt file is not created when
>>>>        >> running R 4.x.x. The same code works perfectly fine with
>>>>        >> R 3.6.x.
>>>>
>>>>        >> Any idea what's going wrong as of version 4? Regards
>>>>        >> Marcel
>>>>
>>>>        > Dear Marcel, I think the solution is embarrassingly
>>>>        > simple:
>>>>
>>>>        >> From the SO post, where she showed a bit more detail than you
>>>>        > show here, it's clear you have confused 'R.exe' and
>>>>        > 'Rscript.exe' and what you say above is not true:
>>>>
>>>>        > 'R.exe' was used for R 3.6.0 but for R 4.0.3, you/she used
>>>>        > 'Rscript.exe' instead.
>>>>
>>>>
>>>>        > ... as you've noticed now, they do behave differently,
>>>>        > indeed!
>>>>
>>>> Well, this was not the solution to their -- Windows-only -- problem.
>>>> The problem *is* indeed visible if they only use  R.exe  (also
>>>> for R 4.0.3).
>>>>
>>>> I've commented more on the SO issue (see above),
>>>> notably asking for a *minimal* repr.ex. (reproducible example),
>>>> and one *not* using "<YOUR PATH>" and setwd() ..
>>>>
>>>
>>> Isn't this purely a Python or user problem?  R shouldn't process
>>> redirection directives like
>>>
>>>      1> "~/log.txt" 2>&1
>>>
>>> because it's the shell's job to process those. If Python is acting as
>>> the shell, it needs to handle those things.  If R was handling the
>>> command via
>>
>> Oops, sent before finishing:
>>
>> If R was handling the command via system() or system2(), it would handle
>> redirection itself. If it was using the Windows-only shell(), it would
>> call cmd.exe (by default) to handle redirection.  (This is a difference
>> between R on Windows and R in Unix:  in Unix a shell is always used.)
>>
>> Duncan Murdoch
>>
>> ______________________________________________
>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Error when calling (R 4.0.x on Windows) from Python

Bill Dunlap-2
I tried the following change, that adds quotes if the argument does
not include ">".
Index: front-ends/rcmdfn.c
===================================================================
--- front-ends/rcmdfn.c (revision 79883)
+++ front-ends/rcmdfn.c (working copy)
@@ -173,9 +173,13 @@
                fprintf(stderr, "command line too long\n");
                return(27);
            }
-           strcat(cmd, "\"");
+           if (!strchr(argv[i], '>')) {
+                strcat(cmd, "\"");
+            }
            strcat(cmd, argv[i]);
-           strcat(cmd, "\"");
+           if (!strchr(argv[i], '>')) {
+               strcat(cmd, "\"");
+            }
        }
        /* the outermost double quotes are needed for cmd.exe */
        strcat(cmd, "\"");

It lets the python example work.  I am not sure that quoting all the
arguments buys you much, as shQuote() is still needed for arguments
that include spaces.  E.g.,  with 3.6.3, 4.0.3, and my development
build with the above patch we get

> stopifnot(dir.create(dirname <- file.path(tempfile(), "A SPACE"), recursive=TRUE))
> logname <- file.path(dirname, "log.txt")
> unlink(logname)
> system(paste( "C:\\R\\R-3.6.3\\bin\\R.exe --quiet --vanilla -e \"commandArgs()\" 1>", logname))
ARGUMENT 'SPACE/log.txt' __ignored__

[1] 0
> tryCatch(readLines(logname), error=function(e)conditionMessage(e))
[1] "cannot open the connection"
Warning message:
In file(con, "r") :
  cannot open file
'C:\Users\willi\AppData\Local\Temp\RtmpM5tsC7\file1a1068734a49/A
SPACE/log.txt': No such file or directory
>
> system(paste( "C:\\R\\R-4.0.3\\bin\\R.exe --quiet --vanilla -e \"commandArgs()\" 1>", logname))
> commandArgs()
[1] "C:\\R\\R-40~1.3/bin/x64/Rterm.exe"
[2] "--quiet"
[3] "--vanilla"
[4] "-e"
[5] "commandArgs()"
[6] "1>"
[7] "C:\\Users\\willi\\AppData\\Local\\Temp\\RtmpM5tsC7\\file1a1068734a49/A"
[8] "SPACE/log.txt"
>
>
[1] 0
> tryCatch(readLines(logname), error=function(e)conditionMessage(e))
[1] "cannot open the connection"
Warning message:
In file(con, "r") :
  cannot open file
'C:\Users\willi\AppData\Local\Temp\RtmpM5tsC7\file1a1068734a49/A
SPACE/log.txt': No such file or directory
>
> unlink(logname)
> system(paste( "C:\\msys64\\home\\willi\\ucrt3\\r\\trunk\\bin\\R.exe --quiet --vanilla -e \"commandArgs()\" 1>", logname))
[1] 0
> tryCatch(readLines(logname), error=function(e)conditionMessage(e))
[1] "cannot open the connection"
Warning message:
In file(con, "r") :
  cannot open file
'C:\Users\willi\AppData\Local\Temp\RtmpM5tsC7\file1a1068734a49/A
SPACE/log.txt': No such file or directory
> tryCatch(readLines(sub(" .*$", "", logname)), error=function(e)conditionMessage(e))
[1] "> commandArgs()"
              "[1]
\"C:\\\\msys64\\\\home\\\\willi\\\\ucrt3\\\\r\\\\trunk/bin/x64/Rterm.exe\""
[3] "[2] \"--quiet\"
"             "[3] \"--vanilla\"
          "
[5] "[4] \"-e\"
"             "[5] \"commandArgs()\"
          "
[7] "[6] \"SPACE/log.txt\"
"             "> "
[9] "> "
>
> unlink(logname)
> system(paste( "C:\\msys64\\home\\willi\\ucrt3\\r\\trunk\\bin\\R.exe --quiet --vanilla -e \"commandArgs()\" 1>", shQuote(logname)))
[1] 0
> tryCatch(readLines(logname), error=function(e)conditionMessage(e))
[1] "> commandArgs()"
              "[1]
\"C:\\\\msys64\\\\home\\\\willi\\\\ucrt3\\\\r\\\\trunk/bin/x64/Rterm.exe\""
[3] "[2] \"--quiet\"
"             "[3] \"--vanilla\"
          "
[5] "[4] \"-e\"
"             "[5] \"commandArgs()\"
          "
[7] "> "
              "> "

-Bill

On Wed, Jan 27, 2021 at 1:25 PM Duncan Murdoch <[hidden email]> wrote:

>
> On 27/01/2021 3:40 p.m., Bill Dunlap wrote:
> > I believe the problem is from svn 77925 in gnuwin/front-ends/rcmdfn.c,
> > which was committed a few days after 3.6.3 was released.  Rterm used
> > to put double quotes around a command line argument only if it
> > contained a space, now it double quotes all arguments.  It sees shell
> > constructs like "1>" and the following file name as arguments and
> > double quoting them hides them from the shell, leading to this
> > problem.  I think we may have to rely on the user supplying quotes as
> > needed instead of blindly adding them.
>
> Okay, now I see what you mean.
>
> If you invoke R using R.exe, it asks cmd.exe to run Rterm.exe, so it is
> possible that redirection would be handled.
>
> If you invoke R directly using Rterm.exe, then my description down below
> would be correct.
>
> Duncan Murdoch
>
>
> >
> > -Bill
> >
> > On Wed, Jan 27, 2021 at 12:28 PM Duncan Murdoch
> > <[hidden email]> wrote:
> >>
> >> On 27/01/2021 3:17 p.m., Duncan Murdoch wrote:
> >>> On 27/01/2021 3:38 a.m., Martin Maechler wrote:
> >>>>>>>>> Martin Maechler
> >>>>>>>>>        on Tue, 26 Jan 2021 12:37:58 +0100 writes:
> >>>>
> >>>>>>>>> Marcel Baumgartner
> >>>>>>>>>        on Tue, 26 Jan 2021 08:55:48 +0100 writes:
> >>>>
> >>>>        >> Dear all, my colleague posted our issue on stackoverflow:
> >>>>
> >>>>        >> Calling R script from Python does not save log file in
> >>>>        >> version 4 - Stack Overflow
> >>>>        >> [https://stackoverflow.com/questions/65887485/calling-r-script-from-python-does-not-save-log-file-in-version-4]
> >>>>
> >>>>        >> It is about this kind of call to R:
> >>>>
> >>>>        >> R.exe -f code.R --args "~/file.txt" 1> "~/log.txt" 2>&1".
> >>>>
> >>>>        >> The issue is that the log.txt file is not created when
> >>>>        >> running R 4.x.x. The same code works perfectly fine with
> >>>>        >> R 3.6.x.
> >>>>
> >>>>        >> Any idea what's going wrong as of version 4? Regards
> >>>>        >> Marcel
> >>>>
> >>>>        > Dear Marcel, I think the solution is embarrassingly
> >>>>        > simple:
> >>>>
> >>>>        >> From the SO post, where she showed a bit more detail than you
> >>>>        > show here, it's clear you have confused 'R.exe' and
> >>>>        > 'Rscript.exe' and what you say above is not true:
> >>>>
> >>>>        > 'R.exe' was used for R 3.6.0 but for R 4.0.3, you/she used
> >>>>        > 'Rscript.exe' instead.
> >>>>
> >>>>
> >>>>        > ... as you've noticed now, they do behave differently,
> >>>>        > indeed!
> >>>>
> >>>> Well, this was not the solution to their -- Windows-only -- problem.
> >>>> The problem *is* indeed visible if they only use  R.exe  (also
> >>>> for R 4.0.3).
> >>>>
> >>>> I've commented more on the SO issue (see above),
> >>>> notably asking for a *minimal* repr.ex. (reproducible example),
> >>>> and one *not* using "<YOUR PATH>" and setwd() ..
> >>>>
> >>>
> >>> Isn't this purely a Python or user problem?  R shouldn't process
> >>> redirection directives like
> >>>
> >>>      1> "~/log.txt" 2>&1
> >>>
> >>> because it's the shell's job to process those. If Python is acting as
> >>> the shell, it needs to handle those things.  If R was handling the
> >>> command via
> >>
> >> Oops, sent before finishing:
> >>
> >> If R was handling the command via system() or system2(), it would handle
> >> redirection itself. If it was using the Windows-only shell(), it would
> >> call cmd.exe (by default) to handle redirection.  (This is a difference
> >> between R on Windows and R in Unix:  in Unix a shell is always used.)
> >>
> >> Duncan Murdoch
> >>
> >> ______________________________________________
> >> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> >> https://stat.ethz.ch/mailman/listinfo/r-help
> >> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> >> and provide commented, minimal, self-contained, reproducible code.
>

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Error when calling (R 4.0.x on Windows) from Python

Marcel Baumgartner
Dear Bill, Duncan and Martin,

thanks for your investigation. Can you clarify on next steps? Is this
now an official bug, or have you found a workaround? For your
information: the issue showed up the first time when I called R 4.0.2
from within a software called "IDEA" (from Caseware Analytics), using
their scripting language (similar to Visual Basic). With my colleague
we then simply reproduce the error calling R from Python, so that we
could share it more easily. When we run this command directly on the
CMD in Windows, all works fine. The issue only happens when R is
called within another software.

Best regards

Marcel

Le 2021-01-27T23:14:36.000+01:00, Bill Dunlap
<[hidden email]> a écrit :

> I tried the following change, that adds quotes if the argument does

> not include ">".

> Index: front-ends/rcmdfn.c

> ===================================================================

> --- front-ends/rcmdfn.c (revision 79883)

> +++ front-ends/rcmdfn.c (working copy)

> @@ -173,9 +173,13 @@

>                 fprintf(stderr, "command line too long\n");

>                 return(27);

>             }

> -           strcat(cmd, "\"");

> +           if (!strchr(argv[i], '>')) {

> +                strcat(cmd, "\"");

> +            }

>             strcat(cmd, argv[i]);

> -           strcat(cmd, "\"");

> +           if (!strchr(argv[i], '>')) {

> +               strcat(cmd, "\"");

> +            }

>         }

>         /* the outermost double quotes are needed for cmd.exe */

>         strcat(cmd, "\"");

> It lets the python example work.  I am not sure that quoting all the

> arguments buys you much, as shQuote() is still needed for arguments

> that include spaces.  E.g.,  with 3.6.3, 4.0.3, and my development

> build with the above patch we get

>>  stopifnot(dir.create(dirname <- file.path(tempfile(), "A SPACE"),
>>  recursive=TRUE))
>>  
>>   logname <- file.path(dirname, "log.txt")
>>  
>>   unlink(logname)
>>  
>>   system(paste( "C:\\R\\R-3.6.3\\bin\\R.exe --quiet --vanilla -e
>>  \"commandArgs()\" 1>", logname))

>  ARGUMENT 'SPACE/log.txt' __ignored__

> [1] 0

>>  tryCatch(readLines(logname), error=function(e)conditionMessage(e))

>  [1] "cannot open the connection"

> Warning message:

> In file(con, "r") :

>  cannot open file

> 'C:\Users\willi\AppData\Local\Temp\RtmpM5tsC7\file1a1068734a49/A

> SPACE/log.txt': No such file or directory

>>  system(paste( "C:\\R\\R-4.0.3\\bin\\R.exe --quiet --vanilla -e
>>  \"commandArgs()\" 1>", logname))
>>  
>>   commandArgs()

>  [1] "C:\\R\\R-40~1.3/bin/x64/Rterm.exe"

> [2] "--quiet"

> [3] "--vanilla"

> [4] "-e"

> [5] "commandArgs()"

> [6] "1>"

> [7]
> "C:\\Users\\willi\\AppData\\Local\\Temp\\RtmpM5tsC7\\file1a1068734a49/A"

> [8] "SPACE/log.txt"

>>  

>  [1] 0

>>  tryCatch(readLines(logname), error=function(e)conditionMessage(e))

>  [1] "cannot open the connection"

> Warning message:

> In file(con, "r") :

>  cannot open file

> 'C:\Users\willi\AppData\Local\Temp\RtmpM5tsC7\file1a1068734a49/A

> SPACE/log.txt': No such file or directory

>>  unlink(logname)
>>  
>>   system(paste(
>>  "C:\\msys64\\home\\willi\\ucrt3\\r\\trunk\\bin\\R.exe --quiet
>>  --vanilla -e \"commandArgs()\" 1>", logname))

>  [1] 0

>>  tryCatch(readLines(logname), error=function(e)conditionMessage(e))

>  [1] "cannot open the connection"

> Warning message:

> In file(con, "r") :

>  cannot open file

> 'C:\Users\willi\AppData\Local\Temp\RtmpM5tsC7\file1a1068734a49/A

> SPACE/log.txt': No such file or directory

>>  tryCatch(readLines(sub(" .*$", "", logname)),
>>  error=function(e)conditionMessage(e))

>  [1] "> commandArgs()"

>  "[1]

> \"C:\\\\msys64\\\\home\\\\willi\\\\ucrt3\\\\r\\\\trunk/bin/x64/Rterm.exe\""

> [3] "[2] \"--quiet\"

> " "[3] \"--vanilla\"

>  "

> [5] "[4] \"-e\"

> " "[5] \"commandArgs()\"

>  "

> [7] "[6] \"SPACE/log.txt\"

> " "> "

> [9] "> "

>>  unlink(logname)
>>  
>>   system(paste(
>>  "C:\\msys64\\home\\willi\\ucrt3\\r\\trunk\\bin\\R.exe --quiet
>>  --vanilla -e \"commandArgs()\" 1>", shQuote(logname)))

>  [1] 0

>>  tryCatch(readLines(logname), error=function(e)conditionMessage(e))

>  [1] "> commandArgs()"

>  "[1]

> \"C:\\\\msys64\\\\home\\\\willi\\\\ucrt3\\\\r\\\\trunk/bin/x64/Rterm.exe\""

> [3] "[2] \"--quiet\"

> " "[3] \"--vanilla\"

>  "

> [5] "[4] \"-e\"

> " "[5] \"commandArgs()\"

>  "

> [7] "> "

>  "> "

> -Bill

> On Wed, Jan 27, 2021 at 1:25 PM Duncan Murdoch
> <[hidden email]> wrote:

>>  On 27/01/2021 3:40 p.m., Bill Dunlap wrote:
>>  
>>>   I believe the problem is from svn 77925 in
>>>   gnuwin/front-ends/rcmdfn.c,
>>>   
>>>    which was committed a few days after 3.6.3 was released. Rterm
>>>   used
>>>   
>>>    to put double quotes around a command line argument only if it
>>>   
>>>    contained a space, now it double quotes all arguments. It sees
>>>   shell
>>>   
>>>    constructs like "1>" and the following file name as arguments
>>>   and
>>>   
>>>    double quoting them hides them from the shell, leading to this
>>>   
>>>    problem. I think we may have to rely on the user supplying
>>>   quotes as
>>>   
>>>    needed instead of blindly adding them.
>>  
>>   Okay, now I see what you mean.
>>  
>>   If you invoke R using R.exe, it asks cmd.exe to run Rterm.exe, so
>>  it is
>>  
>>   possible that redirection would be handled.
>>  
>>   If you invoke R directly using Rterm.exe, then my description
>>  down below
>>  
>>   would be correct.
>>  
>>   Duncan Murdoch
>>  
>>>   -Bill
>>>   
>>>    On Wed, Jan 27, 2021 at 12:28 PM Duncan Murdoch
>>>   
>>>    <[hidden email]> wrote:
>>>   
>>>>    On 27/01/2021 3:17 p.m., Duncan Murdoch wrote:
>>>>    
>>>>>     On 27/01/2021 3:38 a.m., Martin Maechler wrote:
>>>>>     
>>>>>>>>>>>           Martin Maechler
>>>>>>>>>>>           
>>>>>>>>>>>            on Tue, 26 Jan 2021 12:37:58 +0100 writes:
>>>>>>      
>>>>>>>>>>>           Marcel Baumgartner
>>>>>>>>>>>           
>>>>>>>>>>>            on Tue, 26 Jan 2021 08:55:48 +0100 writes:
>>>>>>      
>>>>>>       Dear all, my colleague posted our issue on stackoverflow:
>>>>>>      
>>>>>>       Calling R script from Python does not save log file in
>>>>>>      
>>>>>>       version 4 - Stack Overflow
>>>>>>      
>>>>>>       [stackoverflow.com/questions...
>>>>>>      [https://stackoverflow.com/questions/65887485/calling-r-script-from-python-does-not-save-log-file-in-version-4]]
>>>>>>      
>>>>>>       It is about this kind of call to R:
>>>>>>      
>>>>>>       R.exe -f code.R --args "~/file.txt" 1> "~/log.txt" 2>&1".
>>>>>>      
>>>>>>       The issue is that the log.txt file is not created when
>>>>>>      
>>>>>>       running R 4.x.x. The same code works perfectly fine with
>>>>>>      
>>>>>>       R 3.6.x.
>>>>>>      
>>>>>>       Any idea what's going wrong as of version 4? Regards
>>>>>>      
>>>>>>       Marcel
>>>>>>      
>>>>>>       Dear Marcel, I think the solution is embarrassingly
>>>>>>      
>>>>>>       simple:
>>>>>>      
>>>>>>       From the SO post, where she showed a bit more detail than
>>>>>>      you
>>>>>>      
>>>>>>       show here, it's clear you have confused 'R.exe' and
>>>>>>      
>>>>>>       'Rscript.exe' and what you say above is not true:
>>>>>>      
>>>>>>       'R.exe' was used for R 3.6.0 but for R 4.0.3, you/she
>>>>>>      used
>>>>>>      
>>>>>>       'Rscript.exe' instead.
>>>>>>      
>>>>>>       ... as you've noticed now, they do behave differently,
>>>>>>      
>>>>>>       indeed!
>>>>>>      
>>>>>>       Well, this was not the solution to their -- Windows-only
>>>>>>      -- problem.
>>>>>>      
>>>>>>       The problem *is* indeed visible if they only use R.exe
>>>>>>      (also
>>>>>>      
>>>>>>       for R 4.0.3).
>>>>>>      
>>>>>>       I've commented more on the SO issue (see above),
>>>>>>      
>>>>>>       notably asking for a *minimal* repr.ex. (reproducible
>>>>>>      example),
>>>>>>      
>>>>>>       and one *not* using "<YOUR PATH>" and setwd() ..
>>>>>     
>>>>>      Isn't this purely a Python or user problem? R shouldn't
>>>>>     process
>>>>>     
>>>>>      redirection directives like
>>>>>     
>>>>>      1> "~/log.txt" 2>&1
>>>>>     
>>>>>      because it's the shell's job to process those. If Python is
>>>>>     acting as
>>>>>     
>>>>>      the shell, it needs to handle those things. If R was
>>>>>     handling the
>>>>>     
>>>>>      command via
>>>>    
>>>>     Oops, sent before finishing:
>>>>    
>>>>     If R was handling the command via system() or system2(), it
>>>>    would handle
>>>>    
>>>>     redirection itself. If it was using the Windows-only shell(),
>>>>    it would
>>>>    
>>>>     call cmd.exe (by default) to handle redirection. (This is a
>>>>    difference
>>>>    
>>>>     between R on Windows and R in Unix: in Unix a shell is always
>>>>    used.)
>>>>    
>>>>     Duncan Murdoch
>>>>    
>>>>     ______________________________________________
>>>>    
>>>>     [hidden email] mailing list -- To UNSUBSCRIBE and more,
>>>>    see
>>>>    
>>>>     stat.ethz.ch/mailman/listin...
>>>>    [https://stat.ethz.ch/mailman/listinfo/r-help]
>>>>    
>>>>     PLEASE do read the posting guide
>>>>    www.R-project.org/posting-g...
>>>>    [http://www.R-project.org/posting-guide.html]
>>>>    
>>>>     and provide commented, minimal, self-contained, reproducible
>>>>    code.



        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Error when calling (R 4.0.x on Windows) from Python

Duncan Murdoch-2
On 29/01/2021 3:57 a.m., Marcel Baumgartner wrote:

> Dear Bill, Duncan and Martin,
>
> thanks for your investigation. Can you clarify on next steps? Is this
> now an official bug, or have you found a workaround? For your
> information: the issue showed up the first time when I called R 4.0.2
> from within a software called "IDEA" (from Caseware Analytics), using
> their scripting language (similar to Visual Basic). With my colleague we
> then simply reproduce the error calling R from Python, so that we could
> share it more easily. When we run this command directly on the CMD in
> Windows, all works fine. The issue only happens when R is called within
> another software.
>

I would say this is a bug, with two workarounds.  These are only needed
(and will only work) on Windows.

1.  If you want to include redirection in the command line, then call
cmd.exe yourself, and have it call Rterm.exe.  I think your original command

R.exe -f code.R --args "~/file.txt" 1> "~/log.txt" 2>&1"

could be written as

cmd.exe 1>log.txt 2>&1 /C R.exe -f code.R --args "~/file.txt"

(but I haven't tested it).  You could also use the slightly more efficient

cmd.exe 1>log.txt 2>&1 /C Rterm.exe -f code.R --args "~/file.txt"

if Rterm.exe is on your path; by default I don't think it will be.

2.  Change the Python code to a different function call than
subprocess.call.  It should be one that's equivalent to the C system()
function; I believe os.system() is what you want, so you'd use

os.system("R.exe -f code.R --args "~/file.txt" 1> "~/log.txt" 2>&1")

and the redirection would be handled by the implicit shell that is
called by os.system.  Again, replacing R.exe by Rterm.exe would be a
tiny bit more efficient, but it might not be on the path.

Duncan Murdoch

> Best regards
>
> Marcel
>
>
> Le 2021-01-27T23:14:36.000+01:00, Bill Dunlap <[hidden email]>
> a écrit :
>
>     I tried the following change, that adds quotes if the argument does
>     not include ">".
>     Index: front-ends/rcmdfn.c
>     ===================================================================
>     --- front-ends/rcmdfn.c (revision 79883)
>     +++ front-ends/rcmdfn.c (working copy)
>     @@ -173,9 +173,13 @@
>     fprintf(stderr, "command line too long\n");
>     return(27);
>     }
>     - strcat(cmd, "\"");
>     + if (!strchr(argv[i], '>')) {
>     + strcat(cmd, "\"");
>     + }
>     strcat(cmd, argv[i]);
>     - strcat(cmd, "\"");
>     + if (!strchr(argv[i], '>')) {
>     + strcat(cmd, "\"");
>     + }
>     }
>     /* the outermost double quotes are needed for cmd.exe */
>     strcat(cmd, "\"");
>
>     It lets the python example work. I am not sure that quoting all the
>     arguments buys you much, as shQuote() is still needed for arguments
>     that include spaces. E.g., with 3.6.3, 4.0.3, and my development
>     build with the above patch we get
>
>         stopifnot(dir.create(dirname <- file.path(tempfile(), "A
>         SPACE"), recursive=TRUE))
>         logname <- file.path(dirname, "log.txt")
>         unlink(logname)
>         system(paste( "C:\\R\\R-3.6.3\\bin\\R.exe --quiet --vanilla -e
>         \"commandArgs()\" 1>", logname))
>
>     ARGUMENT 'SPACE/log.txt' __ignored__
>
>     [1] 0
>
>         tryCatch(readLines(logname), error=function(e)conditionMessage(e))
>
>     [1] "cannot open the connection"
>     Warning message:
>     In file(con, "r") :
>     cannot open file
>     'C:\Users\willi\AppData\Local\Temp\RtmpM5tsC7\file1a1068734a49/A
>     SPACE/log.txt': No such file or directory
>
>
>         system(paste( "C:\\R\\R-4.0.3\\bin\\R.exe --quiet --vanilla -e
>         \"commandArgs()\" 1>", logname))
>         commandArgs()
>
>     [1] "C:\\R\\R-40~1.3/bin/x64/Rterm.exe"
>     [2] "--quiet"
>     [3] "--vanilla"
>     [4] "-e"
>     [5] "commandArgs()"
>     [6] "1>"
>     [7]
>     "C:\\Users\\willi\\AppData\\Local\\Temp\\RtmpM5tsC7\\file1a1068734a49/A"
>     [8] "SPACE/log.txt"
>
>
>     [1] 0
>
>         tryCatch(readLines(logname), error=function(e)conditionMessage(e))
>
>     [1] "cannot open the connection"
>     Warning message:
>     In file(con, "r") :
>     cannot open file
>     'C:\Users\willi\AppData\Local\Temp\RtmpM5tsC7\file1a1068734a49/A
>     SPACE/log.txt': No such file or directory
>
>
>         unlink(logname)
>         system(paste(
>         "C:\\msys64\\home\\willi\\ucrt3\\r\\trunk\\bin\\R.exe --quiet
>         --vanilla -e \"commandArgs()\" 1>", logname))
>
>     [1] 0
>
>         tryCatch(readLines(logname), error=function(e)conditionMessage(e))
>
>     [1] "cannot open the connection"
>     Warning message:
>     In file(con, "r") :
>     cannot open file
>     'C:\Users\willi\AppData\Local\Temp\RtmpM5tsC7\file1a1068734a49/A
>     SPACE/log.txt': No such file or directory
>
>         tryCatch(readLines(sub(" .*$", "", logname)),
>         error=function(e)conditionMessage(e))
>
>     [1] "> commandArgs()"
>     "[1]
>     \"C:\\\\msys64\\\\home\\\\willi\\\\ucrt3\\\\r\\\\trunk/bin/x64/Rterm.exe\""
>     [3] "[2] \"--quiet\"
>     " "[3] \"--vanilla\"
>     "
>     [5] "[4] \"-e\"
>     " "[5] \"commandArgs()\"
>     "
>     [7] "[6] \"SPACE/log.txt\"
>     " "> "
>     [9] "> "
>
>
>         unlink(logname)
>         system(paste(
>         "C:\\msys64\\home\\willi\\ucrt3\\r\\trunk\\bin\\R.exe --quiet
>         --vanilla -e \"commandArgs()\" 1>", shQuote(logname)))
>
>     [1] 0
>
>         tryCatch(readLines(logname), error=function(e)conditionMessage(e))
>
>     [1] "> commandArgs()"
>     "[1]
>     \"C:\\\\msys64\\\\home\\\\willi\\\\ucrt3\\\\r\\\\trunk/bin/x64/Rterm.exe\""
>     [3] "[2] \"--quiet\"
>     " "[3] \"--vanilla\"
>     "
>     [5] "[4] \"-e\"
>     " "[5] \"commandArgs()\"
>     "
>     [7] "> "
>     "> "
>
>     -Bill
>
>     On Wed, Jan 27, 2021 at 1:25 PM Duncan Murdoch
>     <[hidden email] <mailto:[hidden email]>> wrote:
>
>
>         On 27/01/2021 3:40 p.m., Bill Dunlap wrote:
>
>             I believe the problem is from svn 77925 in
>             gnuwin/front-ends/rcmdfn.c,
>             which was committed a few days after 3.6.3 was released.
>             Rterm used
>             to put double quotes around a command line argument only if it
>             contained a space, now it double quotes all arguments. It
>             sees shell
>             constructs like "1>" and the following file name as
>             arguments and
>             double quoting them hides them from the shell, leading to this
>             problem. I think we may have to rely on the user supplying
>             quotes as
>             needed instead of blindly adding them.
>
>
>         Okay, now I see what you mean.
>
>         If you invoke R using R.exe, it asks cmd.exe to run Rterm.exe,
>         so it is
>         possible that redirection would be handled.
>
>         If you invoke R directly using Rterm.exe, then my description
>         down below
>         would be correct.
>
>         Duncan Murdoch
>
>
>
>             -Bill
>
>             On Wed, Jan 27, 2021 at 12:28 PM Duncan Murdoch
>             <[hidden email] <mailto:[hidden email]>>
>             wrote:
>
>
>                 On 27/01/2021 3:17 p.m., Duncan Murdoch wrote:
>
>                     On 27/01/2021 3:38 a.m., Martin Maechler wrote:
>
>                                             Martin Maechler
>                                             on Tue, 26 Jan 2021 12:37:58
>                                             +0100 writes:
>
>
>                                             Marcel Baumgartner
>                                             on Tue, 26 Jan 2021 08:55:48
>                                             +0100 writes:
>
>
>                         Dear all, my colleague posted our issue on
>                         stackoverflow:
>
>                         Calling R script from Python does not save log
>                         file in
>                         version 4 - Stack Overflow
>                         [stackoverflow.com/questions...
>                         <https://stackoverflow.com/questions/65887485/calling-r-script-from-python-does-not-save-log-file-in-version-4>]
>
>                         It is about this kind of call to R:
>
>                         R.exe -f code.R --args "~/file.txt" 1>
>                         "~/log.txt" 2>&1".
>
>                         The issue is that the log.txt file is not
>                         created when
>                         running R 4.x.x. The same code works perfectly
>                         fine with
>                         R 3.6.x.
>
>                         Any idea what's going wrong as of version 4? Regards
>                         Marcel
>
>                         Dear Marcel, I think the solution is embarrassingly
>                         simple:
>
>                          From the SO post, where she showed a bit more
>                         detail than you
>                         show here, it's clear you have confused 'R.exe' and
>                         'Rscript.exe' and what you say above is not true:
>
>                         'R.exe' was used for R 3.6.0 but for R 4.0.3,
>                         you/she used
>                         'Rscript.exe' instead.
>
>
>                         ... as you've noticed now, they do behave
>                         differently,
>                         indeed!
>
>                         Well, this was not the solution to their --
>                         Windows-only -- problem.
>                         The problem *is* indeed visible if they only use
>                         R.exe (also
>                         for R 4.0.3).
>
>                         I've commented more on the SO issue (see above),
>                         notably asking for a *minimal* repr.ex.
>                         (reproducible example),
>                         and one *not* using "<YOUR PATH>" and setwd() ..
>
>
>                     Isn't this purely a Python or user problem? R
>                     shouldn't process
>                     redirection directives like
>
>                     1> "~/log.txt" 2>&1
>
>                     because it's the shell's job to process those. If
>                     Python is acting as
>                     the shell, it needs to handle those things. If R was
>                     handling the
>                     command via
>
>
>                 Oops, sent before finishing:
>
>                 If R was handling the command via system() or system2(),
>                 it would handle
>                 redirection itself. If it was using the Windows-only
>                 shell(), it would
>                 call cmd.exe (by default) to handle redirection. (This
>                 is a difference
>                 between R on Windows and R in Unix: in Unix a shell is
>                 always used.)
>
>                 Duncan Murdoch
>
>                 ______________________________________________
>                 [hidden email] <mailto:[hidden email]>
>                 mailing list -- To UNSUBSCRIBE and more, see
>                 stat.ethz.ch/mailman/listin...
>                 <https://stat.ethz.ch/mailman/listinfo/r-help>
>                 PLEASE do read the posting guide
>                 www.R-project.org/posting-g...
>                 <http://www.R-project.org/posting-guide.html>
>                 and provide commented, minimal, self-contained,
>                 reproducible code.
>
>
>

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Error when calling (R 4.0.x on Windows) from Python

Tomas Kalibera
A reproducible example:

system("\"C:\\Program Files\\R\\R-4.0.3\\bin\\R.exe\" -e commandArgs()
 >out")  # does not create "out"
system("\"C:\\Program Files\\R\\R-3.6.3\\bin\\R.exe\" -e commandArgs()
 >out")  # creates "out"

Note that this does not create "out", either:

system("\"C:\\Program Files\\R\\R-3.6.3\\bin\\x64\\Rterm.exe\" -e
commandArgs() >out")

I think redirections should be left to the shell, so e.g. those calls
from Python should use "os.system" if they needed redirection, as Duncan
suggested.

I see that both R.exe and Rterm.exe give "Usage: Rterm [options] [<
infile] [> outfile] [EnvVars]", which may be confusing to programmers
invoking R without a shell, but then talking there about invocation via
shell could confuse typical users, instead.

My best guess is that the redirection in R.exe in 3.6 worked rather by
accident, as a consequence of that R.exe internally invoked (and still
invokes) Rterm.exe via the shell, but R.exe did not protect all
arguments from expansions from that internal shell invocation. If
redirection was meant to work without external shell, it would have
instead been implemented explicitly also in Rterm.exe.

My change 77926 made sure that all arguments to that internal shell
invocation were protected, following bug reports such as PR#17709, where
users had "&" in their file names. Therefore, this works, executing file
code&.r

system("\"C:\\Program Files\\R\\R-4.0.3\\bin\\R.exe\" -f code&.r")

but not with 3.6.3, and I would not be surprised if even more special
characters became popular, soon.

After all, what if someone wanted to pass an argument to R including
">", that should work, too:

---- t.r ----
cat("Header: ", commandArgs(TRUE)[1], "\n")
-------------

system("\"C:\\Program Files\\R\\R-4.0.3\\bin\\R.exe\" -f t.r --args
<html>") # prints Header:  <html>
system("\"C:\\Program Files\\R\\R-3.6.3\\bin\\R.exe\" -f t.r --args
<html>") # fails with  The syntax of the command is incorrect.

So in summary, I don't agree this is a bug.

Best
Tomas

On 1/29/21 11:19 AM, Duncan Murdoch wrote:

> On 29/01/2021 3:57 a.m., Marcel Baumgartner wrote:
>> Dear Bill, Duncan and Martin,
>>
>> thanks for your investigation. Can you clarify on next steps? Is this
>> now an official bug, or have you found a workaround? For your
>> information: the issue showed up the first time when I called R 4.0.2
>> from within a software called "IDEA" (from Caseware Analytics), using
>> their scripting language (similar to Visual Basic). With my colleague
>> we then simply reproduce the error calling R from Python, so that we
>> could share it more easily. When we run this command directly on the
>> CMD in Windows, all works fine. The issue only happens when R is
>> called within another software.
>>
>
> I would say this is a bug, with two workarounds.  These are only
> needed (and will only work) on Windows.
>
> 1.  If you want to include redirection in the command line, then call
> cmd.exe yourself, and have it call Rterm.exe.  I think your original
> command
>
> R.exe -f code.R --args "~/file.txt" 1> "~/log.txt" 2>&1"
>
> could be written as
>
> cmd.exe 1>log.txt 2>&1 /C R.exe -f code.R --args "~/file.txt"
>
> (but I haven't tested it).  You could also use the slightly more
> efficient
>
> cmd.exe 1>log.txt 2>&1 /C Rterm.exe -f code.R --args "~/file.txt"
>
> if Rterm.exe is on your path; by default I don't think it will be.
>
> 2.  Change the Python code to a different function call than
> subprocess.call.  It should be one that's equivalent to the C system()
> function; I believe os.system() is what you want, so you'd use
>
> os.system("R.exe -f code.R --args "~/file.txt" 1> "~/log.txt" 2>&1")
>
> and the redirection would be handled by the implicit shell that is
> called by os.system.  Again, replacing R.exe by Rterm.exe would be a
> tiny bit more efficient, but it might not be on the path.
>
> Duncan Murdoch
>
>> Best regards
>>
>> Marcel
>>
>>
>> Le 2021-01-27T23:14:36.000+01:00, Bill Dunlap
>> <[hidden email]> a écrit :
>>
>>     I tried the following change, that adds quotes if the argument does
>>     not include ">".
>>     Index: front-ends/rcmdfn.c
>> ===================================================================
>>     --- front-ends/rcmdfn.c (revision 79883)
>>     +++ front-ends/rcmdfn.c (working copy)
>>     @@ -173,9 +173,13 @@
>>     fprintf(stderr, "command line too long\n");
>>     return(27);
>>     }
>>     - strcat(cmd, "\"");
>>     + if (!strchr(argv[i], '>')) {
>>     + strcat(cmd, "\"");
>>     + }
>>     strcat(cmd, argv[i]);
>>     - strcat(cmd, "\"");
>>     + if (!strchr(argv[i], '>')) {
>>     + strcat(cmd, "\"");
>>     + }
>>     }
>>     /* the outermost double quotes are needed for cmd.exe */
>>     strcat(cmd, "\"");
>>
>>     It lets the python example work. I am not sure that quoting all the
>>     arguments buys you much, as shQuote() is still needed for arguments
>>     that include spaces. E.g., with 3.6.3, 4.0.3, and my development
>>     build with the above patch we get
>>
>>         stopifnot(dir.create(dirname <- file.path(tempfile(), "A
>>         SPACE"), recursive=TRUE))
>>         logname <- file.path(dirname, "log.txt")
>>         unlink(logname)
>>         system(paste( "C:\\R\\R-3.6.3\\bin\\R.exe --quiet --vanilla -e
>>         \"commandArgs()\" 1>", logname))
>>
>>     ARGUMENT 'SPACE/log.txt' __ignored__
>>
>>     [1] 0
>>
>>         tryCatch(readLines(logname),
>> error=function(e)conditionMessage(e))
>>
>>     [1] "cannot open the connection"
>>     Warning message:
>>     In file(con, "r") :
>>     cannot open file
>> 'C:\Users\willi\AppData\Local\Temp\RtmpM5tsC7\file1a1068734a49/A
>>     SPACE/log.txt': No such file or directory
>>
>>
>>         system(paste( "C:\\R\\R-4.0.3\\bin\\R.exe --quiet --vanilla -e
>>         \"commandArgs()\" 1>", logname))
>>         commandArgs()
>>
>>     [1] "C:\\R\\R-40~1.3/bin/x64/Rterm.exe"
>>     [2] "--quiet"
>>     [3] "--vanilla"
>>     [4] "-e"
>>     [5] "commandArgs()"
>>     [6] "1>"
>>     [7]
>> "C:\\Users\\willi\\AppData\\Local\\Temp\\RtmpM5tsC7\\file1a1068734a49/A"
>>     [8] "SPACE/log.txt"
>>
>>
>>     [1] 0
>>
>>         tryCatch(readLines(logname),
>> error=function(e)conditionMessage(e))
>>
>>     [1] "cannot open the connection"
>>     Warning message:
>>     In file(con, "r") :
>>     cannot open file
>> 'C:\Users\willi\AppData\Local\Temp\RtmpM5tsC7\file1a1068734a49/A
>>     SPACE/log.txt': No such file or directory
>>
>>
>>         unlink(logname)
>>         system(paste(
>>         "C:\\msys64\\home\\willi\\ucrt3\\r\\trunk\\bin\\R.exe --quiet
>>         --vanilla -e \"commandArgs()\" 1>", logname))
>>
>>     [1] 0
>>
>>         tryCatch(readLines(logname),
>> error=function(e)conditionMessage(e))
>>
>>     [1] "cannot open the connection"
>>     Warning message:
>>     In file(con, "r") :
>>     cannot open file
>> 'C:\Users\willi\AppData\Local\Temp\RtmpM5tsC7\file1a1068734a49/A
>>     SPACE/log.txt': No such file or directory
>>
>>         tryCatch(readLines(sub(" .*$", "", logname)),
>>         error=function(e)conditionMessage(e))
>>
>>     [1] "> commandArgs()"
>>     "[1]
>> \"C:\\\\msys64\\\\home\\\\willi\\\\ucrt3\\\\r\\\\trunk/bin/x64/Rterm.exe\""
>>     [3] "[2] \"--quiet\"
>>     " "[3] \"--vanilla\"
>>     "
>>     [5] "[4] \"-e\"
>>     " "[5] \"commandArgs()\"
>>     "
>>     [7] "[6] \"SPACE/log.txt\"
>>     " "> "
>>     [9] "> "
>>
>>
>>         unlink(logname)
>>         system(paste(
>>         "C:\\msys64\\home\\willi\\ucrt3\\r\\trunk\\bin\\R.exe --quiet
>>         --vanilla -e \"commandArgs()\" 1>", shQuote(logname)))
>>
>>     [1] 0
>>
>>         tryCatch(readLines(logname),
>> error=function(e)conditionMessage(e))
>>
>>     [1] "> commandArgs()"
>>     "[1]
>> \"C:\\\\msys64\\\\home\\\\willi\\\\ucrt3\\\\r\\\\trunk/bin/x64/Rterm.exe\""
>>     [3] "[2] \"--quiet\"
>>     " "[3] \"--vanilla\"
>>     "
>>     [5] "[4] \"-e\"
>>     " "[5] \"commandArgs()\"
>>     "
>>     [7] "> "
>>     "> "
>>
>>     -Bill
>>
>>     On Wed, Jan 27, 2021 at 1:25 PM Duncan Murdoch
>>     <[hidden email] <mailto:[hidden email]>> wrote:
>>
>>
>>         On 27/01/2021 3:40 p.m., Bill Dunlap wrote:
>>
>>             I believe the problem is from svn 77925 in
>>             gnuwin/front-ends/rcmdfn.c,
>>             which was committed a few days after 3.6.3 was released.
>>             Rterm used
>>             to put double quotes around a command line argument only
>> if it
>>             contained a space, now it double quotes all arguments. It
>>             sees shell
>>             constructs like "1>" and the following file name as
>>             arguments and
>>             double quoting them hides them from the shell, leading to
>> this
>>             problem. I think we may have to rely on the user supplying
>>             quotes as
>>             needed instead of blindly adding them.
>>
>>
>>         Okay, now I see what you mean.
>>
>>         If you invoke R using R.exe, it asks cmd.exe to run Rterm.exe,
>>         so it is
>>         possible that redirection would be handled.
>>
>>         If you invoke R directly using Rterm.exe, then my description
>>         down below
>>         would be correct.
>>
>>         Duncan Murdoch
>>
>>
>>
>>             -Bill
>>
>>             On Wed, Jan 27, 2021 at 12:28 PM Duncan Murdoch
>>             <[hidden email] <mailto:[hidden email]>>
>>             wrote:
>>
>>
>>                 On 27/01/2021 3:17 p.m., Duncan Murdoch wrote:
>>
>>                     On 27/01/2021 3:38 a.m., Martin Maechler wrote:
>>
>>                                             Martin Maechler
>>                                             on Tue, 26 Jan 2021 12:37:58
>>                                             +0100 writes:
>>
>>
>>                                             Marcel Baumgartner
>>                                             on Tue, 26 Jan 2021 08:55:48
>>                                             +0100 writes:
>>
>>
>>                         Dear all, my colleague posted our issue on
>>                         stackoverflow:
>>
>>                         Calling R script from Python does not save log
>>                         file in
>>                         version 4 - Stack Overflow
>>                         [stackoverflow.com/questions...
>> <https://stackoverflow.com/questions/65887485/calling-r-script-from-python-does-not-save-log-file-in-version-4>]
>>
>>                         It is about this kind of call to R:
>>
>>                         R.exe -f code.R --args "~/file.txt" 1>
>>                         "~/log.txt" 2>&1".
>>
>>                         The issue is that the log.txt file is not
>>                         created when
>>                         running R 4.x.x. The same code works perfectly
>>                         fine with
>>                         R 3.6.x.
>>
>>                         Any idea what's going wrong as of version 4?
>> Regards
>>                         Marcel
>>
>>                         Dear Marcel, I think the solution is
>> embarrassingly
>>                         simple:
>>
>>                          From the SO post, where she showed a bit more
>>                         detail than you
>>                         show here, it's clear you have confused
>> 'R.exe' and
>>                         'Rscript.exe' and what you say above is not
>> true:
>>
>>                         'R.exe' was used for R 3.6.0 but for R 4.0.3,
>>                         you/she used
>>                         'Rscript.exe' instead.
>>
>>
>>                         ... as you've noticed now, they do behave
>>                         differently,
>>                         indeed!
>>
>>                         Well, this was not the solution to their --
>>                         Windows-only -- problem.
>>                         The problem *is* indeed visible if they only use
>>                         R.exe (also
>>                         for R 4.0.3).
>>
>>                         I've commented more on the SO issue (see above),
>>                         notably asking for a *minimal* repr.ex.
>>                         (reproducible example),
>>                         and one *not* using "<YOUR PATH>" and setwd() ..
>>
>>
>>                     Isn't this purely a Python or user problem? R
>>                     shouldn't process
>>                     redirection directives like
>>
>>                     1> "~/log.txt" 2>&1
>>
>>                     because it's the shell's job to process those. If
>>                     Python is acting as
>>                     the shell, it needs to handle those things. If R was
>>                     handling the
>>                     command via
>>
>>
>>                 Oops, sent before finishing:
>>
>>                 If R was handling the command via system() or system2(),
>>                 it would handle
>>                 redirection itself. If it was using the Windows-only
>>                 shell(), it would
>>                 call cmd.exe (by default) to handle redirection. (This
>>                 is a difference
>>                 between R on Windows and R in Unix: in Unix a shell is
>>                 always used.)
>>
>>                 Duncan Murdoch
>>
>>                 ______________________________________________
>>                 [hidden email] <mailto:[hidden email]>
>>                 mailing list -- To UNSUBSCRIBE and more, see
>>                 stat.ethz.ch/mailman/listin...
>> <https://stat.ethz.ch/mailman/listinfo/r-help>
>>                 PLEASE do read the posting guide
>>                 www.R-project.org/posting-g...
>> <http://www.R-project.org/posting-guide.html>
>>                 and provide commented, minimal, self-contained,
>>                 reproducible code.
>>
>>
>>
>
> ______________________________________________
> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Error when calling (R 4.0.x on Windows) from Python

Bill Dunlap-2
R/Rterm now has shell-independent ways to specify its standard input
file, --file=file.R, and environment variables, VAR=VALUE.  Should it
also have shell-independent arguments to specify files to contain
stdout and stderr or both?  Then its help message could omit the '>
output' (it can already leave out the '< infile').

The help message could then say that if those arguments are omitted
then Rterm uses stdin, stdout, and stderr.  Thus the user could come
up with the shell syntax if the user needed to use pipes or output
stream merging, etc.

-Bill

On Tue, Feb 2, 2021 at 5:49 AM Tomas Kalibera <[hidden email]> wrote:

>
> A reproducible example:
>
> system("\"C:\\Program Files\\R\\R-4.0.3\\bin\\R.exe\" -e commandArgs()
>  >out")  # does not create "out"
> system("\"C:\\Program Files\\R\\R-3.6.3\\bin\\R.exe\" -e commandArgs()
>  >out")  # creates "out"
>
> Note that this does not create "out", either:
>
> system("\"C:\\Program Files\\R\\R-3.6.3\\bin\\x64\\Rterm.exe\" -e
> commandArgs() >out")
>
> I think redirections should be left to the shell, so e.g. those calls
> from Python should use "os.system" if they needed redirection, as Duncan
> suggested.
>
> I see that both R.exe and Rterm.exe give "Usage: Rterm [options] [<
> infile] [> outfile] [EnvVars]", which may be confusing to programmers
> invoking R without a shell, but then talking there about invocation via
> shell could confuse typical users, instead.
>
> My best guess is that the redirection in R.exe in 3.6 worked rather by
> accident, as a consequence of that R.exe internally invoked (and still
> invokes) Rterm.exe via the shell, but R.exe did not protect all
> arguments from expansions from that internal shell invocation. If
> redirection was meant to work without external shell, it would have
> instead been implemented explicitly also in Rterm.exe.
>
> My change 77926 made sure that all arguments to that internal shell
> invocation were protected, following bug reports such as PR#17709, where
> users had "&" in their file names. Therefore, this works, executing file
> code&.r
>
> system("\"C:\\Program Files\\R\\R-4.0.3\\bin\\R.exe\" -f code&.r")
>
> but not with 3.6.3, and I would not be surprised if even more special
> characters became popular, soon.
>
> After all, what if someone wanted to pass an argument to R including
> ">", that should work, too:
>
> ---- t.r ----
> cat("Header: ", commandArgs(TRUE)[1], "\n")
> -------------
>
> system("\"C:\\Program Files\\R\\R-4.0.3\\bin\\R.exe\" -f t.r --args
> <html>") # prints Header:  <html>
> system("\"C:\\Program Files\\R\\R-3.6.3\\bin\\R.exe\" -f t.r --args
> <html>") # fails with  The syntax of the command is incorrect.
>
> So in summary, I don't agree this is a bug.
>
> Best
> Tomas
>
> On 1/29/21 11:19 AM, Duncan Murdoch wrote:
> > On 29/01/2021 3:57 a.m., Marcel Baumgartner wrote:
> >> Dear Bill, Duncan and Martin,
> >>
> >> thanks for your investigation. Can you clarify on next steps? Is this
> >> now an official bug, or have you found a workaround? For your
> >> information: the issue showed up the first time when I called R 4.0.2
> >> from within a software called "IDEA" (from Caseware Analytics), using
> >> their scripting language (similar to Visual Basic). With my colleague
> >> we then simply reproduce the error calling R from Python, so that we
> >> could share it more easily. When we run this command directly on the
> >> CMD in Windows, all works fine. The issue only happens when R is
> >> called within another software.
> >>
> >
> > I would say this is a bug, with two workarounds.  These are only
> > needed (and will only work) on Windows.
> >
> > 1.  If you want to include redirection in the command line, then call
> > cmd.exe yourself, and have it call Rterm.exe.  I think your original
> > command
> >
> > R.exe -f code.R --args "~/file.txt" 1> "~/log.txt" 2>&1"
> >
> > could be written as
> >
> > cmd.exe 1>log.txt 2>&1 /C R.exe -f code.R --args "~/file.txt"
> >
> > (but I haven't tested it).  You could also use the slightly more
> > efficient
> >
> > cmd.exe 1>log.txt 2>&1 /C Rterm.exe -f code.R --args "~/file.txt"
> >
> > if Rterm.exe is on your path; by default I don't think it will be.
> >
> > 2.  Change the Python code to a different function call than
> > subprocess.call.  It should be one that's equivalent to the C system()
> > function; I believe os.system() is what you want, so you'd use
> >
> > os.system("R.exe -f code.R --args "~/file.txt" 1> "~/log.txt" 2>&1")
> >
> > and the redirection would be handled by the implicit shell that is
> > called by os.system.  Again, replacing R.exe by Rterm.exe would be a
> > tiny bit more efficient, but it might not be on the path.
> >
> > Duncan Murdoch
> >
> >> Best regards
> >>
> >> Marcel
> >>
> >>
> >> Le 2021-01-27T23:14:36.000+01:00, Bill Dunlap
> >> <[hidden email]> a écrit :
> >>
> >>     I tried the following change, that adds quotes if the argument does
> >>     not include ">".
> >>     Index: front-ends/rcmdfn.c
> >> ===================================================================
> >>     --- front-ends/rcmdfn.c (revision 79883)
> >>     +++ front-ends/rcmdfn.c (working copy)
> >>     @@ -173,9 +173,13 @@
> >>     fprintf(stderr, "command line too long\n");
> >>     return(27);
> >>     }
> >>     - strcat(cmd, "\"");
> >>     + if (!strchr(argv[i], '>')) {
> >>     + strcat(cmd, "\"");
> >>     + }
> >>     strcat(cmd, argv[i]);
> >>     - strcat(cmd, "\"");
> >>     + if (!strchr(argv[i], '>')) {
> >>     + strcat(cmd, "\"");
> >>     + }
> >>     }
> >>     /* the outermost double quotes are needed for cmd.exe */
> >>     strcat(cmd, "\"");
> >>
> >>     It lets the python example work. I am not sure that quoting all the
> >>     arguments buys you much, as shQuote() is still needed for arguments
> >>     that include spaces. E.g., with 3.6.3, 4.0.3, and my development
> >>     build with the above patch we get
> >>
> >>         stopifnot(dir.create(dirname <- file.path(tempfile(), "A
> >>         SPACE"), recursive=TRUE))
> >>         logname <- file.path(dirname, "log.txt")
> >>         unlink(logname)
> >>         system(paste( "C:\\R\\R-3.6.3\\bin\\R.exe --quiet --vanilla -e
> >>         \"commandArgs()\" 1>", logname))
> >>
> >>     ARGUMENT 'SPACE/log.txt' __ignored__
> >>
> >>     [1] 0
> >>
> >>         tryCatch(readLines(logname),
> >> error=function(e)conditionMessage(e))
> >>
> >>     [1] "cannot open the connection"
> >>     Warning message:
> >>     In file(con, "r") :
> >>     cannot open file
> >> 'C:\Users\willi\AppData\Local\Temp\RtmpM5tsC7\file1a1068734a49/A
> >>     SPACE/log.txt': No such file or directory
> >>
> >>
> >>         system(paste( "C:\\R\\R-4.0.3\\bin\\R.exe --quiet --vanilla -e
> >>         \"commandArgs()\" 1>", logname))
> >>         commandArgs()
> >>
> >>     [1] "C:\\R\\R-40~1.3/bin/x64/Rterm.exe"
> >>     [2] "--quiet"
> >>     [3] "--vanilla"
> >>     [4] "-e"
> >>     [5] "commandArgs()"
> >>     [6] "1>"
> >>     [7]
> >> "C:\\Users\\willi\\AppData\\Local\\Temp\\RtmpM5tsC7\\file1a1068734a49/A"
> >>     [8] "SPACE/log.txt"
> >>
> >>
> >>     [1] 0
> >>
> >>         tryCatch(readLines(logname),
> >> error=function(e)conditionMessage(e))
> >>
> >>     [1] "cannot open the connection"
> >>     Warning message:
> >>     In file(con, "r") :
> >>     cannot open file
> >> 'C:\Users\willi\AppData\Local\Temp\RtmpM5tsC7\file1a1068734a49/A
> >>     SPACE/log.txt': No such file or directory
> >>
> >>
> >>         unlink(logname)
> >>         system(paste(
> >>         "C:\\msys64\\home\\willi\\ucrt3\\r\\trunk\\bin\\R.exe --quiet
> >>         --vanilla -e \"commandArgs()\" 1>", logname))
> >>
> >>     [1] 0
> >>
> >>         tryCatch(readLines(logname),
> >> error=function(e)conditionMessage(e))
> >>
> >>     [1] "cannot open the connection"
> >>     Warning message:
> >>     In file(con, "r") :
> >>     cannot open file
> >> 'C:\Users\willi\AppData\Local\Temp\RtmpM5tsC7\file1a1068734a49/A
> >>     SPACE/log.txt': No such file or directory
> >>
> >>         tryCatch(readLines(sub(" .*$", "", logname)),
> >>         error=function(e)conditionMessage(e))
> >>
> >>     [1] "> commandArgs()"
> >>     "[1]
> >> \"C:\\\\msys64\\\\home\\\\willi\\\\ucrt3\\\\r\\\\trunk/bin/x64/Rterm.exe\""
> >>     [3] "[2] \"--quiet\"
> >>     " "[3] \"--vanilla\"
> >>     "
> >>     [5] "[4] \"-e\"
> >>     " "[5] \"commandArgs()\"
> >>     "
> >>     [7] "[6] \"SPACE/log.txt\"
> >>     " "> "
> >>     [9] "> "
> >>
> >>
> >>         unlink(logname)
> >>         system(paste(
> >>         "C:\\msys64\\home\\willi\\ucrt3\\r\\trunk\\bin\\R.exe --quiet
> >>         --vanilla -e \"commandArgs()\" 1>", shQuote(logname)))
> >>
> >>     [1] 0
> >>
> >>         tryCatch(readLines(logname),
> >> error=function(e)conditionMessage(e))
> >>
> >>     [1] "> commandArgs()"
> >>     "[1]
> >> \"C:\\\\msys64\\\\home\\\\willi\\\\ucrt3\\\\r\\\\trunk/bin/x64/Rterm.exe\""
> >>     [3] "[2] \"--quiet\"
> >>     " "[3] \"--vanilla\"
> >>     "
> >>     [5] "[4] \"-e\"
> >>     " "[5] \"commandArgs()\"
> >>     "
> >>     [7] "> "
> >>     "> "
> >>
> >>     -Bill
> >>
> >>     On Wed, Jan 27, 2021 at 1:25 PM Duncan Murdoch
> >>     <[hidden email] <mailto:[hidden email]>> wrote:
> >>
> >>
> >>         On 27/01/2021 3:40 p.m., Bill Dunlap wrote:
> >>
> >>             I believe the problem is from svn 77925 in
> >>             gnuwin/front-ends/rcmdfn.c,
> >>             which was committed a few days after 3.6.3 was released.
> >>             Rterm used
> >>             to put double quotes around a command line argument only
> >> if it
> >>             contained a space, now it double quotes all arguments. It
> >>             sees shell
> >>             constructs like "1>" and the following file name as
> >>             arguments and
> >>             double quoting them hides them from the shell, leading to
> >> this
> >>             problem. I think we may have to rely on the user supplying
> >>             quotes as
> >>             needed instead of blindly adding them.
> >>
> >>
> >>         Okay, now I see what you mean.
> >>
> >>         If you invoke R using R.exe, it asks cmd.exe to run Rterm.exe,
> >>         so it is
> >>         possible that redirection would be handled.
> >>
> >>         If you invoke R directly using Rterm.exe, then my description
> >>         down below
> >>         would be correct.
> >>
> >>         Duncan Murdoch
> >>
> >>
> >>
> >>             -Bill
> >>
> >>             On Wed, Jan 27, 2021 at 12:28 PM Duncan Murdoch
> >>             <[hidden email] <mailto:[hidden email]>>
> >>             wrote:
> >>
> >>
> >>                 On 27/01/2021 3:17 p.m., Duncan Murdoch wrote:
> >>
> >>                     On 27/01/2021 3:38 a.m., Martin Maechler wrote:
> >>
> >>                                             Martin Maechler
> >>                                             on Tue, 26 Jan 2021 12:37:58
> >>                                             +0100 writes:
> >>
> >>
> >>                                             Marcel Baumgartner
> >>                                             on Tue, 26 Jan 2021 08:55:48
> >>                                             +0100 writes:
> >>
> >>
> >>                         Dear all, my colleague posted our issue on
> >>                         stackoverflow:
> >>
> >>                         Calling R script from Python does not save log
> >>                         file in
> >>                         version 4 - Stack Overflow
> >>                         [stackoverflow.com/questions...
> >> <https://stackoverflow.com/questions/65887485/calling-r-script-from-python-does-not-save-log-file-in-version-4>]
> >>
> >>                         It is about this kind of call to R:
> >>
> >>                         R.exe -f code.R --args "~/file.txt" 1>
> >>                         "~/log.txt" 2>&1".
> >>
> >>                         The issue is that the log.txt file is not
> >>                         created when
> >>                         running R 4.x.x. The same code works perfectly
> >>                         fine with
> >>                         R 3.6.x.
> >>
> >>                         Any idea what's going wrong as of version 4?
> >> Regards
> >>                         Marcel
> >>
> >>                         Dear Marcel, I think the solution is
> >> embarrassingly
> >>                         simple:
> >>
> >>                          From the SO post, where she showed a bit more
> >>                         detail than you
> >>                         show here, it's clear you have confused
> >> 'R.exe' and
> >>                         'Rscript.exe' and what you say above is not
> >> true:
> >>
> >>                         'R.exe' was used for R 3.6.0 but for R 4.0.3,
> >>                         you/she used
> >>                         'Rscript.exe' instead.
> >>
> >>
> >>                         ... as you've noticed now, they do behave
> >>                         differently,
> >>                         indeed!
> >>
> >>                         Well, this was not the solution to their --
> >>                         Windows-only -- problem.
> >>                         The problem *is* indeed visible if they only use
> >>                         R.exe (also
> >>                         for R 4.0.3).
> >>
> >>                         I've commented more on the SO issue (see above),
> >>                         notably asking for a *minimal* repr.ex.
> >>                         (reproducible example),
> >>                         and one *not* using "<YOUR PATH>" and setwd() ..
> >>
> >>
> >>                     Isn't this purely a Python or user problem? R
> >>                     shouldn't process
> >>                     redirection directives like
> >>
> >>                     1> "~/log.txt" 2>&1
> >>
> >>                     because it's the shell's job to process those. If
> >>                     Python is acting as
> >>                     the shell, it needs to handle those things. If R was
> >>                     handling the
> >>                     command via
> >>
> >>
> >>                 Oops, sent before finishing:
> >>
> >>                 If R was handling the command via system() or system2(),
> >>                 it would handle
> >>                 redirection itself. If it was using the Windows-only
> >>                 shell(), it would
> >>                 call cmd.exe (by default) to handle redirection. (This
> >>                 is a difference
> >>                 between R on Windows and R in Unix: in Unix a shell is
> >>                 always used.)
> >>
> >>                 Duncan Murdoch
> >>
> >>                 ______________________________________________
> >>                 [hidden email] <mailto:[hidden email]>
> >>                 mailing list -- To UNSUBSCRIBE and more, see
> >>                 stat.ethz.ch/mailman/listin...
> >> <https://stat.ethz.ch/mailman/listinfo/r-help>
> >>                 PLEASE do read the posting guide
> >>                 www.R-project.org/posting-g...
> >> <http://www.R-project.org/posting-guide.html>
> >>                 and provide commented, minimal, self-contained,
> >>                 reproducible code.
> >>
> >>
> >>
> >
> > ______________________________________________
> > [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide
> > http://www.R-project.org/posting-guide.html
> > and provide commented, minimal, self-contained, reproducible code.
>
>

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.