pipe(): input to, and output from, a single process

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

pipe(): input to, and output from, a single process

Greg Minshall
hi.  i'd like to instantiate sed(1), send it some input, and retrieve
its output, all via pipes (rather than an intermediate file).

my sense from pipe and looking at the sources (sys-unix.c) is that is
not possible.  is that true?  are there any thoughts of providing such a
facility?

cheers, Greg

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: pipe(): input to, and output from, a single process

Gábor Csárdi
I am not sure if `pipe()` works for this, but if it turns out that it
does not, then you can use the processx package, e.g.:

> p <- processx::process$new("sed", c("-l", "s/a/x/g"), stdin = "|", stdout = "|")
> p$write_input("foobar\n")
> p$read_output()
[1] "foobxr\n"

The `-l` sed flag is to make sed line-buffered, otherwise it is will
not produce output until there is enough.

`$write_input()` and `$read_output()` are not easy to program, in particular:
* `$write_input()` returns the chunk of data that it hasn't managed to
write into the pipe. You need to call `$write_input() again, with this
data next, usually.
* `$read_output()` returns an empty string if there is no data to
read, so typically you want to call `p$poll()` first, to make sure
that there is something to read.
* `$read_output()` might not read whole lines, so maybe
`$read_output_lines()` is better for you.
* Close the stdin of the process if you want to quit cleanly:
`close(p$get_input_connection())`.
* There is currently no way to poll the input side of the pipe. :(

HTH, Gabor

On Mon, Mar 16, 2020 at 11:31 AM Greg Minshall <[hidden email]> wrote:

>
> hi.  i'd like to instantiate sed(1), send it some input, and retrieve
> its output, all via pipes (rather than an intermediate file).
>
> my sense from pipe and looking at the sources (sys-unix.c) is that is
> not possible.  is that true?  are there any thoughts of providing such a
> facility?
>
> cheers, Greg
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: pipe(): input to, and output from, a single process

Ivan Krylov
In reply to this post by Greg Minshall
On Fri, 13 Mar 2020 20:26:43 +0300
Greg Minshall <[hidden email]> wrote:

> my sense from pipe and looking at the sources (sys-unix.c) is that is
> not possible.  is that true?  are there any thoughts of providing
> such a facility?

Pipes (including those created by popen(3), which R pipe() uses
internally) are uni-directional data channels. While it could be
possible to open two pipes for both stdin and stdout of the child
process, doing so correctly is complicated because of differences in
buffering provided by the runtime: when stdin/stdout is not a terminal,
buffering mode may be set to block-oriented instead of line-oriented,
resulting in both parent and child being dead-locked, waiting to fill
the buffer instead of returning from the blocking call after the first
newline. (Hence the -l flag to sed mentioned by Gábor Csárdi, which
avoids this problem for sed).

Programs designed to first read stdin until end-of-file, then process
the input and print results on the stdout are usually safe to use in
this way, but others may be not. Software specifically designed to
control other software (e.g. Expect [*]) gets around this limitation by
running the child processes inside pseudo-terminals and/or running in
event-driven manner, being ready to service the child process whether
it wants to read its stdin or write to stdout.

Since sed has its -l flag, it should be possible to safely drive it
line-by-line with the help of processx, but not via pipe().

--
Best regards,
Ivan

[*] https://core.tcl-lang.org/expect/index

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: pipe(): input to, and output from, a single process

Dirk Eddelbuettel
In reply to this post by Greg Minshall

On 13 March 2020 at 20:26, Greg Minshall wrote:
| hi.  i'd like to instantiate sed(1), send it some input, and retrieve
| its output, all via pipes (rather than an intermediate file).
|
| my sense from pipe and looking at the sources (sys-unix.c) is that is
| not possible.  is that true?  are there any thoughts of providing such a
| facility?

Octave had this already in the 1990s, see documentation for 'popen2' here:

  https://octave.org/doc/v4.2.1/Controlling-Subprocesses.html

As it says 'Start a subprocess with two-way communication'.

Dirk


--
http://dirk.eddelbuettel.com | @eddelbuettel | [hidden email]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: pipe(): input to, and output from, a single process

Greg Minshall
Dirk,

> Octave had this already in the 1990s, see documentation for 'popen2' here:

thanks.  unix that had since the 1970s... :)

cheers, Greg

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: pipe(): input to, and output from, a single process

Greg Minshall
In reply to this post by Gábor Csárdi
Gabor, thanks.  yes, managing the two-way communication is always a bit
error-prone, as it depends on the input/output characteristics of the
two ends -- they either match, or deadlock.  it's too bad if polling is
always *required* -- i'd think sometimes a programmer would be happy
blocking, though other times one wants better control over when to
block.  cheers, Greg

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: pipe(): input to, and output from, a single process

Gábor Csárdi
Well, if you want blocking, you can poll with an infinite timeout.
This returns if
1) there is output,
2) the process terminates, or
3) you interrupt with CTRL+C / ESC /etc.

and then right after the polling, you can read the output. This still
works if the process has finished already.

Gabor

On Mon, Mar 16, 2020 at 7:06 PM Greg Minshall <[hidden email]> wrote:
>
> Gabor, thanks.  yes, managing the two-way communication is always a bit
> error-prone, as it depends on the input/output characteristics of the
> two ends -- they either match, or deadlock.  it's too bad if polling is
> always *required* -- i'd think sometimes a programmer would be happy
> blocking, though other times one wants better control over when to
> block.  cheers, Greg

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: pipe(): input to, and output from, a single process

Simon Urbanek
In reply to this post by Greg Minshall
FWIW if you're on unix, you can use named pipes (fifos) for that:

> system("mkfifo my.output")
> p = pipe("sed -l s:hello:oops: > my.output", "w")
> i = file("my.output", "r", blocking=FALSE, raw=TRUE)
> writeLines("hello!\n", p)
> flush(p)
> readLines(i, 1)
[1] "oops!"

Cheers,
Simon



> On 14/03/2020, at 6:26 AM, Greg Minshall <[hidden email]> wrote:
>
> hi.  i'd like to instantiate sed(1), send it some input, and retrieve
> its output, all via pipes (rather than an intermediate file).
>
> my sense from pipe and looking at the sources (sys-unix.c) is that is
> not possible.  is that true?  are there any thoughts of providing such a
> facility?
>
> cheers, Greg
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: pipe(): input to, and output from, a single process

Greg Minshall
Simon,

> FWIW if you're on unix, you can use named pipes (fifos) for that:

i've always wondered what named pipes actually were.  thanks!

cheers, Greg

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel