Ruby / Process management

From WhyNotWiki

1 How do I execute an external program?
2 What happens to standard error (stderr)?
3 popen
4 Forking processing
5 [Caveats (category)] Ruby's `...`, system, and exec require executable scripts (non-binaries) to have shebang line!
6 Which techniques work to start/fork a process that needs to interact with the user (as in, a text editor)?
7 Timing out an operation
8 Signals
9 Links

[edit] How do I execute an external program?

method	`stdout`?	`stderr`?	`stdin`?	real-time?	comments	process id?	exit code?
`exec("command")`	no	no	no	yes	Simple, non-interactive invocation; replaces current process with new process (in other words, any Ruby code after the `exec` will not be executed!)	NA	NA
`system("command")`	no	no	no	yes	Simple, non-interactive invocation; waits till execution is done; outputs both stdout and stderr as normal	NA	`$?.exitstatus`
result = `command`	yes	no, unless you do `2>&1`	no	buffered—output is returned only when the command has finished/exited	Same, only it capture the output of that process.	NA	`$?.exitstatus`
`pipe = IO.popen("command", "r")`	yes	no, unless you do `2>&1`	no	yes—can even read a char or a line at a time, if you want	Interactive control of other process (write to its stdin, and then read from its stdout)	`pipe.pid`	`$?.exitstatus`
`pipe = IO.popen("command", "w+")`	yes	no, unless you do `2>&1`	yes	""	""	""	""
`Open3.popen3("command")`	yes	yes	yes	yes	Very similar to `IO.popen`.
`exec("command") if fork.nil?`	NA	NA	NA	yes	Starts a child process running concurrently (in the "background").

[edit] What happens to standard error (stderr)?

For those commands that only capture stdout (`...`, IO.popen), you should probably decide between one of two options:

Discard stderr (result = `command 2>/dev/null`)
Redirect stderr to stdout (result = `command 2>&1`)

The default behavior (if you don't do either of those) is for the standard error to just appear on screen as normally would if you had run the program independently (not from a script). This can be pretty odd/confusing, though, if you were attempting to silently capturing the output of some command but you still got some output from it. It's not always clear where the output is coming from (your command or the command you're trying to capture). That's why I prefer to either discard it or capture it. But you have to remember to one or the other or it may just appear on screen instead!

[edit] `popen`

[edit] How do I use `popen`?

IO.popen("other_program", "w+") do |pipe|
  pipe.puts "here, have some input"
  pipe.close_write  # If other_program process doesn't flush its output, you probably need to use this to send an end-of-file, which tells other_program to give us its output. If you don't do this, the program may hang/block, because other_program is waiting for more input.
  output = pipe.read
end

# You can also use the return value from your block. (exit code stored in $? as usual)
output = IO.popen("other_program", "w+") do |pipe|
  pipe.puts "here, have some input"
  pipe.close_write
  pipe.read
end

http://ruby-doc.org/core/classes/IO.html#M002294

Runs the specified command string as a subprocess; the subprocess’s standard input and output will be connected to the returned IO object. If cmd_string starts with a ``-’’, then a new instance of Ruby is started as the subprocess. The default mode for the new file object is ``r’’, but mode may be set to any of the modes listed in the description for class IO.

If a block is given, Ruby will run the command as a child connected to Ruby with a pipe. Ruby’s end of the pipe will be passed as a parameter to the block. In this case IO::popen returns the value of the block.

If a block is given with a cmd_string of ``-’’, the block will be run in two separate processes: once in the parent, and once in a child. The parent process will be passed the pipe object as a parameter to the block, the child version of the block will be passed nil, and the child’s standard in and standard out will be connected to the parent through the pipe. Not available on all platforms.

[edit] (Example) Getting child process id

irb -> pipe = IO.popen('uname')
    => #<IO:0xb7dd19b0>

irb -> pipe.readlines
    => ["Linux\n"]

irb -> "Parent is #{Process.pid}"
    => "Parent is 27577"

irb -> "popen's child process is #{pipe.pid}"
    => "popen's child process is 31914"

irb -> pipe.close
    => nil

irb -> "popen's child process is #{pipe.pid}"
IOError: closed stream

[edit] What if I want to use `popen` to capture stderr too?

If you just want it combined with stdout, you can simply redirect the stderr stream into the stdout stream (2>&1).

If you want to capture them separately, check out Open3.

http://ruby-doc.org/core/classes/Open3.html#M005449

  Open3.popen3('nroff -man') { |stdin, stdout, stderr| ... }

[edit] How long will it wait?

[edit] If you call `read` (etc.), it will wait until there is more output or the command finished (EOF)

Be careful, because your process will block if there is no more output!

Bad idea:

sleeper = IO.popen("sleep 10000"); puts sleeper.read

You will be waiting for a long time. sleep will never produce any output, because (if you read your docs carefully), "Reads at most length bytes from the I/O stream, or to the end of file if length is omitted or is nil." There is no end of file... so it blocks and waits for one.

Equally bad idea, because sleep will never produce any output, not even 1 character of output.

sleeper = IO.popen("sleep 10000"); puts sleeper.read(1)

Better idea:

irb -> sleeper = IO.popen("sleep 1"); p sleeper.read
""
    => nil

Even though the sleep doesn't return any output, at least read will stop blocking when the subcommand terminates .

In the following example, read(1) causes the process to block until the subprocess flushes its output. It ends up taking a full 5 seconds before we get any input.

irb -> countdown = IO.popen("ruby -e '5.downto(0) {|i| puts i; sleep 1}'"); countdown.read(1)
5
4
3
2
1
0
    => nil

By contrast, this returns immediately, because $stdout.sync tells Ruby to flush the output immediately every time you try to output something:

irb -> countdown = IO.popen("ruby -e '$stdout.sync = true; 5.downto(0) {|i| puts i; sleep 1}'"); countdown.read(1)
    => "5"

[edit] In the case of a bi-directional pipe ('w+') (stdout+stdin)... be sure you call `close_write`!

Good idea:

irb -> IO.popen("grep food", "w+") { |pipe| pipe.puts "have some food"; pipe.close_write; output = pipe.read }
    => "have some food\n"

Bad idea:

irb -> IO.popen("grep food", "w+") { |pipe| pipe.puts "have some food"; output = pipe.read }
[Will block, waiting for input. Have to Ctrl-C to interrupt.]
IRB::Abort: abort then interrupt!!

[edit] When you close a `popen`ed pipe and that process is still running, your process will wait for it...

sleeper = IO.popen("sleep 10000") # The child process goes to sleep
sleeper.close
# The parent process will now wait for the sleep command to finish...

Sometimes you want to have more (interactive) control over the subprocesses that you start up.

You could always use the timeout library and Process.kill the child process if it takes too long.

You might also consider using fork, or using IO.select (with a timeout) to check if any more output is available...

[edit] How do you you close your pipe and kill that process without waiting for it???

irb -> pipe = IO.popen('sleep 1000')
    => #<IO:0xb7d4cd78>

irb -> pipe.pid
    => 28120

irb -> Process.kill 'TERM', pipe.pid

irb -> pipe.close

[edit] How do I read up until a certain input is reached and then exit the popen?

[edit] Simple way

def read_until(pipe, stop_at, verbose = true)
  lines = []
  while line = pipe.gets
    break if line =~ stop_at
    puts line if verbose
    lines << line
  end
  lines
end

[edit] IO.select way

I haven't figured out yet when you would need to use this method, but this is the method used by Jamis Buck's GDB wrapper (http://weblog.jamisbuck.org/assets/2006/9/25/gdb.rb), so it must have some use, right?

def read_until(pipe, stop_at, verbose = true)
  lines = []
  line = ""
  while result = IO.select([pipe])  #, nil, nil, 10)
    next if result.empty?

    c = pipe.read(1)
    break if c.nil?

    line << c
    break if line =~ stop_at

    # Start a new line?
    if line[-1] == ?\n
      puts line if verbose
      lines << line
      line = ""
    end
  end
  lines
end

The online RDocs for IO.select and Kernel.select both say to "See Kernel#select."

So instead, I will rely on my trusty Pickaxe book, p. 528-529 [1]:

IO.select(read_array
[, write_array
[, error_array
[, timeout]]] )
=> array or nil
Performs a low-level select call, which waits for data to become available from input/output devices. The first three parameters are arrays of IO objects or nil. The last is a timeout in seconds, which should be an Integer or a Float. The call waits for data to become available for any of the IO objects in read_array, for buffers to have cleared sufficiently to enable writing to any of the devices in write_array, or for an error to occur on the devices in error_array. If one or more of these conditions are met, the call returns a three-element array containing arrays of the IO objects that were ready. Otherwise, if there is no change in status for timeout seconds, the call returns nil. If all parameters are nil, the current thread sleeps forever.
select( [$stdin], nil, nil, 1.5 )       »       [[#<IO:0x401ba090>], [], []]

[edit] Forking processing

[edit] How do I fork a child process?

Depends on whether you want to run Ruby code (same program) or an external program in your child process...

# Run an external program (command) in the child process
exec(command) if fork.nil?
# ...
Process.wait

-or-

# Run some Ruby code (same program) in the child process
fork do
  puts "In child process. parent pid is #$$"
  exit 99
end
child_pid = Process.wait
puts "Child (pid #{child_pid}) terminated with status #{$?.exitstatus}"

http://corelib.rubyonrails.org/classes/Kernel.html#M002088

The parent process should use Process.wait to collect the termination statuses of its children or use Process.detach to register disinterest in their status; otherwise, the operating system may accumulate zombie processes.

[edit] `Using IO.popen('-')`

irb -> "Parent process is #{Process.pid}"
    => "Parent process is 27577"

irb -> IO.popen('-') {|pipe| $stderr.puts "#{Process.pid} is here, pipe is #{pipe}"}
32579 is here, pipe is
27577 is here, pipe is #<IO:0xb7d700ac>

[edit] Getting process id

http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/9105


> |How can I get the PID of a program run in a ruby script. I've 
> |looked at open, popen, system, and %x//, and various Process 
> |methods, but haven't figured it out.
 
> IO#pid for open, popen.  No way provided for system and `` (yet).

http://www.hostingforum.ca/262611-getting-pid-external-command.html

def my_system(*cmd)
  pid = fork do
    exec(*cmd)
    exit! 127
  end
  yield pid if block_given?
  Process.waitpid(pid)
  $?
end

irb -> my_system("sleep 3") { |pid| 
  puts "other process id is #{pid}"
  Process.kill "TERM", pid 
}
Pid is 22586
    => #<Process::Status: pid=22586,signaled(SIGTERM=15)>

[edit] [Caveats (category)] Ruby's `...`, `system`, and `exec` require executable scripts (non-binaries) to have shebang line!

Bash doesn't care! As long as the executable bit is set, it will be happy to run it for you!

> echo -e "echo 'bash_hi'" > bash_hi_no_shebang; chmod 0700 bash_hi_no_shebang
> cat bash_hi_no_shebang
echo 'bash_hi'
> /home/tyler/bash_hi_no_shebang
bash_hi

Ruby, however, is a bit more picky about what it will run for you. You will get a mysterious command not found if it doesn't like your script.

irb -> `/home/tyler/bash_hi_no_shebang`
(irb):37: command not found: /home/tyler/bash_hi_no_shebang
    => ""

! Even though the command (the file) does exist!

But this does work from ruby:

[tyler: ~]> echo -e '#!'"/bin/bash\necho 'bash_hi'" > bash_hi; chmod 0700 bash_hi
[tyler: ~]> /home/tyler/bash_hi
bash_hi

irb -> `/home/tyler/bash_hi`
    => "bash_hi\n"

The shebang line makes all the difference!

[edit] Which techniques work to start/fork a process that needs to interact with the user (as in, a text editor)?

Let's face it, some command-line programs are interactive and need full access to the [terminal]. They are "[full-screen]" and may do fancy [ANSI] tricks. Examples: editors, top.

For these cases, not all methods of starting process work equally well. system is the technique that works the best.

These work:

system('vim') (seems like the safest bet)
exec('editor') if fork.nil? (with varying degrees of success)
exec('vim') (but you can't execute code after that)

These do not work:

`vim`

Details:

irb -> system('vim temp'); p File.readlines('temp')
[it started up vim, where I wrote "hi there", saved, and exited]
["hi there\n"]
    => nil
/pre>

<pre>
irb -> `vim`
Vim: Warning: Output is not to a terminal
[Then it just hung there, not giving any feedback at all when I typed. Ctrl-C, Ctrl-D, etc. were all ineffectual. The only thing I could do at this point was Ctrl-Z.]
> pkill -9 irb
[11]+  Killed                  irb

Warning: Don't run exec('vim') if fork.nil? from within irb. That had the weirdest behavior of them all. I really do not understand what it did to the terminal, but it really screwed it up visually so it was hard to tell what you were doing or even what program you were doing it in. I was lucky to be able exit out of everything alive (and by alive I mean successfully).

irb -> exec('editor') if fork.nil?
[a bunch of guessing, trying various exit commands, and eventually escaping]

When I finally got out of irb, a reset command was necessary to fix my terminal.

It seemed to work fine though if just executed with straight ruby (not irb):

> cat temp.rb
exec('vim temp') if fork.nil?
Process.wait
p File.readlines('temp')
> ruby temp.rb
[fired up vim]
"temp" 1L, 9C written
["hi there\n"]

[edit] Timing out an operation

http://pleac.sourceforge.net/pleac_ruby/processmanagementetc.html

# implemented thanks to http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/1760
require 'timeout'

begin
    timeout(5) {
        waitsec = rand(10)
        puts "Let's see if a sleep of #{waitsec} seconds is longer than 5 seconds..."
        system("sleep #{waitsec}")
    }
    puts "Timeout didn't occur"
rescue Timeout::Error    
    puts "Timed out!"
end

[edit] Signals

[edit] Sending a Signal

http://pleac.sourceforge.net/pleac_ruby/processmanagementetc.html

Process.kill(9, pid)                    # send $pid a signal 9
Process.kill(-1, Process.getpgrp())     # send whole job a signal 1
Process.kill("USR1", $$)                # send myself a SIGUSR1
Process.kill("HUP", pid1, pid2, pid3)   # send a SIGHUP to processes in @pids
#-----------------------------
begin
    Process.kill(0, minion)
    puts "#{minion} is alive!"
rescue Errno::EPERM                     # changed uid
    puts "#{minion} has escaped my control!";
rescue Errno::ESRCH
    puts "#{minion} is deceased.";      # or zombied
rescue
    puts "Odd; I couldn't check the status of #{minion} : #{$!}"
end

[edit] Installing a Signal Handler (`trap`)

http://pleac.sourceforge.net/pleac_ruby/processmanagementetc.html

Kernel.trap("QUIT", got_sig_quit)       # got_sig_quit = Proc.new { puts "Quit\n" }
trap("PIPE", "got_sig_quit")            # def got_sig_pipe ...
trap("INT") { ouch++ }                  # increment ouch for every SIGINT
#-----------------------------
trap("INT", "IGNORE")                   # ignore the signal INT
#-----------------------------
trap("STOP", "DEFAULT")                 # restore default STOP signal handling

http://svn.tylerrick.com/public/ruby/examples/trap-chaining_two_signal_handlers.rb

# See ~/code/gemables/qualitysmith_extensions/lib/qualitysmith_extensions/kernel/trap_chain.rb for the final product

def trap_chain(*args, &block)
  previous_interrupt_handler = trap("INT") {}
  trap("INT") do
    block.call
    previous_interrupt_handler.call unless previous_interrupt_handler == "DEFAULT"
  end
end
trap_chain("INT") { puts "Handler 1"; }
trap_chain("INT") { puts "Handler 2"; puts "Exiting..."; exit }
trap_chain("INT") { puts "Handler 3" }
trap_chain("INT") { puts "Handler 4" }

Process.kill "INT", 0

loop do
  # Press Ctrl-C to interrupt this loop
end

# Outputs:
Handler 4
Handler 3
Handler 2
Exiting...

[edit] Temporarily Overriding a Signal Handler

http://pleac.sourceforge.net/pleac_ruby/processmanagementetc.html

# the signal handler
def ding
    trap("INT", "ding")
    puts "\aEnter your name!"
end

# prompt for name, overriding SIGINT
def get_name
    save = trap("INT", "ding")

    puts "Kindly Stranger, please enter your name: "
    name = gets().chomp()
    trap("INT", save)
    name
end

[edit] Links

http://pleac.sourceforge.net/pleac_ruby/processmanagementetc.html

Retrieved from "http://whynotwiki.com/Ruby_/_Process_management"

Ruby / Process management

From WhyNotWiki

Contents

[edit] How do I execute an external program?

[edit] What happens to standard error (stderr)?

[edit] `popen`

[edit] How do I use `popen`?

[edit] (Example) Getting child process id

[edit] What if I want to use `popen` to capture stderr too?

[edit] How long will it wait?

[edit] If you call `read` (etc.), it will wait until there is more output or the command finished (EOF)

[edit] In the case of a bi-directional pipe ('w+') (stdout+stdin)... be sure you call `close_write`!

[edit] When you close a `popen`ed pipe and that process is still running, your process will wait for it...

[edit] How do you you close your pipe and kill that process without waiting for it???

[edit] How do I read up until a certain input is reached and then exit the popen?

[edit] Simple way

[edit] IO.select way

[edit] Forking processing

[edit] How do I fork a child process?

[edit] `Using IO.popen('-')`

[edit] Getting process id

[edit] [Caveats (category)] Ruby's `...`, `system`, and `exec` require executable scripts (non-binaries) to have shebang line!

[edit] Which techniques work to start/fork a process that needs to interact with the user (as in, a text editor)?

[edit] Timing out an operation

[edit] Signals

[edit] Sending a Signal

[edit] Installing a Signal Handler (`trap`)

[edit] Temporarily Overriding a Signal Handler

[edit] Links

Views

Personal tools

Software

Collecting...

Other

This wiki (Meta) 1

This wiki (Meta) 2

Me

Search

Toolbox

Ruby / Process management

From WhyNotWiki

Contents

[edit] How do I execute an external program?

[edit] What happens to standard error (stderr)?

[edit] popen

[edit] How do I use popen?

[edit] (Example) Getting child process id

[edit] What if I want to use popen to capture stderr too?

[edit] How long will it wait?

[edit] If you call read (etc.), it will wait until there is more output or the command finished (EOF)

[edit] In the case of a bi-directional pipe ('w+') (stdout+stdin)... be sure you call close_write!

[edit] When you close a popened pipe and that process is still running, your process will wait for it...

[edit] How do you you close your pipe and kill that process without waiting for it???

[edit] How do I read up until a certain input is reached and then exit the popen?

[edit] Simple way

[edit] IO.select way

[edit] Forking processing

[edit] How do I fork a child process?

[edit] Using IO.popen('-')

[edit] Getting process id

[edit] [Caveats (category)] Ruby's `...`, system, and exec require executable scripts (non-binaries) to have shebang line!

[edit] Which techniques work to start/fork a process that needs to interact with the user (as in, a text editor)?

[edit] Timing out an operation

[edit] Signals

[edit] Sending a Signal

[edit] Installing a Signal Handler (trap)

[edit] Temporarily Overriding a Signal Handler

[edit] Links

Views

Personal tools

Software

Collecting...

Other

This wiki (Meta) 1

This wiki (Meta) 2

Me

Search

Toolbox

[edit] `popen`

[edit] How do I use `popen`?

[edit] What if I want to use `popen` to capture stderr too?

[edit] If you call `read` (etc.), it will wait until there is more output or the command finished (EOF)

[edit] In the case of a bi-directional pipe ('w+') (stdout+stdin)... be sure you call `close_write`!

[edit] When you close a `popen`ed pipe and that process is still running, your process will wait for it...

[edit] `Using IO.popen('-')`

[edit] [Caveats (category)] Ruby's `...`, `system`, and `exec` require executable scripts (non-binaries) to have shebang line!

[edit] Installing a Signal Handler (`trap`)