off topic: combined output of concurrent processes

[Cameron Simpson]

On 15Apr2012 02:36, Amadeus W.M. <amadeus84 at verizon.net> wrote:
| > [egreshko at meimei test]$ mkfifo pipe
| > [egreshko at meimei test]$ ./io.sh > pipe
| > [egreshko at meimei test]$ cat pipe | grep ^A | wc
| >     100     100     600
| > 
| 
| It does seem to work, but can you explain why it does? Would it still 
| work if each process outputs, say, 1Mb? 100Mb? 

This is not functionally different to using a pipe on the command line,
btw.

What's happening with the pipe is that your grep does not see end of
file until _all_ the processes using the pipe (for write i.e. the curls)
have exited. So it sees all data.

if your script forks off lots of curls into a file and does not wait for
them all, then you may get to run the grep before they all finish, hence
the weird results.

Note that you can only wait for immediate children (whereas the pipe
does not show EOF until all attached processes have exited - that means
it works for non immediate children too).

Consider:

  for n in `seq 1 100`
  do  echo FOO &
  done >zot
  wait

All the echoes are immediate children of the shell. Wait works fine.

Versus:

  for n in `seq 1 100`
  do
    ( echo foo & )
  done
  wait

The echoes are grandchildren. Wait waits only for the subshells. The
subshells exits very quickly because they in turn are not waiting for
the echoes.

Versus:

  for n in `seq 1 100`
  do
    ( echo foo &
      wait
    )
  done
  wait

and:

  for n in `seq 1 100`
  do
    ( echo foo &
      wait
    ) &
  done
  wait

Both wait for all their children correctly.

Then consider this:

  for n in `seq 1 100`
  do
    { echo foo & }
  done
  wait

All immediate children. Ok.

But:

  for n in `seq 1 100`
  do
    { echo foo & } | grep snot
  done
  wait

In this case the echo is running in a subshell (needed to make the
pipeline). The wait will not wait for the echoes. (However, the grep
won't exit until the echo exits anyway because it will not see EOF on
the pipe, so the script as a whole works anyway.)

The point here is that some control constructs, particularly pipelines
and subshells, run their commands as grandchildren, being children of
the subshell instead of the main script shell. Wait is only one level
deep.

Therefore we may need to see the real code.

Cheers,
-- 
Cameron Simpson <cs at zip.com.au> DoD#743
http://www.cskk.ezoshosting.com/cs/

We're bad-t-the-bone! Bad-t-the-bone!
        - Universally feared Denizen warcry,
          Dave Svoboda, svoboda at void.rtsg.mot.com