[clug] Shell scripting problem using 'process substitution' [ >( pipeline ) ]

[George at Clug]

    Steve,

Have you thought about rsync'ing the wget page individually each time
you get the page, and then on the "other machine", concatenating the
page with a file containing all the other pages ? 

Would my suggestion be easier to implement and still supply the end
product that you require?

I agree with you, I would not want to rsync the same data over and
over again. Though I was my understanding that rsync should only copy
over the new data, though I don't know how much data transfer there
would be for rsync to determine if the whole file had changed or not.

Maybe I have not grasp exactly what you want to achieve?

George.

At Thursday, 12-11-2015 on 09:12 steve jenkin wrote:


Following a line problem, I have a monitoring script running on a
low-power Linux box that uses wget to get the stats page from my
(ADSL) Netcomm router/firewall.
The script runs continuously and I rsync the file back to another
machine.
"Just because", the monitoring script only ever appends to the file,
so it gets large.

Looking at the data, I could break the file into “per day” files,
and then analyse them.
It’s easier admin to leave everything in the one file and just
select the day/s I want to process.
[Manually stop / start the monitor & break-out the days into their own
files. Can’t do this solely on the destination m/c because of
‘rsync’.]

I found myself running the same grep (for the day) twice over the long
file & counting different things, and wondered if there was a way to
use a pipeline. There is no performance reason to do this - doesn’t
take much time, it’s “just because” :)
[A problem I had encountered for work a few times and never came up
with a solution I liked.]

I’ve got two variants below that work, but I’m not happy with the
result…
To see something useful, the ">( process )” have to write to STDERR
(or /dev/tty).

If I pipe the output of the ‘inner’ count to STDOUT, then it gets
sucked up by the next step in the main pipeline and I won’t see it.

I _could_ play with file descriptors (clone STDOUT to FD-3 for
‘inner processes' and STDOUT of last process to /dev/null at the
end), but that seems a bit clumsy.

The fragment as it is now can’t be ‘just used’ in a pipeline
because it doesn’t output to STDOUT [throws it away], but STDERR.

Anyone do anything like this?

Any suggestions?

Thanks in Advance

steve


Using two outputs to STDERR
> for i in {10..12}
> do d=15-11-${i}; echo $d
>    grep "$d" netcomm-link-SNR | tee >( (echo 'tot: ' `wc -l`)
>&2 ) >( (echo 'drops: ' `grep '0$'|wc -l`) >&2 )|cat >/dev/null
> done

variant: one output to STDERR
> grep "$d" netcomm-link-SNR|tee >( (echo 'tot: ' `wc -l`) >&2 )
|(echo 'drops: ' `grep '0$'|wc -l`)


File Descriptor Fiddling. Haven’t tried this properly… Has the
problem of connecting to later process with a pipe.
the last /dev/null creates a problem.
> exec 3>&1

> grep "$d" netcomm-link-SNR | tee >( (echo 'tot: ' `wc -l`) >&3 ) >(
(echo 'drops: ' `grep '0$'|wc -l`) >&3 )|cat >/dev/null



--
Steve Jenkin, IT Systems and Design 
0412 786 915 (+61 412 786 915)
PO Box 48, Kippax ACT 2615, AUSTRALIA

mailto:sjenkin at canb.auug.org.au http://members.tip.net.au/~sjenkin


-- 
linux mailing list
linux at lists.samba.org
https://lists.samba.org/mailman/listinfo/linux