[clug] Shell scripting problem using 'process substitution' [ >( pipeline ) ]

[Kim Holburn]

Isn't this simpler?

grep -c "$d" file

grep -c "$d.*0\$" file

Still does 2 greps per day though.

But even your solution greps the file once each day.  In your example that's 3 greps.

To do it in one pass would be better done in awk or perl or probably several other scripting languages.


On 2015/Nov/12, at 9:12 AM, steve jenkin <sjenkin at canb.auug.org.au> wrote:

> Following a line problem, I have a monitoring script running on a low-power Linux box that uses wget to get the stats page from my (ADSL) Netcomm router/firewall.
> The script runs continuously and I rsync the file back to another machine.
> "Just because", the monitoring script only ever appends to the file, so it gets large.
> 
> Looking at the data, I could break the file into “per day” files, and then analyse them.
> It’s easier admin to leave everything in the one file and just select the day/s I want to process.
> [Manually stop / start the monitor & break-out the days into their own files. Can’t do this solely on the destination m/c because of ‘rsync’.]
> 
> I found myself running the same grep (for the day) twice over the long file & counting different things, and wondered if there was a way to use a pipeline. There is no performance reason to do this - doesn’t take much time, it’s “just because” :)
> [A problem I had encountered for work a few times and never came up with a solution I liked.]
> 
> I’ve got two variants below that work, but I’m not happy with the result…
> To see something useful, the ">( process )” have to write to STDERR (or /dev/tty).
> 
> If I pipe the output of the ‘inner’ count to STDOUT, then it gets sucked up by the next step in the main pipeline and I won’t see it.
> 
> I _could_ play with file descriptors (clone STDOUT to FD-3 for ‘inner processes' and STDOUT of last process to /dev/null at the end), but that seems a bit clumsy.
> 
> The fragment as it is now can’t be ‘just used’ in a pipeline because it doesn’t output to STDOUT [throws it away], but STDERR.
> 
> Anyone do anything like this?
> 
> Any suggestions?
> 
> Thanks in Advance
> 
> steve
> 
> 
> Using two outputs to STDERR
>> for i in {10..12}
>> do d=15-11-${i}; echo $d
>>   grep "$d" netcomm-link-SNR | tee >( (echo 'tot: ' `wc -l`) >&2 ) >( (echo 'drops: ' `grep '0$'|wc -l`) >&2 )|cat >/dev/null
>> done
> 
> variant: one output to STDERR
>> grep "$d" netcomm-link-SNR|tee >( (echo 'tot: ' `wc -l`) >&2 ) |(echo 'drops: ' `grep '0$'|wc -l`)
> 
> 
> File Descriptor Fiddling. Haven’t tried this properly… Has the problem of connecting to later process with a pipe.
> the last /dev/null creates a problem.
>> exec 3>&1
> 
>> grep "$d" netcomm-link-SNR | tee >( (echo 'tot: ' `wc -l`) >&3 ) >( (echo 'drops: ' `grep '0$'|wc -l`) >&3 )|cat >/dev/null
> 
> 
> 
> --
> Steve Jenkin, IT Systems and Design 
> 0412 786 915 (+61 412 786 915)
> PO Box 48, Kippax ACT 2615, AUSTRALIA
> 
> mailto:sjenkin at canb.auug.org.au http://members.tip.net.au/~sjenkin
> 
> 
> -- 
> linux mailing list
> linux at lists.samba.org
> https://lists.samba.org/mailman/listinfo/linux

-- 
Kim Holburn
IT Network & Security Consultant
T: +61 2 61402408  M: +61 404072753
mailto:kim at holburn.net  aim://kimholburn
skype://kholburn - PGP Public Key on request