[clug] Shell scripting problem using 'process substitution' [ >( pipeline ) ]

steve jenkin sjenkin at canb.auug.org.au
Thu Nov 12 04:52:26 UTC 2015


> On 12 Nov 2015, at 12:17 PM, Bob Edwards <bob at cs.anu.edu.au> wrote:
> 
> Hi Steve,
> 
> Not really addressing your question, but I wonder why "Just because"
> you are appending to one file? This won't scale very well and will
> become harder to manage as time goes on.
> 
> In bash, it is trivial to create a per-day file with the date encoded
> into the file name. I would recommend something like RFC 3339 format
> (YYYY-MM-DD) to keep lexical ordering of the files consistent.
> 
> rsync will have no problem copying these to your destination box.
> 
> removing old data is as easy as: rm ADSL_WGET_2012-* etc.
> 
> and your low-power Linux box won't break when your monolithic file
> fills the file-system and doesn't leave you any wiggle room to
> edit it etc.
> 
> grep'ing over the data multiple times will then be a non-issue.
> 
> cheers,
> 
> Bob Edwards.

Bob,

Thanks for the great response.

You’re right, it won’t scale and has multiple pretty nasty failure modes - all silent, which is worse.

The “just because” is that I hacked together an infinite while loop with:
 ‘do wget | {select & process line}; sleep 2; done >>logfile'

Simple and robust and not what I’d put up on a production machine.

Because I redirect STDOUT on the loop construct, not on the wget pipeline, if I rename the file, the inode that’s capturing output won’t be changed :(

BUT, thanks very much for the comment - it gives me a nice way forward, but creates a headache with ‘rsync’ - I no longer know the exact filename to copy.
If I run ‘date’ in the loop to generate a filename and move the STDOUT redirection/append to the wget pipeline

I tried the obvious on the low-power box, but it doesn’t work [old embedded distro. not going to update it]:
> date --rfc-3339=date


The old form does work :)
> date +%Y-%m-%d

Now I have to work on the ‘rsync’, but that’ll be trivial
>  rsync user at remotehost:dir/${LOGFILE}_* ./

then think about my summary stats scripts - the ones that I was using the " >( process-list ) “ construct with.
On my Desktop, I have to pick-up the latest file…
[My usual is file=$(‘ls -1tr dir/${LOGFILE}_*|tail -1) ]

Again, thanks for the suggestions.

cheers
steve

New loop:
> OUTFILE=netcomm-link-SNR 

> while :
> do
>  DT=`date +%y-%m-%d`
>  TM=`date +%H:%M:%S`
>  (echo "${DT} ${TM}"
>   wget -q -O-  --user=${USER} "--password=${PASS}" http://192.168.1.1/statsadsl.html|\
>        sed -n -e "/SNR Margin/,+2p" -e "/Rate/,+2p" -e "/Super Frames:/,+2p" -e "/Super Frame Errors:/,+2p" -e "/RS Words:/,+2p" -e "/RS Correctable Errors:/,+2p" -e "/RS
> Uncorrectable Errors:/,+2p")|\
>        sed -n -e '/^[0-9]/s/ / /' -e 's/^  *//' -e 's/<[/]*td[^>]*>//g' -e 's/<[/]*nobreak>//g' -e 's/&nbsp /0/g' -e 's/ //g' -ep|\
>        sed -n -e '/SNR Margin/{N;N;s/\n/\t/g}' -e '/Rate/{N;N;s/\n/\t/g}' -e '/Super Frames:/{N;N;s/\n/\t/g}' -e '/Super Frame Errors:/{N;N;s/\n/\t/g}' -e '/RS Words:/{N;N
> ;s/\n/\t/g}' -e '/RS Correctable Errors:/{N;N;s/\n/\t/g}' -e '/RS Uncorrectable Errors:/{N;N;s/\n/\t/g}' -ep|\
>        sed -n -e '/^[0-9]/{N;N;N;N;N;N;N;N;s/\n/\t/g}' -ep |\
>        cut -f1-2,4-5,7-8,10-11,13-14,16-17,19-20,22-23,25-26 >>"${OUTFILE}_${DT}"
>  sleep ${delay}
> done 


Old loop:
> OUTFILE=netcomm-link-SNR 
> while :
> do
>  DT=`date +%y-%m-%d`
>  TM=`date +%H:%M:%S`
>  (echo "${DT} ${TM}"
>   wget -q -O-  --user=${USER} "--password=${PASS}" http://192.168.1.1/statsadsl.html|\
>        sed -n -e "/SNR Margin/,+2p" -e "/Rate/,+2p" -e "/Super Frames:/,+2p" -e "/Super Frame Errors:/,+2p" -e "/RS Words:/,+2p" -e "/RS Correctable Errors:/,+2p" -e "/RS
> Uncorrectable Errors:/,+2p")|\
>        sed -n -e '/^[0-9]/s/ / /' -e 's/^  *//' -e 's/<[/]*td[^>]*>//g' -e 's/<[/]*nobreak>//g' -e 's/&nbsp /0/g' -e 's/ //g' -ep|\
>        sed -n -e '/SNR Margin/{N;N;s/\n/\t/g}' -e '/Rate/{N;N;s/\n/\t/g}' -e '/Super Frames:/{N;N;s/\n/\t/g}' -e '/Super Frame Errors:/{N;N;s/\n/\t/g}' -e '/RS Words:/{N;N
> ;s/\n/\t/g}' -e '/RS Correctable Errors:/{N;N;s/\n/\t/g}' -e '/RS Uncorrectable Errors:/{N;N;s/\n/\t/g}' -ep|\
>        sed -n -e '/^[0-9]/{N;N;N;N;N;N;N;N;s/\n/\t/g}' -ep |\
>        cut -f1-2,4-5,7-8,10-11,13-14,16-17,19-20,22-23,25-26
>  sleep ${delay}
> done >>${OUTFILE}


--
Steve Jenkin, IT Systems and Design 
0412 786 915 (+61 412 786 915)
PO Box 48, Kippax ACT 2615, AUSTRALIA

mailto:sjenkin at canb.auug.org.au http://members.tip.net.au/~sjenkin




More information about the linux mailing list