[clug] Shell scripting problem using 'process substitution' [ >( pipeline ) ]
steve jenkin
sjenkin at canb.auug.org.au
Thu Nov 12 04:52:26 UTC 2015
> On 12 Nov 2015, at 12:17 PM, Bob Edwards <bob at cs.anu.edu.au> wrote:
>
> Hi Steve,
>
> Not really addressing your question, but I wonder why "Just because"
> you are appending to one file? This won't scale very well and will
> become harder to manage as time goes on.
>
> In bash, it is trivial to create a per-day file with the date encoded
> into the file name. I would recommend something like RFC 3339 format
> (YYYY-MM-DD) to keep lexical ordering of the files consistent.
>
> rsync will have no problem copying these to your destination box.
>
> removing old data is as easy as: rm ADSL_WGET_2012-* etc.
>
> and your low-power Linux box won't break when your monolithic file
> fills the file-system and doesn't leave you any wiggle room to
> edit it etc.
>
> grep'ing over the data multiple times will then be a non-issue.
>
> cheers,
>
> Bob Edwards.
Bob,
Thanks for the great response.
You’re right, it won’t scale and has multiple pretty nasty failure modes - all silent, which is worse.
The “just because” is that I hacked together an infinite while loop with:
‘do wget | {select & process line}; sleep 2; done >>logfile'
Simple and robust and not what I’d put up on a production machine.
Because I redirect STDOUT on the loop construct, not on the wget pipeline, if I rename the file, the inode that’s capturing output won’t be changed :(
BUT, thanks very much for the comment - it gives me a nice way forward, but creates a headache with ‘rsync’ - I no longer know the exact filename to copy.
If I run ‘date’ in the loop to generate a filename and move the STDOUT redirection/append to the wget pipeline
I tried the obvious on the low-power box, but it doesn’t work [old embedded distro. not going to update it]:
> date --rfc-3339=date
The old form does work :)
> date +%Y-%m-%d
Now I have to work on the ‘rsync’, but that’ll be trivial
> rsync user at remotehost:dir/${LOGFILE}_* ./
then think about my summary stats scripts - the ones that I was using the " >( process-list ) “ construct with.
On my Desktop, I have to pick-up the latest file…
[My usual is file=$(‘ls -1tr dir/${LOGFILE}_*|tail -1) ]
Again, thanks for the suggestions.
cheers
steve
New loop:
> OUTFILE=netcomm-link-SNR
> while :
> do
> DT=`date +%y-%m-%d`
> TM=`date +%H:%M:%S`
> (echo "${DT} ${TM}"
> wget -q -O- --user=${USER} "--password=${PASS}" http://192.168.1.1/statsadsl.html|\
> sed -n -e "/SNR Margin/,+2p" -e "/Rate/,+2p" -e "/Super Frames:/,+2p" -e "/Super Frame Errors:/,+2p" -e "/RS Words:/,+2p" -e "/RS Correctable Errors:/,+2p" -e "/RS
> Uncorrectable Errors:/,+2p")|\
> sed -n -e '/^[0-9]/s/ / /' -e 's/^ *//' -e 's/<[/]*td[^>]*>//g' -e 's/<[/]*nobreak>//g' -e 's/  /0/g' -e 's/ //g' -ep|\
> sed -n -e '/SNR Margin/{N;N;s/\n/\t/g}' -e '/Rate/{N;N;s/\n/\t/g}' -e '/Super Frames:/{N;N;s/\n/\t/g}' -e '/Super Frame Errors:/{N;N;s/\n/\t/g}' -e '/RS Words:/{N;N
> ;s/\n/\t/g}' -e '/RS Correctable Errors:/{N;N;s/\n/\t/g}' -e '/RS Uncorrectable Errors:/{N;N;s/\n/\t/g}' -ep|\
> sed -n -e '/^[0-9]/{N;N;N;N;N;N;N;N;s/\n/\t/g}' -ep |\
> cut -f1-2,4-5,7-8,10-11,13-14,16-17,19-20,22-23,25-26 >>"${OUTFILE}_${DT}"
> sleep ${delay}
> done
Old loop:
> OUTFILE=netcomm-link-SNR
> while :
> do
> DT=`date +%y-%m-%d`
> TM=`date +%H:%M:%S`
> (echo "${DT} ${TM}"
> wget -q -O- --user=${USER} "--password=${PASS}" http://192.168.1.1/statsadsl.html|\
> sed -n -e "/SNR Margin/,+2p" -e "/Rate/,+2p" -e "/Super Frames:/,+2p" -e "/Super Frame Errors:/,+2p" -e "/RS Words:/,+2p" -e "/RS Correctable Errors:/,+2p" -e "/RS
> Uncorrectable Errors:/,+2p")|\
> sed -n -e '/^[0-9]/s/ / /' -e 's/^ *//' -e 's/<[/]*td[^>]*>//g' -e 's/<[/]*nobreak>//g' -e 's/  /0/g' -e 's/ //g' -ep|\
> sed -n -e '/SNR Margin/{N;N;s/\n/\t/g}' -e '/Rate/{N;N;s/\n/\t/g}' -e '/Super Frames:/{N;N;s/\n/\t/g}' -e '/Super Frame Errors:/{N;N;s/\n/\t/g}' -e '/RS Words:/{N;N
> ;s/\n/\t/g}' -e '/RS Correctable Errors:/{N;N;s/\n/\t/g}' -e '/RS Uncorrectable Errors:/{N;N;s/\n/\t/g}' -ep|\
> sed -n -e '/^[0-9]/{N;N;N;N;N;N;N;N;s/\n/\t/g}' -ep |\
> cut -f1-2,4-5,7-8,10-11,13-14,16-17,19-20,22-23,25-26
> sleep ${delay}
> done >>${OUTFILE}
--
Steve Jenkin, IT Systems and Design
0412 786 915 (+61 412 786 915)
PO Box 48, Kippax ACT 2615, AUSTRALIA
mailto:sjenkin at canb.auug.org.au http://members.tip.net.au/~sjenkin
More information about the linux
mailing list