[clug] stdio library, feof(3) can't detect EOF in a script [was Re: using 'dd' in a pipeline.]

[John Mills]

On Tue, Nov 25, 2014 at 01:22:12PM +1100, steve jenkin wrote:
> 
> On 25 Nov 2014, at 11:10 am, John Mills <johnmi at effect.net.au> wrote:
> 
> > Well, my solution is not correct. Nor complete so far...:-)
> 
> John,
> 
> Thanks for continuing to help and work through this.
> 
> How many mistakes can I make in the one thread?
> Apparently it’s unlimited :)
> My bad for not actually running your script & checking for myself :(
> 
> I checked and stdio (fread()) ignored / deleted nulls, at least on one system. Not sure how to count the nulls (\0) in those files.
> 
> This behaviour goes back to the earliest days of ‘C’ where everything was an int and logical “false” was zero, “true” was non-zero, nulls marked the end of strings [still do] and simple input/output was not “8-bit safe”...
> 
> This, roughly, was all you needed to copy from stdin to stdout. No, I haven’t compiled any version of this :)
> 
>  while (ch=getc(stdin))
>     (void) putc(ch, stdout) ;
> 
> It incorrectly terminates when a null is encountered (at least, it used to).
> While stdio library is very useful for line-based text, if you needed to process binary files, read(2) & write(2) syscalls were needed - which meant handling buffering and syncs yourself.
> 
> All my best
> steve
> 
> --
> Steve Jenkin, IT Systems and Design 
> 0412 786 915 (+61 412 786 915)
> PO Box 48, Kippax ACT 2615, AUSTRALIA
> 
> mailto:sjenkin at canb.auug.org.au http://members.tip.net.au/~sjenkin
> 

If anyone needs a more correct version of my Tcl eof-aware filter, herewith:
It uses a pipe open to get around a quirk in exec in this context.
(http://wiki.tcl.tk/1180
"When spawning an application which returns binary data via stdout do not 
"use exec, but the [open "|..."] idiom as only the latter allows you to 
change the pipe channel to binary...")

for f in white.jpg brown.jpg; do echo -n "$f";cat $f|./pipetestj3.tcl;done

# ==================== pipetestj3.tcl ==========================
set bs 4096
set blocks 0
set data {}

proc GetData {chan} {
    upvar data data bs bs blocks blocks
    set blk [read $chan $bs]
    if {$blocks eq 0} { puts " blocksize=$bs" }
    if {$blk ne {}} {
	append data $blk
	incr blocks
 	puts -nonewline "$blocks "
 	set mcmd [open "|/usr/bin/md5sum -" r+]
	fconfigure $mcmd -buffering none -blocking 1 -translation binary
	puts -nonewline $mcmd "$data"
	flush $mcmd
	chan close $mcmd write
	set mymd [read -nonewline $mcmd]
	close $mcmd
	puts $mymd
    }
    if {[eof $chan]} {
	exit
    }
}

fconfigure stdin -blocking 0 -encoding binary -translation binary -buffering none
fileevent stdin readable [list GetData stdin]

vwait forever
======================================================================

HTH
John Mills