[clug] command to reverse 'xxd -b'??

[Hal Ashburner]

Hey Steve,

I'd just use a lookup table of size 16 and treat it nibble by nibble
Can do this in bash if you don't want to go the C route.

#!/bin/bash
set -e
set -u


declare -a table
table["0000"]="0";
table["0001"]="1";
table["0010"]="2";
table["0011"]="3";
table["0100"]="4";
table["0101"]="5";
table["0110"]="6";
table["0111"]="7";
table["1000"]="8";
table["1001"]="9";
table["1010"]="a";
table["1011"]="b";
table["1100"]="c";
table["1101"]="d";
table["1110"]="e";
table["1111"]="f";


echo \${table[1010]} = ${table[1010]} ==  a


loop, incrementing in units of 4.
with something like:
echo ${input_line:$position:4}

and if it's not divisible by 4 you have a different problem ;)

(I think I'd go the C route myself unless I was wanting to play with a
different language - x86-64 asm using AVX instructions might be fun if
you've got a Sandy Bridge or newer core i7  CPU)




On 29 October 2013 15:50, steve jenkin <sjenkin at canb.auug.org.au> wrote:

> I'm trying to understand the von Neumann 'extractor', also called
> 'corrector'. It removes 'bias' (more 0's or 1's in a bit-stream) from a
> Random Number Generator output.
>
> It looks at pair of bits, discarding '00' and '11' (bits same).
> For '01', '10', it emits the first bit and discards the second.
>
> <https://en.wikipedia.org/wiki/Randomness_extractor#Von_Neumann_extractor>
>
> So I wrote a shell script... Nothing like reusing existing wheels.
>
> Only I hit a speed bump, reversing the output of 'xxd -b'.
> [the man page says "No, we dont do this."]
>
> 'xxd -r -p' will happily reverse hex values into binary.
> [v. neat, learned that today]
>
> 'xxd -b' dumps file in binary, strings of ascii '0' and '1'.
> But I couldn't find a command that would nicely rewrite binary back into
> hex...
>
> I ended up using 'awk' :(
>
> Questions:
>
>  - anyone got a way to just use commands to do this?
>    [I could've used 'sed' to convert '0000' .. '1111' to hex.]
>    [was going to be long & tedious]
>    Any other ideas?
>
>  - PERL can probably do this trivially... example welcome!
>
>  - Python may or may not be good for bit-banging like this.
>    example welcome, if it's simple :)
>
> Thanks In Advance
> steve
>
>
> ============= sh/bash Script, commented ===========
>
> Pass input file(s) as command line parameters,
> script writes binary output to STDOUT
>
> call as:
>  script input-file >new-bin-file
>
>
> xxd -b -g0 "$@"|\       # convert input to binary, with line nrs
>                         # & ascii dump. '-p' doesn't work with '-b'
>
>  sed -e 's/^[0-9a-f][0-9a-f]*: //' -e 's/  .*//' -e 's/../&|/g'|\
>  tr '|' '\n'|grep -v -e '^$' |\ # remove line nrs & ascii dump
>                         # break binary into two bits per line
>
>  grep -v -e '^00' -e '^11'|sed -e 's/.$//'|\
>                         # von Neumann corrector
>
> (tr -d '\n';echo) |\    # join output back into one long line
>
>  sed -e 's/......../&|/g'|tr '|' '\n' |\
>                         # reformat into 8-bits per line
>
>  awk '{s = $1; n=0;
>    for (i=1; i <= length(s); i++)
>      { n=n*2; n += index("01", substr(s, i, 1))-1;}
>     printf("%02X\n", n);
>    }' |\                # not fancy. rewrite as Hex
>
>  (tr -d '\n';echo)|\    # back to one long line
>  sed -e 's/..../&|/g'|tr '|' '\n'|\     # 4hex (2bytes) per line for xxd
>
>  xxd -r -p                      # use xxd to reverse hexdump
>
> ================================================
>
>
> ============= sh/bash Script, usable [watch for line-breaks] ===========
>
> xxd -b -g0 "$@"|\
>  sed -e 's/^[0-9a-f][0-9a-f]*: //' -e 's/  .*//' -e 's/../&|/g'|tr '|'
> '\n'|grep -v -e '^$' |\
>  grep -v -e '^00' -e '^11'|sed -e 's/.$//'|(tr -d '\n';echo) |\
>  sed -e 's/......../&|/g'|tr '|' '\n' |\
>  awk '{s = $1; n=0;
>    for (i=1; i <= length(s); i++) { n=n*2; n += index("01", substr(s, i,
> 1))-1;}
>     printf("%02X\n", n);}' |\
>  (tr -d '\n';echo)|sed -e 's/..../&|/g'|tr '|' '\n'|\
>  xxd -r -p
>
> ================================================
>
>
> --
> Steve Jenkin, Info Tech, Systems and Design Specialist.
> 0412 786 915 (+61 412 786 915)
> PO Box 48, Kippax ACT 2615, AUSTRALIA
>
> sjenkin at canb.auug.org.au http://members.tip.net.au/~sjenkin
> --
> linux mailing list
> linux at lists.samba.org
> https://lists.samba.org/mailman/listinfo/linux
>