[Trilug-ontopic] [bash] copying `find`ed files across network

[Tom Roche]

http://www.trilug.org/pipermail/trilug-ontopic/2012-May/000315.html
Tom Roche Wed, 23 May 2012 15:36:07 -0400
>>>> I'm trying to copy a buncha files across a network[:]

>>>> me at remote:~ $ find /work/MOD3EVAL/nsu/boundary/BLD_ddm_saprc07tc -type f | grep -ve 'CCTM\|CVS\|~$\|\.o$' | sort | xargs du -ch | tail -n 1
>>>> > 36M   total

>>>> and I want to copy only those files.

http://www.trilug.org/pipermail/trilug-ontopic/2012-May/000316.html
Alan Porter Wed May 23 15:45:15 EDT 2012
>>> tar -zcf - $(find /work/MOD3EVAL/nsu/boundary/BLD_ddm_saprc07tc -type f | grep -ve 'CCTM\|CVS\|~$\|\.o$') | ssh root at remote "tar -zxvf -"

http://www.trilug.org/pipermail/trilug-ontopic/2012-May/000317.html
Tom Roche Wed, May 23, 2012 at 05:19:04PM -0400
>> making a similar adjustment adds a datapoint (starred below):
...

>> > $ mkdir -p /home/me/code/CMAQ/BLD_ddm_saprc07tc
>> > $ pushd /home/me/code/CMAQ/BLD_ddm_saprc07tc
...
>> # case 1: `raw find`
>> * $ ssh t "pushd /work/MOD3EVAL/nsu/boundary/BLD_ddm_saprc07tc ; find . -type f | grep -ve 'CCTM\|CVS\|~$\|\.o$' | wc -l"
>> * /work/MOD3EVAL/nsu/boundary/BLD_ddm_saprc07tc ~
>> * 266

>> # case 2: `find` inside `tar`
>> * $ ssh t "pushd /work/MOD3EVAL/nsu/boundary/BLD_ddm_saprc07tc ; tar cfvz - $(find . -type f | grep -ve 'CCTM\|CVS\|~$\|\.o$')" | tar xvfz -
>> * tar: Cowardly refusing to create an empty archive
>> * Try `tar --help' or `tar --usage' for more information.
...

>> Why is the raw `find` working, but the same `find` inside `tar`
>> draws the 'empty archive' warning (presumably due to *not* finding
>> the files)?

http://www.trilug.org/pipermail/trilug-ontopic/2012-May/000318.html
Brian McCullough Wed, 23 May 2012 17:41:07 -0400
> I suspect that it is because of the order of the parameters.

> "f" requires a value, which it is, I suspect, taking [as] "z".

IIUC, your hypothesis is, 'f' should be the final parameter--no?
Above I used `tar` flags={cfvz, xvfz}. Dropping the 'v', since it
seems to merely complicate this issue, that would seem to equate to
using `tar` flags={cfz, xfz}. So fix that:

# here I use `tar` flags={zcf, zxf} -- should use a bash array :-)
REMOT_PATH='/work/MOD3EVAL/nsu/boundary/BLD_ddm_saprc07tc'
LEAF="$(basename ${REMOT_PATH})"
LOCAL_ROOT="${HOME}/code/CMAQ/${LEAF}"
REMOT_FIND="find . -type f | grep -ve 'CCTM\|CVS\|~$\|\.o$'"
REMOT_TAR="pushd ${REMOT_PATH} ; tar zcf - \$(${REMOT_FIND})"

for CMD in \
  "mkdir -p ${LOCAL_ROOT}" \
  "pushd ${LOCAL_ROOT}" \
  "ssh t \"pushd ${REMOT_PATH} ; ${REMOT_FIND} | wc -l\"" \
  "ssh t \"${REMOT_TAR}\" | tar zxf -" \
  "find . -type f | wc -l" \
  "popd" \
; do
  echo -e "$ ${CMD}"
  eval "${CMD}"
done

> $ mkdir -p /home/tlroche/code/CMAQ/BLD_ddm_saprc07tc
> $ pushd /home/tlroche/code/CMAQ/BLD_ddm_saprc07tc
> ~/code/CMAQ/BLD_ddm_saprc07tc ~
> $ ssh t "pushd /work/MOD3EVAL/nsu/boundary/BLD_ddm_saprc07tc ; find . -type f | grep -ve 'CCTM\|CVS\|~$\|\.o$' | wc -l"
> /work/MOD3EVAL/nsu/boundary/BLD_ddm_saprc07tc ~
> 266
> $ ssh t "pushd /work/MOD3EVAL/nsu/boundary/BLD_ddm_saprc07tc ; tar zcf - $(find . -type f | grep -ve 'CCTM\|CVS\|~$\|\.o$')" | tar zxf -
> tar: Cowardly refusing to create an empty archive
> Try `tar --help' or `tar --usage' for more information.
>
> gzip: stdin: not in gzip format
> tar: Child returned status 1
> tar: Error is not recoverable: exiting now
> $ find . -type f | wc -l
> 0
> $ popd

No change :-(
For completeness, I checked the other permutation ending in 'f':

# here I use `tar` flags={czf, xzf}
REMOT_PATH='/work/MOD3EVAL/nsu/boundary/BLD_ddm_saprc07tc'
LEAF="$(basename ${REMOT_PATH})"
LOCAL_ROOT="${HOME}/code/CMAQ/${LEAF}"
REMOT_FIND="find . -type f | grep -ve 'CCTM\|CVS\|~$\|\.o$'"
REMOT_TAR="pushd ${REMOT_PATH} ; tar czf - \$(${REMOT_FIND})"

for CMD in \
  "mkdir -p ${LOCAL_ROOT}" \
  "pushd ${LOCAL_ROOT}" \
  "ssh t \"pushd ${REMOT_PATH} ; ${REMOT_FIND} | wc -l\"" \
  "ssh t \"${REMOT_TAR}\" | tar xzf -" \
  "find . -type f | wc -l" \
  "popd" \
; do
  echo -e "$ ${CMD}"
  eval "${CMD}"
done
> $ mkdir -p /home/tlroche/code/CMAQ/BLD_ddm_saprc07tc
> $ pushd /home/tlroche/code/CMAQ/BLD_ddm_saprc07tc
> ~/code/CMAQ/BLD_ddm_saprc07tc ~
> $ ssh t "pushd /work/MOD3EVAL/nsu/boundary/BLD_ddm_saprc07tc ; find . -type f | grep -ve 'CCTM\|CVS\|~$\|\.o$' | wc -l"
> /work/MOD3EVAL/nsu/boundary/BLD_ddm_saprc07tc ~
> 266
> $ ssh t "pushd /work/MOD3EVAL/nsu/boundary/BLD_ddm_saprc07tc ; tar czf - $(find . -type f | grep -ve 'CCTM\|CVS\|~$\|\.o$')" | tar xzf -
> tar: Cowardly refusing to create an empty archive
> Try `tar --help' or `tar --usage' for more information.
>
> gzip: stdin: not in gzip format
> tar: Child returned status 1
> tar: Error is not recoverable: exiting now
> $ find . -type f | wc -l
> 0
> $ popd

So it seems the hypothesis fails--no? Thanks anyway.

your assistance is appreciated! Tom Roche <Tom_Roche at pobox.com>