[TriLUG] Data manipulation over Samba
slitt at troubleshooters.com
Tue May 22 08:51:47 EDT 2007
On Tuesday 22 May 2007 07:30, Jim Tuttle wrote:
> So, this has been bothering me. I'm hoping someone has an answer and,
> perhaps, a reference.
> Ok, I'm running some python data processing scripts against an
> orthophoto collection residing on a disk array in the basement. There
> are about 4,300 images each about 76MB. There are several smaller files
> with each image. Part of the processing includes copying each file to
> another partition on our 14TB ATABeast. The question is this: Is any of
> this data moving over the network to my machine?
> The processing is taking forever. 215 images in 8 hours. I wondered if
> the images are being read into memory by my machine then written to the
> other partition on the array. I have this fantasy that python tells the
> processor on the disk array to do the copying, but I imagine that isn't
> true. To make matters worse, there are several connections through
> which this data traverses. The array is mounted via fiber channel to a
> Solaris cluster which offers it to a linux machine in the cube next to
> me via NFS and I'm mounting that via samba on my desktop.
> I could have and probably should have run this on the intermediate
> machine, but wasn't thinking last night. Neither the ATABeast nor the
> Solaris cluster have python installed and that's a non-starter.
If the python scripts run on your local client, then the data goes over the
wire to your local client, and back to the server's other partition.
You're not running the python scrips on the server because the server doesn't
have python. Assuming for some reason you can't install python on the server,
then you can have your client python write a shellscript (or batch file if
the server's Windows based) to the server, and then find a way to execute the
shellscript/batch file on the server. If the server's Windows based, you can
create a phony Samba server whose print command= the shellscript. If the
server is Unix, a cron or some sort of rsh can do it. If it's Unix you could
also use sockets to make a client-server setup, which wouldn't require python
on the server.
Author: Universal Troubleshooting Process books and courseware
More information about the TriLUG