[TriLUG] Scheduling file transfers

[John Turner]

On Apr 7, 2005, at 3:18 PM, Mark Freeze wrote:

> The way I need my script to read will not allow for picking up part of
> a file then picking the rest of the file up later, unless by later you
> guys mean later like a couple of ms later.  My script would need to do
> the following things:
>
> 1. Check for the existance of a file.
> 2. Download the file.
> 3. Run the file through a parsing program.
> 4. Import the parsed file into a database.
> 5. Query the database and email the results to a recipient list.
> (Quantity and dollar totals of the downloaded file.)
> 6. Export the file into a Samba directory so my Windows box can pick
> it up and process it through some canned software then place the
> result file back into the Samba directory.
> 7. Use php to convert the result file into seperate pdf images.
> 8. Place the pdf images into a directory and index the pdf file list
> into a web-enabled database so users can log into my website and view
> customers bills (the pdfs) online.
>
> On some small files this process would be almost instanateous.  But on
> larger files the download might take a while so I didnt want my script
> getting part of the file and starting the next step before the
> download was complete.  In the past instance I processed a file of
> 80,000 records after I had only downloaded about 60,000.  Would rsync
> or ncftpget still work in this situation?  I am deleteing the files
> after I download.  Downloading a second file behind the first sounds
> like a good idea. I'd just have to have the script check that
> condition before continuing.  How would I go about checking the file
> size on the remote machine before download  to detect things like
> failed transfers, etc... ?
>
> Thanks,
> Mark.
>

I would suggest that the second file uploaded by the clients is an md5 
sum of the just uploaded file. You can use that as a trigger as Jeff 
suggested and as a check that the file is complete.

John