[TriLUG] FTP (Again...)

[William Sutton]

I suppose it would be too much trouble to ask $HOSPITAL to submit the file 
with a .tmp extension and rename it when the upload is complete?

barring that...I would suggest running your process every N minutes (say 
10), and sucking in files whose last modification date/time is older than 
M minutes (say 5), which lets you process only those files that haven't 
been touched in some time (or at least some amount of time greater than 
the typical delay in sending bytes to an ftp server ;) )

you should still check the format of the file before attempting to import 
it but that's another discussion.

-- 
William Sutton


On Tue, 14 Nov 2006, Mark Freeze wrote:

> I'd like to dredge up the old ftp problem I've discussed several times
> on this list to see if there are any new ideas, or really just to
> waste everyone's time...
> 
> (Preface - We use vsftpd with SSL)
> 
> We typically receive anywhere between ten and fifty files per day on
> our ftp site from our customers.  I have some scripts that monitor the
> incoming ftp directory  that take action on the uploaded files (based
> on the filename) when the upload is complete.
> 
> My question is simply this: How do you know the upload is complete?
> 
> For example, we receive five files from $LOCAL_HOSPITAL every evening
> at 10pm.  At 11pm we cron up the ol perl script to parse these files
> into databases, etc...   One evening last week, $LOCAL_HOSPITAL was
> late in transferring the files and the cron job fired in the middle of
> the upload, grabbed a few incomplete files, and crashed the rest of
> the upload.  The short(-sighted) answer that I get is, "Move the cron
> job to later in the night."  While that would work for
> $LOCAL_HOSPITAL, we receive many other files from various sources at
> differing (and unannounced) times during the day and night.   I really
> want my process to  watch this directory for me, but only get
> completed uploads.
> 
> I know I will suffer the smack-down for saying this, but WS_FTP for
> windows used to accomplish this by naming the uploaded file with a
> ".tmp" extension until the upload completes, then renaming the file
> back to its original name.  I think later versions used a temp
> directory then moved the file after completion.  I may be confusing
> the product with something else, but if it didn't work that way it
> should have.
> 
> The solutions I hate and that are against my religionization:
> 
> * - Sending a 'token' or 'semaphore' after the file has completed.  I
> can't ask my customers to do this.  It's enough to ask them to
> remember their passwords.
> 
> * - Checking the file size over and over and over and over. Anyway,
> even if the file size stays the same, how do you know that the upload
> completed and wasn't interrupted? (That one maybe unsolveable with
> FTP.)
> 
> Does anyone have anything to offer on this matter?  A bone to throw
> out? A perl of wisdom? (He! He!)
> 
> Would anyone like to help me edit the source of vsftpd to accomplish
> the same stuff as in the windows examples?
> 
> Is there another solution?
> 
> Can I quit adding stuff to this message?
> 
> Thanks in advance!
> 
> Regards,
> Mark.
>