[TriLUG] Failure Copying Large Amounts of Data

Rodney Radford rradford at mindspring.com
Fri Apr 16 22:19:55 EDT 2010


No reasonable file copy program would work like you describe.  Even if it malloc'ed a space for the entire file and read it all in memory, there would only be need for one copy.  However, I have never heard of a file copy that would malloc that much space, putting it out to swap, as that would require just additional disk reads/writes.

Perhaps you are confusing the idea that many systems will allow the kernel to continue to allocate in memory buffers for the input and output buffers so that portions of the file can be read/written in larger quantities (ie: track at a time, or for RAID an entire stripe at a time, etc). But it would only use 'free' or available memory, not pushing it out to swap, and would never allocate sizes such as you suggest (2x 14GB file size).


-----Original Message-----
>From: Steve Litt <slitt at troubleshooters.com>
>Sent: Apr 16, 2010 7:03 PM
>To: Triangle Linux Users Group General Discussion <trilug at trilug.org>
>Subject: Re: [TriLUG] Failure Copying Large Amounts of Data
>
>Depends on how the software is written. Typically it's a loop that copies the 
>source to a memory buffer, and then writes from the memory buffer. If the 
>software, for some reason, malloc()ed a file-sized buffer for the read and the 
>write, then it would be 28GB of RAM, which is more than my 24GB swapfile. My 
>experience tells me that upsizing the buffer improves things only to maybe a 
>half a megabyte, but who knows, on a 64bit machine a 14GB buffer is possible.
>
>STeveT
>
>On Friday 16 April 2010 16:36:23 Lance A. Brown wrote:
>> This is a boondoggle.  Unless I don't know something about linux swap
>> space, it's not used when moving files around.  It's used to store pages
>> from memory when a process gets swapped out to run another program.
>> 
>> BTW,  that's a LOT of swap space for a linux machine. :-)
>> 
>> Steve Litt wrote:
>> > Fascinating! I have 24GB of swap, and the total of the 9 big files is
>> > 14GB. So if somehow two different copies, one for src and one for dst, of
>> > all 14GB were stored in swap, that would have indeed overloaded swap. The
>> > diagnostic test to rule that in or out would be to make a shellscript
>> > that copies one file at a time and blocks til the copy is complete.
>> > Presumably the swap would be released after completion of a copy.
>> 
>-- 
>TriLUG mailing list : http://www.trilug.org/mailman/listinfo/trilug
>TriLUG FAQ          : http://www.trilug.org/wiki/Frequently_Asked_Questions




More information about the TriLUG mailing list