[TriLUG] Files in a directory question

[Steve Litt]

On Friday 13 February 2009 02:23:43 pm Steve Kuekes wrote:
> Fellow triluggers,
>
> Here's a question, how many files should I put in one directory on an
> ext3 file system?  I've got an app that needs a bunch of jpg image
> files.  Right now its ~8,000-10,000 but later it could be 2 or 3 times
> that many.  So for performance reasons, should I split these files up
> into multiple sub-directories.  I can split them up according to the
> names and put just 100 or 1000 in each directory so all the files that
> start with a,b,c,d are in one folder called abcd, etc.  or I can just
> put them all in one folder.
>
> I know that theoretically it's probably a very large file number limit
> before it breaks, but for practical performance reasons in accessing
> files in the folder how many is too many?

This is just one man's opinion.

Unless you access this directory very, very rarely, you've already passed the 
point where you should make it into a directory tree rather than a single 
directory. If I remember correctly, 10,000 files yields a several second 
latency.

I'd suggest you test this by writing a perl or ruby program that creates 
10,000 different-named small files. Then cat one of the files, and see how 
long it takes to cat the thing. To give you an idea of how fast a file can 
load when unencumbered by massive directories, my UMENU program, which has a 
separate file for each submenu, displays before your keypressing finger has 
left the key, even on files that aren't cached. On a human scale, it's 
instant.

HTH

SteveT

Steve Litt
Recession Relief Package
http://www.recession-relief.US