[TriLUG] Request for a new file system

[Sean Korb]

On Thu, Dec 9, 2010 at 3:13 PM, Scott Lambdin <lopaki at gmail.com> wrote:

> Someone write me a new file system driver, please.   It is for an
> application's archive storage.  The file system must be fast when an
> application writes a file, so the file will at first not be compressed.
> But after some time, the system will compress the file during down cycles.
> But, the files will look the same to the dumb application, whether
> compressed or not.
>

I know that a filesystem is really great for a solution... especially if
it's open source and widely available.  Maybe we can make a filesystem using
zlib.

But I get to play with someone else's money, and there are providers in this
space at the appliance level for Linux.  I use Ocarina products, and for
170TB, they offline compress and deduplicate based on file type with matched
compression algorithms based on policy.  They seamlessly offer the file back
as if it was the real deal, and optomize for decompression so in the
unlikely event you actualy want the file, it is presented in full.  With
mixed unstructured data and 5 weeks of snapshots I'm getting better than 50%
compression... though I bet I'm 30% duplicated in my environment anyway.
I'm only doing this on the mirror, but I understand that some major content
providers are using this on the front end.

Kudos where deserved, I suppose.  I'm not a sales droid, but I play one on
TV.

-- 
Sean Korb spkorb at spkorb.org http://www.spkorb.org
'65,'68 Mustangs,'68 Cougar,'78 R100/7,'60 Metro,'59 A35,'71 Pantera #1382
"The more you drive, the less intelligent you get" --Miller
"Computers are useless.  They can only give you answers." -P. Picasso