|
|
Posted by Erwin Moller on 07/27/06 10:31
b007uk@gmail.com wrote:
> Hi all,
> I really need your advice.
> I have to store over a million files, 10 - 15 kb each, in one folder.
> The files are created by my php script, sometimes the old files are
> deleted and new ones are written.
> So, basically on every connection my script reads/deletes/ writes files
> from/to that folder.
> Right now i have only around 300 000 files in that folder, and it feels
> like its getting slower for that script to work. It does work at the
> moment, but i am not sure what will happen when there is over a million
> files there...
> Are there any limits of files that can be stored in a folder?
> Would it be better for me to use mysql? I am not sure how mysql will
> cope with millions of writes/reads
> What would you recommend?
> Thank you very much!
> p.s.I am running linux, fedora core 3
Hi,
Don't. :P
If you know that folder will contain millions of files, the underlying OS
will need more and more time to get the right file.
Some OS's are smarter than others, I do not know the details.
But better be safe than sorry, so if possible, use a database. These things
are set up to easily handle massive tablelookups by means of (smart)
indexing.
A simple approach:
(Postgresnotation, not mySQL which I avoid)
create a table that holds your content
create table files{
fileid serial primary key,
filename text,
content text
}
Now you can get the content of each file based on its id very fast because
fileid is primary key, and thus indexed.
If you want to use the filename, index that one too.
So (very fast)
SELECT content from files WHERE (fileid=238756);
because fileid is indexed.
And if you indexed filename too, this will also be very fast:
SELECT content FROM files WHERE (filename='myfile_3_4_2006.txt');
Alternatively: Maybe you can translate your whole approach to a database and
produce the results when needed instead of making millions of files.
This is hard to say since I do not know the underlying problem, but in
general you can solve this with a good designed database.
Hope that helps,
Regards,
Erwin Moller
Navigation:
[Reply to this message]
|