Posted by trlists on 05/09/05 05:56

[Note that I changed the thread title to reflect that this isn't really
about templating.]

> Yes I thought of this, but in my case a flat file would be better. The
> same problem applies though:
> [quote]
> This is all no problem, except that these lists can be pretty big. And I
> wouldn't like to load them all into the memory, or have to loop through
> the file every time translate is called.
> [/quote]

I don't understand your comments. Why would a flat file be better?
Unless you have fixed-length lines and a sorted file so that you can do
a binary search, a database is going to be several orders of magnitude
faster than a flat file.

Look, fundamentally you have a simple problem -- to find a needle in a
very well-defined, orderly haystack. This is a problem that many
people have solved in the past and for which very robust solutions
exist. Those solutions are things like tree search, binary search, and
hash table lookups. An indexed database might use any of these, arrays
in memory use hashing.

So these solutions exist but it seems you are saying "I cannot use any
of those, I need to use a flat file, now tell me how to make it fast
enough to be able to do perhaps hundreds of lookups per page". My
guess is you simply can't meet that goal with nothing but a flat file,
unless your standards for page display time are very low.

The best way to make it faster is to convert your flat file to a
database and/or load it as an array. So my first question is, *why*
are you objecting to these tried and true methods of answering exactly
the question you are asking?

If you must use a pure flat file, the fastest search method I know of
is to maintain it in sorted order with fixed-length lines and then do a
binary search. Algorithms for this are readily available if you are
not familiar with the technique [a great if expensive source for
algorithms of this type is the classic book "The Art of Computer
Programming", volume 3, Searching and Sorting by Donald E. Knuth]. But
binary search on a large flat file is still likely to be unacceptably
slow for hundreds of lookups per page.

Caching will help but you will have the exact same problem with the
cache -- how do you find the data you want within the cache?

--
Tom

[Back to original message]