mysql - Searching for existence within 100 million+ strings 5000 times efficiently -
i have text file containing 121 million string, want perform 5000 search existence (i.e. if given string exists in text file or not) , want finish these 5000 checks in 2 seconds or less.
i thought of different ways this, tried put these strings text file sql table primary index on string column , perform query 5000 times:
select * table string=given_string
then check if result exists or not, way led long execution time whole 5000 query, between 20 , 30 seconds.
i wonder if there efficient way me index/handle data , search among 5000 times in efficient way.
you add (temporary) table (even in memory) , bulk insert 5000 search values in there. afterwards create query joins table big one.
this way have 2 queries instead of 5000.
maybe speeds things bit.
Comments
Post a Comment