Efficiently querying one string against multiple regexes

Question

Lets say that I have 10,000 regexes and one string and I want to find out if the string matches any of them and get all the matches. The trivial way to do it would be to jus

Answer 1

I've come across a similar problem in the past. I used a solution similar to the one suggested by akdom.

I was lucky in that my regular expressions usually had some substring that must appear in every string it matches. I was able to extract these substrings using a simple parser and index them in an FSA using the Aho-Corasick algorithms. The index was then used to quickly eliminate all the regular expressions that trivially don't match a given string, leaving only a few regular expressions to check.

I released the code under the LGPL as a Python/C module. See esmre on Google code hosting.

Answer 2

You'd need to have some way of determining if a given regex was "additive" compared to another one. Creating a regex "hierarchy" of sorts allowing you to determine that all regexs of a certain branch did not match

Answer 3

I think that the short answer is that yes, there is a way to do this, and that it is well known to computer science, and that I can't remember what it is.

The short answer is that you might find that your regex interpreter already deals with all of these efficiently when |'d together, or you might find one that does. If not, it's time for you to google string-matching and searching algorithms.

Answer 4

I use Ragel with a leaving action:

action hello {...}
action ello {...}
action ello2 {...}
main := /[Hh]ello/  % hello |
        /.+ello/ % ello |
        any{0,20} "ello"  % ello2 ;

The string "hello" would call the code in the action hello block, then in the action ello block and lastly in the action ello2 block.

Their regular expressions are quite limited and the machine language is preferred instead, the braces from your example only work with the more general language.

Answer 5

Try combining them into one big regex?

Answer 6

You could combine them in groups of maybe 20.

(?=(regex1)?)(?=(regex2)?)(?=(regex3)?)...(?=(regex20)?)

As long as each regex has zero (or at least the same number of) capture groups, you can look at what what captured to see which pattern(s) matched.

If regex1 matched, capture group 1 would have it's matched text. If not, it would be undefined/None/null/...