Seeking Unicode-savvy function for searching text in binary data

问题

I need to find unicode text inside binary data (files).

I'm seeking any C or C++ code or library that I can use on macOS. Since I guess this is also useful to other platforms, so I rather make this question not specific to macOS.

On macOS, the NSString functions, meeting my unicode savvyness needs, can't be used because they do not work on binary data.

As an alternative I've tried the POSIX complient regex functions provided on macOS, but they have some limitations:

They are not normalization-savvy, i.e. if I search for a precomposed (NFC) character, it won't find the characher if it's occuring in decomposed (NFD) form in the target data.
Case insensitive search does not work for latin NFC characters (searching for Ü does not find ü).

Example code showing these results is below.

What code or library is out there that fulfills these needs?

I do not need regex capabilities, but if there's a regex lib that can handle these requirements, I'm fine with that, too.

Basically, I need unicode text search with these options:

case-insensitive
normalization-insensitive
diacritics-insensitive
works on arbitrary binary data, finding matching UTF-8 text fragments

Here's the test code showing the results from using the TRE regex implementation on macOS:

#include <stdio.h>
#include <regex.h>

void findIn (const char *what, const char *data, int whatPre, int dataPre) {
    regex_t re;
    regcomp (&re, what, REG_ICASE | REG_LITERAL);
    int found = regexec(&re, data, 0, NULL, 0) == 0;
    printf ("Found %s (%s) in %s (%s): %s\n", what, whatPre?"pre":"dec", data, dataPre?"pre":"dec", found?"yes":"no");
}

void findInBoth (const char *what, int whatPre) {
    char dataPre[] = { '<', 0xC3, 0xA4, '>', 0};        // precomposed
    char dataDec[] = { '<', 0x61, 0xCC, 0x88, '>', 0};  // decomposed
    findIn (what, dataPre, whatPre, 1);
    findIn (what, dataDec, whatPre, 0);
}

int main(int argc, const char * argv[]) {
    char a_pre[] = { 0xC3, 0xA4, 0};        // precomposed ä
    char a_dec[] = { 0x61, 0xCC, 0x88, 0};  // decomposed ä
    char A_pre[] = { 0xC3, 0x84, 0};        // precomposed Ä
    char A_dec[] = { 0x41, 0xCC, 0x88, 0};  // decomposed Ä

    findInBoth (a_pre, 1);
    findInBoth (a_dec, 0);
    findInBoth (A_pre, 1);
    findInBoth (A_dec, 0);

    return 0;
}

Output is:

Found ä (pre) in <ä> (pre): yes
Found ä (pre) in <ä> (dec): no
Found ä (dec) in <ä> (pre): no
Found ä (dec) in <ä> (dec): yes
Found Ä (pre) in <ä> (pre): no
Found Ä (pre) in <ä> (dec): no
Found Ä (dec) in <ä> (pre): no
Found Ä (dec) in <ä> (dec): yes

Desired output: All cases should give "yes"

回答1:

I've solved the issue by writing my own pre-precessor, generating a regular expression that combines all the alternatices (case and normalization but not diacritics) and passing that to the regex function.

The complete solution is documented here.

来源：https://stackoverflow.com/questions/55885490/seeking-unicode-savvy-function-for-searching-text-in-binary-data

标签

regex

boyer-moore