Best way to convert whole file to lowercase in C

Question

I was wondering if theres a realy good (performant) solution how to Convert a whole file to lower Case in C. I use fgetc convert the char to lower case and write it in another t

Answer 1

I'd say you've hit the nail on the head. Temp file means that you don't delete the original until you're sure that you're done processing it which means upon error the original remains. I'd say that's the correct way of doing it.

As suggested by another answer (if file size permits) you can do a memory mapping of the file via the mmap function and have it readily available in memory (no real performance difference if the file is less than the size of a page as it's probably going to get read into memory once you do the first read anyway)

Answer 2

Well, you can definitely speed this up a lot, if you know what the character encoding is. Since you're using Linux and C, I'm going to go out on a limb here and assume that you're using ASCII.

In ASCII, we know A-Z and a-z are contiguous and always 32 apart. So, what we can do is ignore the safety checks and locale checks of the toLower() function and do something like this:

(pseudo code) foreach (int) char c in the file: c -= 32.

Or, if there may be upper and lowercase letters, do a check like if (c > 64 && c < 91) // the upper case ASCII range then do the subtract and write it out to the file.

Also, batch writes are faster, so I would suggest first writing to an array, then all at once writing the contents of the array to the file.

This should be considerable faster.

Answer 3

If you're processing big files (big as in, say, multi-megabytes) and this operation is absolutely speed-critical, then it might make sense to go beyond what you've inquired about. One thing to consider in particular is that a character-by-character operation will perform less well than using SIMD instructions.

I.e. if you'd use SSE2, you could code the toupper_parallel like (pseudocode):

for (cur_parallel_word = begin_of_block;
     cur_parallel_word < end_of_block;
     cur_parallel_word += parallel_word_width) {
    /*
     * in SSE2, parallel compares are either about 'greater' or 'equal'
     * so '>=' and '<=' have to be constructed. This would use 'PCMPGTB'.
     * The 'ALL' macro is supposed to replicate into all parallel bytes.
     */
    mask1 = parallel_compare_greater_than(*cur_parallel_word, ALL('A' - 1));
    mask2 = parallel_compare_greater_than(ALL('Z'), *cur_parallel_word);
    /*
     * vector op - and all bytes in two vectors, 'PAND'
     */
    mask = mask1 & mask2;
    /*
     * vector op - add a vector of bytes. Would use 'PADDB'.
     */
    new = parallel_add(cur_parallel_word, ALL('a' - 'A'));
    /*
     * vector op - zero bytes in the original vector that will be replaced
     */
    *cur_parallel_word &= !mask;           // that'd become 'PANDN'
    /*
     * vector op - extract characters from new that replace old, then or in.
     */
    *cur_parallel_word |= (new & mask);    // PAND / POR
}

I.e. you'd use parallel comparisons to check which bytes are uppercase, and then mask both original value and 'uppercased' version (one with the mask, the other with the inverse) before you or them together to form the result.

If you use mmap'ed file access, this could even be performed in-place, saving on the bounce buffer, and saving on many function and/or system calls.

There is a lot to optimize when your starting point is a character-by-character 'fgetc' / 'fputc' loop; even shell utilities are highly likely to perform better than that.

But I agree that if your need is very special-purpose (i.e. something as clear-cut as ASCII input to be converted to uppercase) then a handcrafted loop as above, using vector instruction sets (like SSE intrinsics/assembly, or ARM NEON, or PPC Altivec), is likely to make a significant speedup possible over existing general-purpose utilities.

Answer 4

You can usually get a little bit faster on big inputs by using fread and fwrite to read and write big chunks of the input/output. Also you should probably convert a bigger chunk (whole file if possible) into memory and then write it all at once.

edit: I just rememberd one more thing. Sometimes programs can be faster if you select a prime number (at the very least not a power of 2) as the buffer size. I seem to recall this has to do with specifics of the cacheing mechanism.

Answer 5

This doesn't really answer the question (community wiki), but here's an (over?)-optimized function to convert text to lowercase:

#include <assert.h>
#include <ctype.h>
#include <stdio.h>

int fast_lowercase(FILE *in, FILE *out)
{
    char buffer[65536];
    size_t readlen, wrotelen;
    char *p, *e;
    char conversion_table[256];
    int i;

    for (i = 0; i < 256; i++)
        conversion_table[i] = tolower(i);

    for (;;) {
        readlen = fread(buffer, 1, sizeof(buffer), in);
        if (readlen == 0) {
            if (ferror(in))
                return 1;
            assert(feof(in));
            return 0;
        }

        for (p = buffer, e = buffer + readlen; p < e; p++)
            *p = conversion_table[(unsigned char) *p];

        wrotelen = fwrite(buffer, 1, readlen, out);
        if (wrotelen != readlen)
            return 1;
    }
}

This isn't Unicode-aware, of course.

I benchmarked this on an Intel Core 2 T5500 (1.66GHz), using GCC 4.6.0 and i686 (32-bit) Linux. Some interesting observations:

• It's about 75% as fast when buffer is allocated with malloc rather than on the stack.
• It's about 65% as fast using a conditional rather than a conversion table.