Is fopen() limited by the filesystem?

Question

I wrote a program to generate large .SQL files for quickly populating very large databases. I scripted it in PHP. When I started coding I was using fopen()

Answer 1

What's probably occuring is the underlying PHP build is 32bit and can't handle file pointers >4GB - see this related question.

Your underlying OS is obviously capable of storing large files, which why you're able to redirect stdout to a large file.

Incidentally, an SQL file is likely to be highly compressible, so you might like to consider using the gzip fopen wrapper to compress the file as you write it.

$file = 'compress.zlib:///path/to/my/file.sql.gz';
$f = fopen($file, 'wb');

    //just write as normal...
    fwrite($f, 'CREATE TABLE foo (....)');

fclose($f);

Your dump will be a fraction of the original size, and you can restore it simply piping the output from zcat into an SQL client, e.g. for mysql

zcat /path/to/my/file.sql.gz | mysql mydatabase

Answer 2

Is it possible that your script is taking too long to execute and has timed out?

Alternatively, is it possible that you're reaching your memory limits within the script?

Answer 3

You can write more than 2GB of data in a stream, not in a file (as the fseek internal pointer of the file is exceeding PHP limits, and streams are not usually seekable)

<? $target = fopen('test.tar', 'w'); //this is a file, limited by php to 2GB $body = str_repeat("===", 1024 * 1024); while(true) fwrite($target, $test);

<? $target = popen('cat > test.tar', 'w'); //this is a stream, no limitation here $body = str_repeat("===", 1024 * 1024); while(true) fwrite($target, $test);

Answer 4

Yes and no. Its not fopen() or fwrite() which are limited directly, its the files, which cannot exceed some dimensions depending on the filesystem. Have a look at Comparison of filesystems on Wikipedia.