How to increase the performance of a file upload using native sftp functions and fwrite in PHP

问题

Hi I am using the following code to upload a huge file (500MB) to a sftp server.

<?php

$connection = ssh2_connect($this->host, $this->port, null);
$sftp = ssh2_sftp($connection);

$connection_string = ((int) $sftp) . $remotePath . $remoteFilename;
$stream = fopen('ssh2.sftp://' . $connection_string, 'w');
$source = fopen($localFilepath, 'r');

if (!$stream) {
    throw new Exception('Could not create file: ' . $connection_string);
}

while (!feof($source)) {
    // Chunk size 32 MB
    if (fwrite($stream, fread($source, 33554432)) === false) {
        throw new Exception('Could not send data: ' . $connection_string);
    }
}

fclose($source);
fclose($stream);

But the upload is very slow. The code is running on Google Cloud Run. The upload speed is around 8 MiB/s.

I also tried to use lftp via shell_exec but this lead to even more issues due to Cloud Run.

The uplink can't be the problem as I can send files via CURL post without any issues.

Anyone able to help here?

Many thanks and best, intxcc

回答1:

The issue is that even though 32MB are read and then written to the sftp stream, fwrite will chunk at a different size. I think just a few KB.

For filesystems (which is the usual case with fwrite) this is fine, but not with high latency due to fwriting to a remote server.

So the solution is to increase the chunk size of the sftp stream with

stream_set_chunk_size($stream, 1024 * 1024);

So the final working code is:

<?php

$connection = ssh2_connect($this->host, $this->port, null);
$sftp = ssh2_sftp($connection);

$connection_string = ((int) $sftp) . $remotePath . $remoteFilename;
$stream = fopen('ssh2.sftp://' . $connection_string, 'w');
$source = fopen($localFilepath, 'r');

// Stream chunk size 1 MB
stream_set_chunk_size($stream, 1024 * 1024);

if (!$stream) {
    throw new Exception('Could not create file: ' . $connection_string);
}

while (!feof($source)) {
    // Chunk size 32 MB
    if (fwrite($stream, fread($source, 33554432)) === false) {
        throw new Exception('Could not send data: ' . $connection_string);
    }
}

fclose($source);
fclose($stream);

Hope this helps the next person that is getting gray hair trying to figure that out ;)

来源：https://stackoverflow.com/questions/64668913/how-to-increase-the-performance-of-a-file-upload-using-native-sftp-functions-and

标签

php

upload

sftp

fwrite

ssh2-sftp