How can I import a large (14 GB) MySQL dump file into a new MySQL database?

Question

How can I import a large (14 GB) MySQL dump file into a new MySQL database?

Answer 1

I have made a PHP script which is designed to import large database dumps which have been generated by phpmyadmin or mysql dump (from cpanel) . It's called PETMI and you can download it here [project page] [gitlab page].

It works by splitting an. sql file into smaller files called a split and processing each split one at a time. Splits which fail to process can be processed manually by the user in phpmyadmin. This can be easily programmed as in sql dumps, each command is on a new line. Some things in sql dumps work in phpmyadmin imports but not in mysqli_query so those lines have been stripped from the splits.

It has been tested with a 1GB database. It has to be uploaded to an existing website. PETMI is open source and the sample code can be seen on Gitlab.

A moderator asked me to provide some sample code. I'm on a phone so excuse the formatting.

Here is the code that creates the splits.

                 //gets the config page
                  if (isset($_POST['register']) && $_POST['register'])
                {
                 echo " <img src=\"loading.gif\">";
        $folder = "split/";
        include ("config.php");
        
        $fh = fopen("importme.sql", 'a') or die("can't open file");
        $stringData = "-- --------------------------------------------------------";
        fwrite($fh, $stringData);
        fclose($fh);
        
        
        $file2 = fopen("importme.sql","r");
        
        //echo "<br><textarea class=\"mediumtext\" style=\"width: 500px; height: 200px;\">";
        $danumber = "1";
        while(! feof($file2)){
            //echo fgets($file2)."<!-- <br /><hr color=\"red\" size=\"15\"> -->"; 
            $oneline = fgets($file2); //this is fgets($file2) but formatted nicely
            //echo "<br>$oneline";
            
            $findme1  = '-- --------------------------------------------------------';
            $pos1 = strpos($oneline, $findme1);
            $findme2  = '-- Table structure for';
            $pos2 = strpos($oneline, $findme2);
            $findme3  = '-- Dumping data for';
            $pos3 = strpos($oneline, $findme3);
            $findme4  = '-- Indexes for dumped tables';
            $pos4 = strpos($oneline, $findme4);
            $findme5  = '-- AUTO_INCREMENT for dumped tables';
            $pos5 = strpos($oneline, $findme5);
            if ($pos1 === false && $pos2 === false && $pos3 === false && $pos4 === false && $pos5 === false) {

                // setcookie("filenumber",$i);
                // if ($danumber2 == ""){$danumber2 = "0";} else { $danumber2 = $danumber2 +1;}                 
                $ourFileName = "split/sql-split-$danumber.sql";
                // echo "writing danumber is $danumber";
                $ourFileHandle = fopen($ourFileName, 'a') or die("can't edit file. chmod directory to 777");

                $stringData = $oneline;
                $stringData = preg_replace("/\/[*][!\d\sA-Za-z@_='+:,]*[*][\/][;]/", "", $stringData);
                $stringData = preg_replace("/\/[*][!]*[\d A-Za-z`]*[*]\/[;]/", "", $stringData);
                $stringData = preg_replace("/DROP TABLE IF EXISTS `[a-zA-Z]*`;/", "", $stringData);
                $stringData = preg_replace("/LOCK TABLES `[a-zA-Z` ;]*/", "", $stringData);
                $stringData = preg_replace("/UNLOCK TABLES;/", "", $stringData);

                fwrite($ourFileHandle, $stringData);
                fclose($ourFileHandle);
            
            } else {
                    //write new file;
                    if ($danumber == ""){$danumber = "1";} else { $danumber = $danumber +1;}
                    $ourFileName = "split/sql-split-$danumber.sql"; 
                    //echo "$ourFileName has been written with the contents above.\n";
                    
                    $ourFileName = "split/sql-split-$danumber.sql";
                    $ourFileHandle = fopen($ourFileName, 'a') or die("can't edit file. chmod directory to 777");
                    $stringData = "$oneline";
                    fwrite($ourFileHandle, $stringData);
                    fclose($ourFileHandle);
            }
        }
        //echo "</textarea>";
        
    
    fclose($file2);

Here is the code that imports the split

<?php
ob_start();
// allows you to use cookies
include ("config.php");
//gets the config page
if (isset($_POST['register']))
{
echo "<div id**strong text**=\"sel1\"><img src=\"loading.gif\"></div>";

// the above line checks to see if the html form has been submitted
$dbname = $accesshost;
$dbhost = $username;
$dbuser = $password;
$dbpasswd = $database;
$table_prefix = $dbprefix;
//the above lines set variables with the user submitted information
    //none were left blank!  We continue...


//echo "$importme";

echo "<hr>";

$importme = "$_GET[file]";
$importme = file_get_contents($importme);
//echo "<b>$importme</b><br><br>";
$sql = $importme;
$findme1  = '-- Indexes for dumped tables';
$pos1 = strpos($importme, $findme1);
$findme2 = '-- AUTO_INCREMENT for dumped tables';
$pos2 = strpos($importme, $findme2);

$dbhost = '';
@set_time_limit(0);


if($pos1 !== false){
    $splitted = explode("-- Indexes for table", $importme);
    // print_r($splitted);
    for($i=0;$i<count($splitted);$i++){
        $sql = $splitted[$i];
        $sql = preg_replace("/[`][a-z`\s]*[-]{2}/", "", $sql);
        
        // echo "<b>$sql</b><hr>";
        if($table_prefix !== 'phpbb_') $sql = preg_replace('/phpbb_/', $table_prefix, $sql);
        $res = mysql_query($sql);
    }
    if(!$res) { echo '<b>error in query </b>', mysql_error(), '<br /><br>Try importing the split .sql file in phpmyadmin under the SQL tab.'; /* $i = $i +1; */ } else {
    echo ("<meta http-equiv=\"Refresh\" content=\"0; URL=restore.php?page=done&file=$filename\"/>Thank You! You will be redirected");
    }   
 
} elseif($pos2 !== false){
    $splitted = explode("-- AUTO_INCREMENT for table", $importme);
    // print_r($splitted);
    for($i=0;$i<count($splitted);$i++){
        $sql = $splitted[$i];
        $sql = preg_replace("/[`][a-z`\s]*[-]{2}/", "", $sql);
        
        // echo "<b>$sql</b><hr>";
        if($table_prefix !== 'phpbb_') $sql = preg_replace('/phpbb_/', $table_prefix, $sql);
        $res = mysql_query($sql);
    }
    if(!$res) { echo '<b>error in query </b>', mysql_error(), '<br /><br>Try importing the split .sql file in phpmyadmin under the SQL tab.'; /* $i = $i +1; */ } else {
    echo ("<meta http-equiv=\"Refresh\" content=\"0; URL=restore.php?page=done&file=$filename\"/>Thank You! You will be redirected");
    }   
} else {
    if($table_prefix !== 'phpbb_') $sql = preg_replace('/phpbb_/', $table_prefix, $sql);
    $res = mysql_query($sql);
    if(!$res) { echo '<b>error in query </b>', mysql_error(), '<br /><br>Try importing the split .sql file in phpmyadmin under the SQL tab.'; /* $i = $i +1; */ } else {
    echo ("<meta http-equiv=\"Refresh\" content=\"0; URL=restore.php?page=done&file=$filename\"/>Thank You! You will be redirected");
    }


 }




//echo 'done (', count($sql), ' queries).';

}

    

Answer 2

Use source command to import large DB

mysql -u username -p

> source sqldbfile.sql

this can import any large DB

Answer 3

Simple solution is to run this query: mysql -h yourhostname -u username -p databasename < yoursqlfile.sql

And if you want to import with progress bar, try this: pv yoursqlfile.sql | mysql -uxxx -pxxxx databasename