How can I use var_dump + output buffering without memory errors?

Question

I\'m using a debugging aid in an application that uses var_dump() with output buffering to capture variables and display them. However, I\'m running into an iss

Answer 1

Well, if the physical memory is limited (you see the fatal error:)

Fatal error: Allowed memory size of 536870912 bytes exhausted

I would suggest to do the output buffering on disk (see callback parameter on ob_start). Output buffering works chunked, that means, if there still is enough memory to keep the single chunk in memory, you can store it into a temporary file.

// handle output buffering via callback, set chunksize to one kilobyte
ob_start($output_callback, $chunk_size = 1024);

However you must keep in mind that this will only prevent the fatal error while buffering. If you now want to return the buffer, you still need to have enough memory or you return the file-handle or file-path so that you can also stream the output.

However you can use that file then to obtain the size in bytes needed. The overhead for PHP strings is not much IIRC, so if there still is enough memory free for the filesize this should work well. You can substract offset to have a little room and play safe. Just try and error a little what it makes.

Some Example code (PHP 5.4):

<?php
/**
 * @link http://stackoverflow.com/questions/5446647/how-can-i-use-var-dump-output-buffering-without-memory-errors/
 */

class OutputBuffer
{
    /**
     * @var int
     */
    private $chunkSize;

    /**
     * @var bool
     */
    private $started;

    /**
     * @var SplFileObject
     */
    private $store;

    /**
     * @var bool Set Verbosity to true to output analysis data to stderr
     */
    private $verbose = true;

    public function __construct($chunkSize = 1024) {
        $this->chunkSize = $chunkSize;
        $this->store     = new SplTempFileObject();
    }

    public function start() {
        if ($this->started) {
            throw new BadMethodCallException('Buffering already started, can not start again.');
        }
        $this->started = true;
        $result = ob_start(array($this, 'bufferCallback'), $this->chunkSize);
        $this->verbose && file_put_contents('php://stderr', sprintf("Starting Buffering: %d; Level %d\n", $result, ob_get_level()));
        return $result;
    }

    public function flush() {
        $this->started && ob_flush();
    }

    public function stop() {
        if ($this->started) {
            ob_flush();
            $result = ob_end_flush();
            $this->started = false;
            $this->verbose && file_put_contents('php://stderr', sprintf("Buffering stopped: %d; Level %d\n", $result, ob_get_level()));
        }
    }

    private function bufferCallback($chunk, $flags) {

        $chunkSize = strlen($chunk);

        if ($this->verbose) {
            $level     = ob_get_level();
            $constants = ['PHP_OUTPUT_HANDLER_START', 'PHP_OUTPUT_HANDLER_WRITE', 'PHP_OUTPUT_HANDLER_FLUSH', 'PHP_OUTPUT_HANDLER_CLEAN', 'PHP_OUTPUT_HANDLER_FINAL'];
            $flagsText = '';
            foreach ($constants as $i => $constant) {
                if ($flags & ($value = constant($constant)) || $value == $flags) {
                    $flagsText .= (strlen($flagsText) ? ' | ' : '') . $constant . "[$value]";
                }
            }

            file_put_contents('php://stderr', "Buffer Callback: Chunk Size $chunkSize; Flags $flags ($flagsText); Level $level\n");
        }

        if ($flags & PHP_OUTPUT_HANDLER_FINAL) {
            return TRUE;
        }

        if ($flags & PHP_OUTPUT_HANDLER_START) {
            $this->store->fseek(0, SEEK_END);
        }

        $chunkSize && $this->store->fwrite($chunk);

        if ($flags & PHP_OUTPUT_HANDLER_FLUSH) {
            // there is nothing to d
        }

        if ($flags & PHP_OUTPUT_HANDLER_CLEAN) {
            $this->store->ftruncate(0);
        }

        return "";
    }

    public function getSize() {
        $this->store->fseek(0, SEEK_END);
        return $this->store->ftell();
    }

    public function getBufferFile() {
        return $this->store;
    }

    public function getBuffer() {
        $array = iterator_to_array($this->store);
        return implode('', $array);
    }

    public function __toString() {
        return $this->getBuffer();
    }

    public function endClean() {
        return ob_end_clean();
    }
}


$buffer  = new OutputBuffer();
echo "Starting Buffering now.\n=======================\n";
$buffer->start();

foreach (range(1, 10) as $iteration) {
    $string = "fill{$iteration}";
    echo str_repeat($string, 100), "\n";
}
$buffer->stop();

echo "Buffering Results:\n==================\n";
$size = $buffer->getSize();
echo "Buffer Size: $size (string length: ", strlen($buffer), ").\n";
echo "Peeking into buffer: ", var_dump(substr($buffer, 0, 10)), ' ...', var_dump(substr($buffer, -10)), "\n";

Output:

STDERR: Starting Buffering: 1; Level 1
STDERR: Buffer Callback: Chunk Size 1502; Flags 1 (PHP_OUTPUT_HANDLER_START[1]); Level 1
STDERR: Buffer Callback: Chunk Size 1503; Flags 0 (PHP_OUTPUT_HANDLER_WRITE[0]); Level 1
STDERR: Buffer Callback: Chunk Size 1503; Flags 0 (PHP_OUTPUT_HANDLER_WRITE[0]); Level 1
STDERR: Buffer Callback: Chunk Size 602; Flags 4 (PHP_OUTPUT_HANDLER_FLUSH[4]); Level 1
STDERR: Buffer Callback: Chunk Size 0; Flags 8 (PHP_OUTPUT_HANDLER_FINAL[8]); Level 1
STDERR: Buffering stopped: 1; Level 0
Starting Buffering now.
=======================
Buffering Results:
==================
Buffer Size: 5110 (string length: 5110).
Peeking into buffer: string(10) "fill1fill1"
 ...string(10) "l10fill10\n"

Answer 2

As all the others are mentioning what you ask is impossible. The only thing you can do is try to handle it as good as possible.

What you can try is to split it up into smaller pieces and then combine it. I've created a little test to try and get the memory error. Obviously a real world example might behave differently, but this seems to do the trick.

<?php
define('mem_limit', return_bytes(ini_get('memory_limit'))); //allowed memory

/*
SIMPLE TEST CLASS
*/
class test { }
$loop = 260;
$t = new Test();
for ($x=0;$x<=$loop;$x++) {
  $v = 'test'.$x;
  $t->$v = new Test();
  for ($y=0;$y<=$loop;$y++) {
    $v2 = 'test'.$y;
    $t->$v->$v2 = str_repeat('something to test! ', 200);
  }
}
/* ---------------- */


echo saferVarDumpObject($t);

function varDumpToString($v) {
  ob_start();
  var_dump($v);
  $content = ob_get_contents();
  ob_end_clean();
  return $content;
}

function saferVarDumpObject($var) {
  if (!is_object($var) && !is_array($var))
    return varDumpToString($var);

  $content = '';
  foreach($var as $v) {
    $content .= saferVarDumpObject($v);
  }
  //adding these smaller pieces to a single var works fine.
  //returning the complete larger piece gives memory error

  $length = strlen($content);
  $left = mem_limit-memory_get_usage(true);

  if ($left>$length)
    return $content; //enough memory left

  echo "WARNING! NOT ENOUGH MEMORY<hr>";
  if ($left>100) {
    return substr($content, 0, $left-100); //100 is a margin I choose, return everything you have that fits in the memory
  } else {
    return ""; //return nothing.
  }  
}

function return_bytes($val) {
    $val = trim($val);
    $last = strtolower($val[strlen($val)-1]);
    switch($last) {
        // The 'G' modifier is available since PHP 5.1.0
        case 'g':
            $val *= 1024;
        case 'm':
            $val *= 1024;
        case 'k':
            $val *= 1024;
    }

    return $val;
}
?>

UPDATE The version above still has some error. I recreated it to use a class and some other functions

• Check for recursion
• Fix for single large attribute
• Mimic var_dump output
• trigger_error on warning to be able to catch/hide it

As shown in the comments, the resource identifier for a class is different from the output of var_dump. As far as I can tell the other things are equal.

<?php  
/*
RECURSION TEST
*/
class sibling {
  public $brother;
  public $sister;
}
$brother = new sibling();
$sister = new sibling();
$brother->sister = $sister;
$sister->sister = $brother;
Dump::Safer($brother);


//simple class
class test { }

/*
LARGE TEST CLASS - Many items
*/
$loop = 260;
$t = new Test();
for ($x=0;$x<=$loop;$x++) {
  $v = 'test'.$x;
  $t->$v = new Test();
  for ($y=0;$y<=$loop;$y++) {
    $v2 = 'test'.$y;
    $t->$v->$v2 = str_repeat('something to test! ', 200);
  }
}
//Dump::Safer($t);
/* ---------------- */


/*
LARGE TEST CLASS - Large attribute
*/
$a = new Test();
$a->t2 = new Test();
$a->t2->testlargeattribute = str_repeat('1', 268435456 - memory_get_usage(true) - 1000000);
$a->smallattr1 = 'test small1';
$a->smallattr2 = 'test small2';
//Dump::Safer($a);
/* ---------------- */

class Dump
{
  private static $recursionhash;
  private static $memorylimit;
  private static $spacing;
  private static $mimicoutput = true;


  final public static function MimicOutput($v) {
    //show results similar to var_dump or without array/object information
    //defaults to similar as var_dump and cancels this on out of memory warning
    self::$mimicoutput = $v===false ? false : true;
  }

  final public static function Safer($var) {
    //set defaults
    self::$recursionhash = array();
    self::$memorylimit = self::return_bytes(ini_get('memory_limit'));

    self::$spacing = 0;

    //echo output
    echo self::saferVarDumpObject($var);
  }  

  final private static function saferVarDumpObject($var) {
    if (!is_object($var) && !is_array($var))
      return self::Spacing().self::varDumpToString($var);

    //recursion check
    $hash = spl_object_hash($var);
    if (!empty(self::$recursionhash[$hash])) {
      return self::Spacing().'*RECURSION*'.self::Eol();
    }
    self::$recursionhash[$hash] = true;


    //create a similar output as var dump to identify the instance
    $content = self::Spacing() . self::Header($var);
    //add some spacing to mimic vardump output
    //Perhaps not the best idea because the idea is to use as little memory as possible.
    self::$spacing++;
    //Loop trough everything to output the result
    foreach($var as $k=>$v) {
      $content .= self::Spacing().self::Key($k).self::Eol().self::saferVarDumpObject($v);
    }
    self::$spacing--;
    //decrease spacing and end the object/array
    $content .= self::Spacing().self::Footer().self::Eol();
    //adding these smaller pieces to a single var works fine.
    //returning the complete larger piece gives memory error

    //length of string and the remaining memory
    $length = strlen($content);
    $left = self::$memorylimit-memory_get_usage(true);

     //enough memory left?
    if ($left>$length)
      return $content;

    //show warning  
    trigger_error('Not enough memory to dump "'.get_class($var).'" memory left:'.$left, E_USER_WARNING);
    //stop mimic output to prevent fatal memory error
    self::MimicOutput(false);
    if ($left>100) {
      return substr($content, 0, $left-100); //100 is a margin I chose, return everything you have that fits in the memory
    } else {
      return ""; //return nothing.
    }  
  }

  final private static function Spacing() {
    return self::$mimicoutput ? str_repeat(' ', self::$spacing*2) : '';
  }

  final private static function Eol() {
    return self::$mimicoutput ? PHP_EOL : '';
  }

  final private static function Header($var) {
    //the resource identifier for an object is WRONG! Its always 1 because you are passing around parts and not the actual object. Havent foundnd a fix yet
    return self::$mimicoutput ? (is_array($var) ? 'array('.count($var).')' : 'object('.get_class($var).')#'.intval($var).' ('.count((array)$var).')') . ' {'.PHP_EOL : '';
  }

  final private static function Footer() {
    return self::$mimicoutput ? '}' : '';
  }

  final private static function Key($k) {
    return self::$mimicoutput ? '['.(gettype($k)=='string' ? '"'.$k.'"' : $k ).']=>' : '';
  }

  final private static function varDumpToString($v) {
    ob_start();
    var_dump($v);

    $length = strlen($v);
    $left = self::$memorylimit-memory_get_usage(true);

     //enough memory left with some margin?
    if ($left-100>$length) {
      $content = ob_get_contents();
      ob_end_clean();
      return $content;
    }
    ob_end_clean();

    //show warning  
    trigger_error('Not enough memory to dump "'.gettype($v).'" memory left:'.$left, E_USER_WARNING);

    if ($left>100) {
      $header = gettype($v).'('.strlen($v).')';
      return $header . substr($v, $left - strlen($header));
    } else {
      return ""; //return nothing.
    }  
  }

  final private static function return_bytes($val) {
      $val = trim($val);
      $last = strtolower($val[strlen($val)-1]);
      switch($last) {
          // The 'G' modifier is available since PHP 5.1.0
          case 'g':
              $val *= 1024;
          case 'm':
              $val *= 1024;
          case 'k':
              $val *= 1024;
      }

      return $val;
  }
}
?>

Answer 3

When you insall xdebug you can limit how deep var_dump follows objects. In some software products you might encounter a kind of recursion, which bloats the output of var_dump. Other than that, you could raise the memory limit.

See http://www.xdebug.org/docs/display

Answer 4

I do not believe that there is any way to determine how much memory a specific function will eventually take up. One thing you can do is use memory_get_usage() to check how much memory the script is currently taking right before $largeVar is set, then compare it with the amount after. This will give you a good idea of the size of $largeVar, and you can run trials to determine what a maximum acceptable size limit would be before you exit gracefully.

You could also reimplement the var_dump() function yourself. Have the function walk through the structure and echo the resulting content as it is generated, or store it in a temp file, rather that storing a gigantic string in memory. This will allow you to get the same desired result, but without the memory problems you are encountering.

Answer 5

I'm sorry, but I think there is no solution for your problem. You are asking for the determination of a size to prevent the memory allocation for that size. PHP can't give you an answer about "how much memory will it consume", as the ZVAL structs are created at the time of usage in PHP. Please refer to Programming PHP - 14.5. Memory Management for an overview of PHP's memory allocation internals.

You gave the correct hint "there can be anything in it" and this is the problem from my point of view. There is an architectural problem that leads to the case you describe. And I think you try to solve it on the wrong end.

For example: you can start with a switch for each type in php and try to set limits for each size. This lasts as long as nobody comes to the idea of changing the memory limit within the process.

Xdebug is a good solution as it keeps you application from exploding because of a (even non-business-critical) log function and it is a bad solution as you should no activate xdebug in production.

I think that a memory exception is the correct behavior and you should not try to work around it.

[rant]If the one who dumps a 50 megabytes or more string does not care about his/her app behavior, he/she deserves to suffer from it ;)[/rant]