phenomenon

When running the archiving of edu OCR img table, every tens of thousands of data runs out of memory

PHP Fatal error:  Allowed memory size of 134217728 bytesexhausted (tried toallocate 135168 bytes)

Trace code discovery is caused by the following code at insert time:

EduOCRTaskBackup::getDb()->createCommand()->batchInsert(EduOCRTaskBackup::tableName(), $fields, $data)->execute();

After execute, the usage memory will increase, and some of the unset variable memory will not be deleted until the memory is exhausted.

After tracing the specific code block of execute in Yii2, it is found that a high memory will be used when log ging. After analyzing the code, the code block causing the leak is as follows:

Leaking code block

/**
 * Logs a message with the given type and category.
 * If [[traceLevel]] is greater than 0, additional call stack information about
 * the application code will be logged as well.
 * @param string|array $message the message to be logged. This can be a simple string or a more
 * complex data structure that will be handled by a [[Target|log target]].
 * @param integer $level the level of the message. This must be one of the following:
 * `Logger::LEVEL_ERROR`, `Logger::LEVEL_WARNING`, `Logger::LEVEL_INFO`, `Logger::LEVEL_TRACE`,
 * `Logger::LEVEL_PROFILE_BEGIN`, `Logger::LEVEL_PROFILE_END`.
 * @param string $category the category of the message.
 */
public function log($message, $level, $category = 'application')
{
    $time = microtime(true);
    $traces = [];
    if ($this->traceLevel > 0) {
        $count = 0;
        $ts = debug_backtrace(DEBUG_BACKTRACE_IGNORE_ARGS);
        array_pop($ts); // remove the last trace since it would be the entry script, not very useful
        foreach ($ts as $trace) {
            if (isset($trace['file'], $trace['line']) && strpos($trace['file'], YII2_PATH) !== 0) {
                unset($trace['object'], $trace['args']);
                $traces[] = $trace;
                if (++$count >= $this->traceLevel) {
                    break;
                }
            }
        }
    }
    
    // This is the main cause of memory
    $this->messages[] = [$message, $level, $category, $time, $traces];
    if ($this->flushInterval > 0 && count($this->messages) >= $this->flushInterval) {
        $this->flush();
    }
}

Cause analysis of memory leak

After 156 lines of vendor/yiisoft/yii2/log/Logger.php:156 log function in Yii2 framework, count ($this - > messages) > = $this - > flushinterval will be judged
That is, the number of messages stored in memory must be greater than or equal to the preset $this - > flushinterval to brush the messages in memory to disk.

If the 128M memory set by php.ini is full before refreshing to disk, an error will be reported directly to apply for memory exhaustion.

A lot of discussion about the memory leak of YII2 for other reasons
https://github.com/yiisoft/yii2/issues/13256

Solution

1. At the beginning of the program, set flushInterval to a smaller value

\Yii::getLogger()->flushInterval = 100; // Set to a smaller value

1. During program execution, flush the message in memory after each execution

\Yii::getLogger()->flush(true); // Passing the parameter "true" means that the message will be cleaned up to the disk every time