What's the best approach to sending email to hundreds of recipients from a Zend Framework application?

Question

I\'m trying to implement a mailing list system for my application. I\'m currently using Zend_Mail_Transport_Smtp(\'localhost\') as my transport, looping through

Answer 1

NOTE: when I first read your question, I thought it said hundreds of thousand emails at once. When I double checked, I noticed it actually said hundreds to thousands. I'm too lazy to change my post now, so here are some caveats: From my experience, you can probably run fine without a commercial tool to about 40K. At about 10K you'll want to be following the 'minimum' list to prevent major pain when you start reaching larger list sizes. I do recommend implementing it all right away though.

I've said this before, there are two sides to sending email:

1. The technical side -- basically all of the RFC's around the smtp protocol, email formats, DNS records, etc. This is mildly complicated but solvable.
2. The magical side -- email delivery management is voodoo. You will get frustrated, things will break for no apparent reason and you will consider leaving for another job that doesn't involve email.

I recommend not writing your own bulk sender. I'm sure PHP can do a fine job, but you should probably spend your time elsewhere. The two products I've used in the past and recommend are Strongmail and PowerMTA. Be warned -- they have a high price tag, but I can almost guarantee that you'll spend more building your own solution in the long run.

One area that you'll get nailed with writing your own in PHP is throttling/tar pitting. Mail servers will start adding in sleep(30)'s after you've sent a few messages to slow you down and stop you from spamming.

Typically, these commercial bulk senders run the SMTP protocol for queueing. You would continue to use Zend_Mail, but hard code it to connect to your server. It'll queue the mail just about as fast as you can send it, then use it's own engine to send the mail to their destinations.

At a 100K list, you will have to employ email best practices. At a minimum, you'll need:

• SPF Records, possibly DKIM as well
• Multiple IPs to segment traffic over -- have 3 IP's, one for quality address you trust, one for medium risk IP addresses and one for high risk IP addresses. This design helps minimize the risk to getting mail to your best customers.
• Proper reverse DNS for sending IP addresses
• Use the feedback loops from AOL, hotmail, yahoo and others for processing spam complaints
• Unsubscribe and bounce management -- make sure you're pruning these addresses
• Having open/click tracking is also important -- if you're customers on the A list aren't opening your emails, you need to degrade them to the B list and so forth. This is important because ISP's will turn inactive accounts into a honeypot. Hotmail is famous for this.

Finally, if you're really serious about sending email, you'll want some other tools like Return Path.

Answer 2

Use Zend_Queue to place the emails in the queue for asychronous background processing. You will need a cron job to processes the queue in the background.

protected function _enqueueEmail(WikiEmailArticle $email)
{
    static $intialized = false; 

    if (!$initialized) {

        $this->_initializeMailQueue("wikiappwork_queue");
        $initialized = true;
    }

    $this->_mailQueue->send(serialize($email));  
}
protected function _initializeMailQueue()
{
    /* See: 1.) http://framework.zend.com/manual/en/zend.queue.adapters.html and
     *      2.) Zend/Queue/Adapter/Db/mysql.sql. 
     */

 $ini = Zend_Controller_Front::getInstance()->getParam('bootstrap')
                                            ->getOptions(); 

     $queueAdapterOptions =    array( 'driverOptions' => array(
    'host' => $ini['resources']['multidb']['zqueue']['host'],
    'username' => $ini['resources']['multidb']['zqueue']['username'],
    'password' => $ini['resources']['multidb']['zqueue']['password'],
    'dbname' => $ini['resources']['multidb']['zqueue']['dbname'],
    'type' => $ini['resources']['multidb']['zqueue']['adapter'] ),
    'name' => $ini['resources']['multidb']['zqueue']['queueName'] );

    $this->_mailQueue = new Zend_Queue('Db', $queueAdapterOptions);

 }

Then for the cron job, a script like

<?php
use \Wiki\Email\WikiEmailArticle;

// Change this define to correspond to the location of the wikiapp.work/libary
define('APPLICATION_PATH', '/home/kurt/public_html/wikiapp.work/application');

set_include_path(implode(PATH_SEPARATOR, array(
     APPLICATION_PATH . '/../library',
     get_include_path(),
 )));

// autoloader (uses closure) for loading both WikiXXX classes and Zend_ classes.
spl_autoload_register(function ($className) { 

  // Zend classes need underscore converted to PATH_SEPARATOR
  if (strpos($className, 'Zend_' ) === 0) {

        $className = str_replace('_', '/', $className );   
  }

  $file = str_replace('\\', '/', $className . '.php');

  // search include path for the file.
  $include_dirs = explode(PATH_SEPARATOR, get_include_path());

  foreach($include_dirs as $dir) {

    $full_file = $dir . '/'. $file;

    if (file_exists($full_file)) { 

        require_once $full_file; 
        return true; 
    }
  }

  return false; 
 }); 

// Load and parese ini file, grabing sections we need.
$ini = new Zend_Config_Ini(APPLICATION_PATH . 
                          '/configs/application.ini', 'production');

$queue_config = $ini->resources->multidb->zqueue;

$smtp_config = $ini->email->smtp;

$queueAdapterOptions =  array( 'driverOptions' => array(
                                        'host'      => $queue_config->host,
                    'username'  => $queue_config->username,
                    'password'  => $queue_config->password,
                    'dbname'    => $queue_config->dbname,
                    'type'      => $queue_config->adapter),
                'name' => $queue_config->queuename);

$queue = new Zend_Queue('Db', $queueAdapterOptions);


$smtp = new Zend_Mail_Transport_Smtp($smtp_config->server, array(
                'auth'      => $smtp_config->auth,
        'username'  => $smtp_config->username,
        'password'  => $smtp_config->password,
        'port'      => $smtp_config->port,
        'ssl'       => $smtp_config->ssl
        ));

Zend_Mail::setDefaultTransport($smtp);

$messages = $queue->receive(10); 

foreach($messages as $message) {

        // new WikiEmailArticle.     
    $email = unserialize($message->body);

        try {

            $email->send();

        }  catch(Zend_Mail_Exception $e) {

               // Log the error?
               $msg = $e->getMessage();
               $str = $e->__toString();
               $trace =  preg_replace('/(\d\d?\.)/', '\1\r', $str);
        } // end try

$queue->deleteMessage($message);

} // end foreach

Answer 3

From the PHP.net Documentation.

Note: It is worth noting that the mail() function is not suitable for larger volumes of email in a loop. This function opens and closes an SMTP socket for each email, which is not very efficient.
For the sending of large amounts of email, see the » PEAR::Mail, and » PEAR::Mail_Queue packages.

The Zend Mail class is probably pretty good (most of Zend's stuff is good) But if you want other options. Here they are.

Answer 4

I've developed a Newsletter Management System with Swiftmailer and it's very easy to implement. It supports SMTP, encryption, attachments, batch send, ...

Answer 5

I implemented a bulk mailer in php where each email was customized to an individual. It wasn't hard and didn't take too long. I used swiftmailer and cron. Zend Mail might be ok as well. I started with the PEAR mail queue, but queueing up the emails was much too slow.

The process of queueing emails went like so:

1. Create the email template and add placeholders (or use a template engine) for areas where unique content will be substituted.
2. In a loop, substitute placeholders with in any unique content, insert the resulting email content, subject, addresses, batch id, and optionally a priority value to a database table.

I used a cron job to send out batches of emails. The cron time interval and the number of emails sent per batch were important since I was on a shared host with limits. The script that was called by the cron job was only accessible by cron. The script read x number of emails from the table ordered by batch id and, optionally, priority. If an email was sent successfully, it was deleted from the database queue. If an email couldn't be sent, it remained in the queue and a counter was incremented for that record. If a counter was over a set number, then the email was deleted from the queue.

Answer 6

The Zend Mail class looks good, and simple to use, it also allows you to send a plain text and HTML version of the email, which in Email marketing is very important.

If your familiar with the frame work I'd stick with it.

Important things to consider when sending emails to large volumes of people are:

• Can your webserver cope with the image requests when e-mails are opened + the load on the server of people visiting your site.

If the answer is no or your unsure, using apache benchmark should be able to help you work out if it can. If your still unsure, it's always best to batch send emails (which can be timed with crontab) to spread the load.

I hope this helps.