How to cause sent emails to appear threaded in GMail recipient's view with Message-ID, In-Reply-To and References

Question

I've read some great online resources like http://www.jwz.org/doc/threading.html, and it seems that any email is send with a Message-ID header, then any replies to it include In-Reply-To naming that ID and Refences which can name a list of parent message id's, and email clients use this information to construct threads when viewing a list of emails in threaded view.

My question is: Can a series of emails be sent to a recipient with faked headers, to make them appear in a thread without the recipient replying to them? If so, why does my attempt below not work?

We have a system that sends out several emails pertaining to a specific entity in our system. Lets say we sell widgets and email users several times about each widget. We want all emails for a specific widget ID to appear as an email thread in our user's email clients.

The trip here seems to be that normally emails are sent, then replied to. Our system simply wants to send out several emails, and fake the In-Reply-To and References headers, to trick email clients into displaying them in a tree.

The Message-ID format I'm using is: 'foobar' + widgetId + sequence

• widgetId = a number unique to each widget e.g. 1234
• sequence = a sequential number incremented each time we send an email

First email:

• Message-ID <foobar-1234-0@server.com>
• In-Reply-To: not provided
• References: not provided

Second email:

• Message-ID <foobar-1234-1@server.com>
• In-Reply-To: <foobar-1234-0@server.com>
• References: <foobar-1234-0@server.com>

Third email:

• Message-ID <foobar-1234-2@server.com>
• In-Reply-To: <foobar-1234-1@server.com>
• References: <foobar-1234-0@server.com> <foobar-1234-1@server.com>

(incidentally, including the @server.com portion of a message ID appears to be vital. Without that, using e.g. foobar-123-0, our SMTP server simply ignored it and used it own autogenerated message ID)

Emails appear correctly in Thunderbird, as a tree, but not in Gmail, they just get listed one after the other in the Inbox while other conversations are properly threaded right next to them. I'm not sure if I'm getting it wrong and Thunderbird is doing the best it can with bad data, or if Gmail needs some extra nonstandard sugar I'm not providing.

Here is my node.js test script:

/*jshint dojo:true */
/*global console:true */
'use strict';
var Q = require('q'),
    nconf = require('nconf'),
    optimist = require('optimist'),
    nodemailer = require('nodemailer');

console.log('Started to run.');
var argv = optimist.argv,
    config = nconf.argv().env().file('conf.json'),
    smtpConfig = config.get('smtp'),
        smtpTransport = nodemailer.createTransport('SMTP', {
            service: smtpConfig.service, // 'Gmail',
            auth: {
                user: smtpConfig.user, //'blah@gmail.com',
                pass: smtpConfig.pass //'xyz'
            }
        }),
    rand = Math.floor(Math.random() * 5000), // a random enough unique id
    messageIdPrefix = 'foobar-' + rand + '-';

var promises = [],
    references = '';

for (var i = 0 ; i < 3 ; i ++) {
    // Prepare email content
    var subject = 'This is test email ' + i,
        htmlMessage = '<h1>Am I threaded? Email ' + i + '</h1><p>???</p>',
        textMessage = 'Am I threaded? Email ' + i + '\n\n???';

    var recipients = 'recipient@server.com';

    // Each email in this sequence has a common prefix
    // In Reply To should be the single immediate parent message id
    // References should list all parents, top most first
    var messageId = messageIdPrefix + i + '@server.com',
        inReplyTo = (i > 0) ? ('<' + (messageIdPrefix + (i-1)) + '@server.com>') : false;

    // setup e-mail data with unicode symbols
    var mailOptions = {
        from: config.get('ourEmail'),
        to: recipients,
        subject: subject,
        text: textMessage,
        html: htmlMessage,
        messageId: messageId,
        inReplyTo: inReplyTo,
        references: references,
        headers: {
            // 'in-Reply-To': inReplyTo
        }
    };

    // send mail with defined transport object
    var q = Q.defer();
    promises.push(q.promise);
    smtpTransport.sendMail(mailOptions, function (error, response) {
        if (error) {
            console.error(error);
            q.reject('error');
        } else {
            console.log('Message sent: ' + response.message);
            q.resolve('yay!');
        }
    });

    // next time round loop, if any, includes this id in the references list
    references = (references ? (references + ' ') : '') + messageId;
}

Q.all(promises).then(function (results) {
    console.log('All done, closing mail connection: ', results);
    smtpTransport.close(); // shut down the connection pool, no more messages
});

Requires a conf file like:

{
    "ourEmail": "me@server.com",
    "smtp": {
        "service": "Gmail",
        "user": "me@server.com",
        "pass": "ilikecheese"
    }
}

For bonus points, please hint why my attempt to use Q.all doesn't seem to fire and the script does not exit cleanly, despite sending all emails correctly :)

Answer 1

The answer to why they are not threaded in Gmail is because Gmail's threading is done according to the subject of the messages (it is not based on the "in-reply-to" or "references" field in the header).

See the answers to this question on stackexchange for more details on how Gmail does threading: https://webapps.stackexchange.com/questions/965/how-does-gmail-decide-to-thread-email-messages..

The subjects in your case are "This is test email 1", "This is test email 2" and "This is test email 3" which will not cause threading by the rules Gmail use.