IMAP4 NOTIFY addon for Thunderbird!

Jesús Cea Avión

2014-10-11 14:42

Warning: I publish this addon as is. It is a Proof Of Concept. It works well enough for my needs, and I don't have the time neither the Javascript and Thunderbird internals expertise to create a real addon usable by regular users.

Sorry for that. I already invested too much time on this. I am a consultant, btw, so you could just pay me :-).

This code is BSD licensed.

Background

When you get a lot of email you need filtering. Filtering allows you to classify incoming email in folders, tag it, flag it, delete it, etc. automatically.

Thunderbird can do client-side filtering easily. And you can do it both for POP3 and IMAP4 servers. But one of the real advantages of IMAP4 is that you can access the same folders, concurrently, from different devices. If you access your email from your smartphone but your filtering is done by Thunderbird running on your laptop, you will have a mess in your inbox when you are in the road and Thunderbird is not running.

Moving the filters to the server is very convenient: the filters are execute for every incoming email, even if all your devices are offline. Your mailboxes are tidy and ready to be accessed anytime by any device.

The standard for server-side filtering is called Sieve.

There is an issue, though. With Sieve you can get new email in any of your email folders (mailboxes, in IMAP4 speak), not only in inbox. Standard IMAP4 only notify changes in folders you have selected, and you can only select a folder per connection. Even if you email client opens ten simultaneous connections to the IMAP4 server you can only monitor ten folders. If you have twenty folders and all of them can get new email, you will miss updates.

You can see this clearly in Thunderbird. You see that a particular folder has seven unread emails. You select that folder in the GUI and now the counters are updated and you actually have nine unread messages. You don't get the update until you select that folder.

What can be done

This problem is a limitation of standard IMAP4. Thunderbird has two ways to cope with it:

You can click on a folder, right-click on it and select Properties. There you can enable "When getting new messsages for this account, always check this folder".

Now when Thunderbird checks for new email (for default, every ten minutes) it will poll this folder too.

There are a few problems here:
1. You have to do it for every folder you are interested in.
2. If you enable this for many folders it can affect server performance.
3. Thunderbird IMAP4 implementation is synchronous, so it will poll each folder in sequence and will wait for a reply before moving to the next folder. Than can be an issue if you are polling many folders.
4. Internet traffic increases, even if there is no new email. This can be important for mobile devices paying per byte. Battery life is impacted too.
5. You have a delay of up to ten minutes (by default) to know that you have new email. You can reduce this to one minute but that impact performance both in the server and in Thunderbird.
The need to enable monitoring of each folder manually can be eliminated as described here. When you enable that preference Thunderbird will poll all mail folders everytime it checks for new email.

Interestingly, Thunderbird automatically do this when you are connecting to a GMail account.

Anyway, polling taxes server performance, imposes a delay and increases data traffic. There is no other option under standard IMAP4.

Nevertheless there is an optional IMAP4 extension just perfect for this: NOTIFY.

IMAP4 NOTIFY extension

When we enable NOTIFY extension on a IMAP4 connection we will receive notifications of new emails, flag changes (for example, "message read") and deleted emails for every folder. In real time [1]. Using only a connection that you can share with regular IMAP4 activity. No extra data traffic because of polling, just real notificacions when there is something to be notified.

[1]	Your IMAP4 server could impose a delay of a few seconds for performance and bundling reasons.

NOTIFY is optional and you need an IMAP4 server that supports it. Recent Dovecot versions give you NOTIFY, and depending of your OS and your storage backend it could have no impact in performance at all.

The problem is that Thunderbird doesn't support NOTIFY.

Thunderbird and NOTIFY

Thunderbird evolution has basically stopped. It is a not prioritary Mozilla project anymore. Webmail is good enough for most users and many people doesn't apreciate the advantages of downloading email to the safety of your own computer, the convenience of being able to work offline and the speed of local operations.

Bringing NOTIFY support in Thunderbird is an open issue with no delivery date:

Bug 479133 - Add support for IMAP NOTIFY extension rfc rfc5465.
Bug 701325 - Monitoring multiple folder using IMAP IDLE.
Bug 716343 - Tabs don't IDLE, nor refresh with F5.

Thunderbird internals are messy and badly documented. It is C++ and you need a big computer with tons of memory (and quite a few minutes) to compile Thunderbird yourself. You can try to patch Thunderbird yourself (good luck) but the deal is to get the patch into Thunderbird itself. Thunderbird core developers don't think this issue is a priority.

Thunderbird addon

Addons are a good way of extending a software to fulfill your needs without the core product implementing all the non overlaping requirements of different parties. Mozilla products, Thunderbird in particular, support addons.

Could we use that feature to implement NOTIFY in Thunderbird?.

Yes, you can. Sort of. The proof is extension-notify-thunderbird@jcea.es. A hybrid Javascript + Python extension.

Download version 0.0.14.

There are dragons out there

This extension is a Proof Of Concept. It works very well in my personal setup but you will have to modify the code yourself to suit you and it will probably not work in your setup without additional surgery. Sorry, but my Javascript skills are non-existant and current code is good enough for me. Would be nice if somebody would pay me to improve this addon or somebody else uses it as a foundation for a product ready for regular users.

I am publishing the code because I know that some people need NOTIFY and Thunderbird is not delivering. If we are lucky maybe soon we have native NOTIFY support in Thunderbird. Not holding my breath though.

Now the bad news:

All account configuration is hardcoded in the code. You need to modify both Javascript and Python components to accomodate your configuration.

Would be easy to enumerate email accounts, identify IMAP4 servers and monitor them if they support NOTIFY. Easy, but work for somebody else.
Connection to the IMAP4 server is done in plain text mode. No encryption. It is not an issue for me because my IMAP4 server in running in my own laptop.

Would be easy to implement TLS and verify X.509 certificates.
Your username and password will be store in plaintext.

I don't know if an addon can access account access credentials. something to investigate.
It uses IDLE and the fact that stock Dovecot doesn't timeout connections in IDLE state, even after hours.

IDLE can be done optional simply sending a NOOP every few seconds. Even using IDLE, connection should exit and reenter IDLE state every 25 minutos.
I don't do any kind of error control. In fact I simply ignore any reply not related to a status update.

We should interpreted command replies and act on them.
If the IMAP4 connection is severed, the addon will reconnect in a few seconds. But changes happened during that time will be lost.

I am enabling IMAP4 CONDSTORE and QRESYNC extensions with the idea of detecting changes even after a long disconnection, but it is not implemented yet. My IMAP4 connection is local (127.0.0.1), so this is a rare event.

The good news is that your IMAP4 server doesn't need to implement CONDSTORE and QRESYNC extensions. I will try to enable them and it will fail in your case. But I am not using this functionalities so far and I am ignoring errors, so...
You need Python 3 installed.

Show me the code

Let's analize the code.

First stop, install.rdf:

 <?xml version="1.0"?>

 <RDF xmlns="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
      xmlns:em="http://www.mozilla.org/2004/em-rdf#">

   <Description about="urn:mozilla:install-manifest">
     <em:id>extension-notify-thunderbird@jcea.es</em:id>
     <em:name>NOTIFY IMAP</em:name>
     <em:version>0.0.14</em:version>
     <em:creator>jcea@jcea.es</em:creator>
     <em:unpack>true</em:unpack>

     <em:bootstrap>true</em:bootstrap>

     <em:targetApplication>
       <Description>
         <em:id>{3550f703-e582-4d05-9a08-453d09bdfdc6}</em:id>
         <em:minVersion>31.0</em:minVersion>
         <em:maxVersion>31.*</em:maxVersion>
       </Description>
     </em:targetApplication>

   </Description>
 </RDF>

As declared in lines 18-19, this addon is supported only in Thunderbird 31. When a new Thunderbird is available I will update this restriction.

In line 13 we declare this addon to be bootstrapped. That means that it can be installed, uninstalled, disabled, enabled and updated without restarting Thunderbird. Cool. It is nice to be able to update it in fast iterations during development without breaking my regular email workflow. Restarting Thunderbird every five minutes would be really annoying.

Line 11 tells Thunderbird to install this extension unpacked. This is needed in order to be able to locate and launch the Python component.

Main Javascript resides in bootstrap.js:

 const {classes: Cc, interfaces: Ci, utils: Cu} = Components;

 /*
 * https://developer.mozilla.org/en-US/docs/Mozilla/JavaScript_code_modules
 */
 Components.utils.import("resource://gre/modules/Timer.jsm");
 Components.utils.import("resource://gre/modules/Services.jsm");
 Components.utils.import("resource://gre/modules/osfile.jsm");
 Components.utils.import("resource://gre/modules/NetUtil.jsm");

 var incomingServer = null;
 var rootFolder = null;
 var timeoutID = null;
 var process = null;
 var execPath = null;
 var notifyPath = null;
 var notify = null; // null = nada, -1 = conectando, OTRO = notify

 function launch_process() {
     process = Components.classes["@mozilla.org/process/util;1"]
         .createInstance(Components.interfaces.nsIProcess);
     process.init(execPath);
     var args = [];
     process.run(false, args, args.length);
 }

 // OJO, esta función no se dispara hasta que se ha cargado
 // todo el fichero en memoria. Como es una PIPE, no se
 // lanza nunca hasta que matamos el proceso python externo,
 // cerrando la PIPE.
 function connect_notify_callback(inputStream, status) {
     try {
         // Cancelado
         if (notify == null) {
             return;
         }

         if (!Components.isSuccessCode(status)) {
             notify = null;
             return;
         }
         var buf = NetUtil.readInputStreamToString(inputStream, inputStream.available(), {charset:"utf-8"}).split("\n");
         for (var i=0; i < buf.length; i++) {
             var folder = buf[i];
             if (folder == "")
                 continue; // El \n final, si lo hay
             folder = incomingServer.getMsgFolderFromURI(rootFolder, folder);
             Services.console.logStringMessage(folder.prettiestName+" "+folder.URI);

             // Costly, and running in the main thread.
             // Would be wonderful if we could do this in a worker thread
             // or update status directly, without fetching headers.
             // Check methods "updateSummaryTotals()" and "summaryChanged()".
             folder.updateFolder(null);
         }

         // No intentamos abrir el fichero hasta que lo hemos
         // borrado, para evitar procesar un mismo fichero más de una vez.
         OS.File.remove(notifyPath.path).then(
             function onSuccess() {
                 notify = null;
             },
             function onFailure(reason) {
                 notify = null;
                 throw reason;
             }
         );
     } catch (err) {
         notify = null;
         throw reason;
     }
 }

 function connect_notify() {
     notify = -1;
     NetUtil.asyncFetch(notifyPath, connect_notify_callback);
 }

 function ejecuta2() {
     try {
         // https://developer.mozilla.org/en-US/docs/Mozilla/Thunderbird/Account_examples
         var acctMgr = Components.classes["@mozilla.org/messenger/account-manager;1"].getService(Components.interfaces.nsIMsgAccountManager);
         incomingServer = acctMgr.FindServer("jcea", "127.0.0.1", "imap");
         rootFolder = incomingServer.rootFolder;
     } catch(err) {
         incomingServer = null;
         rootFolder = null;
         notify = null;
         throw err;
     }

     if ((process==null) || (!process.isRunning)) {
         launch_process();
         notify = null;
     }
     if (notify==null) {
         connect_notify();
     }
     if ((notify==null) || (notify==-1))  // notify==null si error en el callback
         return;

 }

 function ejecuta() {
     t = Date.now();
     timeoutID = null;  // RACE CONDITION con SHUTDOWN
     timeoutID = setTimeout(ejecuta, 10000);

     ejecuta2();

     t = Date.now()-t ;
     t = t.toString();
     //Services.console.logStringMessage("DONE "+t+"ms");
 }

 function startup(data, reason) {
     execPath = data.installPath.clone();
     execPath.append("notify.py");
     notifyPath = data.installPath.clone();
     notifyPath.append("notify.info");
     timeoutID = setTimeout(ejecuta, 10000);
 }

 function shutdown(data, reason) {
     notify = null;
     if(timeoutID != null)
         clearTimeout(timeoutID);
     if(process != null)
         process.kill()
 }

 function install(data, reason) {}

 function uninstall(data, reason) {}

When the addon starts (lines 116-121) it will locate the Python component and the notification file. Then it will execute mainloop every ten seconds.

Shutdown is easy (lines 124-130): just cancel pending activity, kill Python component and shutdown gracefully.

Lines 104-114 are a wrapper for profiling and instrumentation. If you enable lines 113 you can see periodic activity and profiling in the Thunderbird error console.

Mainloop is lines 79-102. It is not really a loop, it is executed by Thunderbird every ten seconds.

First step: we pin details of your IMAP4 account. Note the hardcoded values. If something goes wrong it will throw an exception and will retry again in ten seconds. This is important because Thunderbird will start the addon before the account manager is ready. We just insists (every ten seconds) until we succeed.

If everything is ok, we get details of your IMAP4 account. We store them for future use.

Next step is to be sure that the Python component is running. If not, launch it.

Finally we deal with intercomunication (lines 92-100). We can be in three states: idling, reading notifications and processing notifications (comment in line 17).

if we are idling, just check the existence of the notification file every ten seconds (lines 74-77). If we are cancelling (shutting down) or the file doesn't exist or reading fails, just go to sleep again and will try later (lines 33-40). Notice that NetUtil.asyncFetch will not fire until an error happened or the entire file is loaded in RAM. Too bad you can't use it for reading from a FIFO. The good thing is that this operation will be done in background, without stopping the mainloop.

If we were able load the notification time in RAM we parse it and call folder updateFolder() method (lines 42-55). That method schedules a background refresh of that particular folder. That is far more expensive that doing a simple IMAP4 STATUS but looks like I can't access that functionality from Javascript. In my laptop this function requires 35ms. The good thing is that updateFolder() will update the Thunderbird cache for that folder, so opening it in the GUI will be instantaneous. But I rather use IMAP4 STATUS if I could.

After requesting updates for all notified folders we must delete the notification file (lines 59-66). Beware race conditions. Whatever happens we go back to the idle state.

That is.

I tried to keep Javascript code as simple as possible. I don't actually feel comfortable with Javascript so I want to keep all the intelligence and hard work in the Python component.

There are things I don't like. Method updateFolder() is overkill for my needs but I don't know any other way. Also we will request updates of folders we are currently seeing in the Thunderbird GUI and that is not needed because Thunderbird takes care of that itself, but I don't know how to know if a given folder is opened in the GUI or not.

We are doing more work that needed, but you would need to get quite a few messages per minute in order to notice some kind of slugginess.

Something you could find strange is the use of a notification file. First I tried to use a FIFO or named pipe, but reading from it asynchonously was really painful. The resulting code worked but it was huge and quite magical. Being a Python guy I value simplicity a lot and I was being too clever for my own good. I keep that code in my private Mercurial repository, but I went back a few revisions and opened a new branch with the code you see now. Far more easy to follow. But yes, the notification file is a strange artifact.

It works this way:

Javascript side: just wait until you see the notification file. When you do, read it, process it and delete it. Repeat.
Python side: process IMAP4 traffic and record notified folders internally. If you have anything to notify, check if the notification file exists. It it does, just try later. In the meantime keep recording information. When the notification file is not there anymore, create a temporal file with everything pending. Be sure the file is safe on the harddisk (sync) and then rename it to become the new notification file. Repeat.

This way the Javascript part always sees the complete notification file, never a partially written version. Even if the computer crashes, full harddisk, etc. This is critical because a malformed notification file would block the communication forever: Javascript component crashes and never deletes the file, and Python component never send new notifications because the old notification file is still there. I really make sure this never happens, no matter what.

There are no performance problems because sync writes, since we write a new notification file only once every ten seconds, independently of the notification activity. The only difference would be the size of that file.

Kind of a hack. Kind clever too, I presume. If you have a lot of notifications they are bundled together (duplicates dropped) and delivered in bulk every ten secons.

The final piece of the puzzle is the Python component notify.py:

 #!/usr/bin/env python3

 import os
 import time
 import threading
 import socket

 nombre = os.path.join(os.path.dirname(os.path.abspath(__file__)),'notify.info')

 buzones_notificados = set()

 PRINTABLE = set(range(0x20, 0x26)) | set(range(0x27, 0x7f))

 def modified_utf7(s):
     # encode to utf-7: '\xff' => b'+AP8-', decode from latin-1 => '+AP8-'
     s_utf7 = s.encode('utf-7').decode('latin-1')
     return s_utf7[1:-1].replace('/', ',')

 def encode(s):
     """Encode a folder name using IMAP modified UTF-7 encoding.

     Despite the function's name, the output is still a unicode string.
     """

     r = []
     _in = []

     def extend_result_if_chars_buffered():
         if _in:
             r.extend(['&', modified_utf7(''.join(_in)), '-'])
             del _in[:]

     for c in s:
         if ord(c) in PRINTABLE:
             extend_result_if_chars_buffered()
             r.append(c)
         elif c == '&':
             extend_result_if_chars_buffered()
             r.append('&-')
         else:
             _in.append(c)

     extend_result_if_chars_buffered()

     return ''.join(r)

 def notify() :
     s=socket.socket()
     s.connect(('127.0.0.1', 143))
     s.send(b'a login a a\n')
     s.send(b'b enable condstore qresync\n')
     s.send(b'c notify set (personal (messagenew messageexpunge flagchange))\n')
     s.send(b'd idle\n')

     buf = b''
     while True :
         buf += s.recv(9999).replace(b'\r', b'')
         #print(repr(buf))
         while True :
             p = buf.find(b')\n')
             if p == -1 :
                 p = buf.rfind(b'\n')
                 if p != -1 :
                     buf = buf[p+1:]
                 break

             fragmento, buf = buf[:p], buf[p+2:]
             p = fragmento.rfind(b' (')
             buzon = fragmento[:p]
             p = buzon.rfind(b'\n')
             if p != -1 :
                 buzon = buzon[p+1:]
             if buzon[0:1] == b'*' :  # Slicing debido a que son bytes
                 buzon = buzon[len(b'* STATUS '):]
             if buzon[0:1] == b'"' :
                 buzon = buzon[1:-1]
             buzon = buzon.decode('utf-8')
             if buzon not in ['INBOX', 'Trash'] :
                 buzones_notificados.add(buzon)
                 #print(buzon)

 def notify_thread() :
     while True :
         try :
             notify()
         except :
             raise
             time.sleep(10)

 def main() :
     t = threading.Thread(target=notify_thread)
     t.setDaemon(True)
     t.start()

     ppid = os.getppid()
     # Si muere el padre, nos suicidamos, pero
     # tenemos una RACE CONDITION, donde el padre
     # ha muerto muy rápido. Ojo, requiere INIT = 1,
     # cosa que no se cumple con las zonas Solaris.
     ppid = ppid if ppid != 1 else -999

     valores = [
             'imap://jcea@127.0.0.1/atari/Hatari',
             'imap://jcea@127.0.0.1/atari/coldfire',
             ]

     while ppid == os.getppid() :
         if (not len(buzones_notificados)) or os.path.exists(nombre) :
             time.sleep(1)
             continue
         f = open(nombre+'.new', 'w')
         for i in buzones_notificados.copy() :
             buzones_notificados.remove(i)
             f.write('imap://jcea@127.0.0.1/'+encode(i)+'\n')
         f.flush()
         os.fsync(f.fileno())
         f.close()
         os.rename(nombre+'.new', nombre)
         valores = valores[1:]+[valores[0]]

 if __name__ == '__main__' :
     main()

We get the patch of the notification file (line 8).

We will keep pending notification in a Python set, getting duplicate deletion for free.

Lines 12-45 contains a mUTF-7 (IMAP4 standard) to UTF-8 (rest of the -sane- world) transcoder. Just ignore that ugly detail.

Lines 91-93 launch a thread to take care of IMAP4 communication.

Javascript code makes sure that Python program dies correctly when the addon is shutdown (line 129 of the Javascript code). But lines 95-100 and 107 in the Python code makes sure that this program dies even if the Thunderbird process crashes unexpectly.

Main thread enters a loop. If there is nothing to notify or the notification file is still there, wait for a while a repeat (lines 108-110).

If we have something to notify and the notification file is not there, we create a temporary file, write the pending notifications on it, make sure the temporal file is in the disk (sync, lines 115-117) and then rename the file to become the new notification file to be read by the Javascript component.

Lines 102-105, and 119... just forget it. It is dead code from a previous version, to be deleted :-).

The IMAP4 thread starts at line 47. We connect to the IMAP4 server (note the hardcoded values) and send several commands. We don't care about the result, we assume everything is correct. Then we read from the socket and parse STATUS lines by hand, taking care of IMAP4 literals (Dovecot sends folder names as IMAP4 literals if they are not ASCII, encoded as UTF-8). Each notification is added to the pending set, to be notified by the main thread. If this thread dies for whatever reason (for instance, the IMAP4 connection is severed) it will be catched in lines 83-88 and restarted after a short delay.

Note that I am ignoring changes in the Inbox folder (line 78), because Thunderbird will be watching it already and you probably already have it opened in the GUI. I ignore Trash folder too just because I think it is the right thing to do. Both ignores are arbitrary and your mileage could be different. Performance is not an issue.

This code is not perfect, but it is good enough for my current needs. Just be aware of the caveats described in section There are dragons out there.

Code can be greatly improved with minimal effort. Some day, maybe.

If you are interested in building something really usable by regular users, please email me.

Is that working?

Open the Thunderbird error console and verify that Thunderbird is being notified when something happens in a folder different of Inbox. New email, deletions, marking messages as read, etc.

Then buy me a drink.

Updates

20181213 Update: Compatibility with Thunderbird 60 in IMAP4 NOTIFY addon for Thunderbird! (Thunderbird 60 compatible!).