Sorry for that. I already invested too much time on this. I am a consultant, btw, so you could just pay me :-).
This code is BSD licensed.
When you get a lot of email you need filtering. Filtering allows you to classify incoming email in folders, tag it, flag it, delete it, etc. automatically.
Thunderbird can do client-side filtering easily. And you can do it both for POP3 and IMAP4 servers. But one of the real advantages of IMAP4 is that you can access the same folders, concurrently, from different devices. If you access your email from your smartphone but your filtering is done by Thunderbird running on your laptop, you will have a mess in your inbox when you are in the road and Thunderbird is not running.
Moving the filters to the server is very convenient: the filters are execute for every incoming email, even if all your devices are offline. Your mailboxes are tidy and ready to be accessed anytime by any device.
The standard for server-side filtering is called Sieve.
There is an issue, though. With Sieve you can get new email in any of your email folders (mailboxes, in IMAP4 speak), not only in inbox. Standard IMAP4 only notify changes in folders you have selected, and you can only select a folder per connection. Even if you email client opens ten simultaneous connections to the IMAP4 server you can only monitor ten folders. If you have twenty folders and all of them can get new email, you will miss updates.
You can see this clearly in Thunderbird. You see that a particular folder has seven unread emails. You select that folder in the GUI and now the counters are updated and you actually have nine unread messages. You don't get the update until you select that folder.
What can be done
You can click on a folder, right-click on it and select Properties. There you can enable "When getting new messsages for this account, always check this folder".
Now when Thunderbird checks for new email (for default, every ten minutes) it will poll this folder too.
There are a few problems here:
- You have to do it for every folder you are interested in.
- If you enable this for many folders it can affect server performance.
- Thunderbird IMAP4 implementation is synchronous, so it will poll each folder in sequence and will wait for a reply before moving to the next folder. Than can be an issue if you are polling many folders.
- Internet traffic increases, even if there is no new email. This can be important for mobile devices paying per byte. Battery life is impacted too.
- You have a delay of up to ten minutes (by default) to know that you have new email. You can reduce this to one minute but that impact performance both in the server and in Thunderbird.
Anyway, polling taxes server performance, imposes a delay and increases data traffic. There is no other option under standard IMAP4.
IMAP4 NOTIFY extension
When we enable NOTIFY extension on a IMAP4 connection we will receive notifications of new emails, flag changes (for example, "message read") and deleted emails for every folder. In real time . Using only a connection that you can share with regular IMAP4 activity. No extra data traffic because of polling, just real notificacions when there is something to be notified.
|||Your IMAP4 server could impose a delay of a few seconds for performance and bundling reasons.|
NOTIFY is optional and you need an IMAP4 server that supports it. Recent Dovecot versions give you NOTIFY, and depending of your OS and your storage backend it could have no impact in performance at all.
Thunderbird and NOTIFY
Thunderbird evolution has basically stopped. It is a not prioritary Mozilla project anymore. Webmail is good enough for most users and many people doesn't apreciate the advantages of downloading email to the safety of your own computer, the convenience of being able to work offline and the speed of local operations.
- Bug 479133 - Add support for IMAP NOTIFY extension rfc rfc5465.
- Bug 701325 - Monitoring multiple folder using IMAP IDLE.
- Bug 716343 - Tabs don't IDLE, nor refresh with F5.
Thunderbird internals are messy and badly documented. It is C++ and you need a big computer with tons of memory (and quite a few minutes) to compile Thunderbird yourself. You can try to patch Thunderbird yourself (good luck) but the deal is to get the patch into Thunderbird itself. Thunderbird core developers don't think this issue is a priority.
Addons are a good way of extending a software to fulfill your needs without the core product implementing all the non overlaping requirements of different parties. Mozilla products, Thunderbird in particular, support addons.
There are dragons out there
I am publishing the code because I know that some people need NOTIFY and Thunderbird is not delivering. If we are lucky maybe soon we have native NOTIFY support in Thunderbird. Not holding my breath though.
Now the bad news:
Your username and password will be store in plaintext.
I don't know if an addon can access account access credentials. something to investigate.
I don't do any kind of error control. In fact I simply ignore any reply not related to a status update.
We should interpreted command replies and act on them.
I am enabling IMAP4 CONDSTORE and QRESYNC extensions with the idea of detecting changes even after a long disconnection, but it is not implemented yet. My IMAP4 connection is local (127.0.0.1), so this is a rare event.
The good news is that your IMAP4 server doesn't need to implement CONDSTORE and QRESYNC extensions. I will try to enable them and it will fail in your case. But I am not using this functionalities so far and I am ignoring errors, so...
You need Python 3 installed.
Show me the code
Let's analize the code.
First stop, install.rdf:
In line 13 we declare this addon to be bootstrapped. That means that it can be installed, uninstalled, disabled, enabled and updated without restarting Thunderbird. Cool. It is nice to be able to update it in fast iterations during development without breaking my regular email workflow. Restarting Thunderbird every five minutes would be really annoying.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134
Shutdown is easy (lines 124-130): just cancel pending activity, kill Python component and shutdown gracefully.
Lines 104-114 are a wrapper for profiling and instrumentation. If you enable lines 113 you can see periodic activity and profiling in the Thunderbird error console.
Mainloop is lines 79-102. It is not really a loop, it is executed by Thunderbird every ten seconds.
First step: we pin details of your IMAP4 account. Note the hardcoded values. If something goes wrong it will throw an exception and will retry again in ten seconds. This is important because Thunderbird will start the addon before the account manager is ready. We just insists (every ten seconds) until we succeed.
If everything is ok, we get details of your IMAP4 account. We store them for future use.
Next step is to be sure that the Python component is running. If not, launch it.
Finally we deal with intercomunication (lines 92-100). We can be in three states: idling, reading notifications and processing notifications (comment in line 17).
if we are idling, just check the existence of the notification file every ten seconds (lines 74-77). If we are cancelling (shutting down) or the file doesn't exist or reading fails, just go to sleep again and will try later (lines 33-40). Notice that NetUtil.asyncFetch will not fire until an error happened or the entire file is loaded in RAM. Too bad you can't use it for reading from a FIFO. The good thing is that this operation will be done in background, without stopping the mainloop.
After requesting updates for all notified folders we must delete the notification file (lines 59-66). Beware race conditions. Whatever happens we go back to the idle state.
There are things I don't like. Method updateFolder() is overkill for my needs but I don't know any other way. Also we will request updates of folders we are currently seeing in the Thunderbird GUI and that is not needed because Thunderbird takes care of that itself, but I don't know how to know if a given folder is opened in the GUI or not.
We are doing more work that needed, but you would need to get quite a few messages per minute in order to notice some kind of slugginess.
Something you could find strange is the use of a notification file. First I tried to use a FIFO or named pipe, but reading from it asynchonously was really painful. The resulting code worked but it was huge and quite magical. Being a Python guy I value simplicity a lot and I was being too clever for my own good. I keep that code in my private Mercurial repository, but I went back a few revisions and opened a new branch with the code you see now. Far more easy to follow. But yes, the notification file is a strange artifact.
It works this way:
Python side: process IMAP4 traffic and record notified folders internally. If you have anything to notify, check if the notification file exists. It it does, just try later. In the meantime keep recording information. When the notification file is not there anymore, create a temporal file with everything pending. Be sure the file is safe on the harddisk (sync) and then rename it to become the new notification file. Repeat.
There are no performance problems because sync writes, since we write a new notification file only once every ten seconds, independently of the notification activity. The only difference would be the size of that file.
Kind of a hack. Kind clever too, I presume. If you have a lot of notifications they are bundled together (duplicates dropped) and delivered in bulk every ten secons.
The final piece of the puzzle is the Python component notify.py:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122
#!/usr/bin/env python3 import os import time import threading import socket nombre = os.path.join(os.path.dirname(os.path.abspath(__file__)),'notify.info') buzones_notificados = set() PRINTABLE = set(range(0x20, 0x26)) | set(range(0x27, 0x7f)) def modified_utf7(s): # encode to utf-7: '\xff' => b'+AP8-', decode from latin-1 => '+AP8-' s_utf7 = s.encode('utf-7').decode('latin-1') return s_utf7[1:-1].replace('/', ',') def encode(s): """Encode a folder name using IMAP modified UTF-7 encoding. Despite the function's name, the output is still a unicode string. """ r =  _in =  def extend_result_if_chars_buffered(): if _in: r.extend(['&', modified_utf7(''.join(_in)), '-']) del _in[:] for c in s: if ord(c) in PRINTABLE: extend_result_if_chars_buffered() r.append(c) elif c == '&': extend_result_if_chars_buffered() r.append('&-') else: _in.append(c) extend_result_if_chars_buffered() return ''.join(r) def notify() : s=socket.socket() s.connect(('127.0.0.1', 143)) s.send(b'a login a a\n') s.send(b'b enable condstore qresync\n') s.send(b'c notify set (personal (messagenew messageexpunge flagchange))\n') s.send(b'd idle\n') buf = b'' while True : buf += s.recv(9999).replace(b'\r', b'') #print(repr(buf)) while True : p = buf.find(b')\n') if p == -1 : p = buf.rfind(b'\n') if p != -1 : buf = buf[p+1:] break fragmento, buf = buf[:p], buf[p+2:] p = fragmento.rfind(b' (') buzon = fragmento[:p] p = buzon.rfind(b'\n') if p != -1 : buzon = buzon[p+1:] if buzon[0:1] == b'*' : # Slicing debido a que son bytes buzon = buzon[len(b'* STATUS '):] if buzon[0:1] == b'"' : buzon = buzon[1:-1] buzon = buzon.decode('utf-8') if buzon not in ['INBOX', 'Trash'] : buzones_notificados.add(buzon) #print(buzon) def notify_thread() : while True : try : notify() except : raise time.sleep(10) def main() : t = threading.Thread(target=notify_thread) t.setDaemon(True) t.start() ppid = os.getppid() # Si muere el padre, nos suicidamos, pero # tenemos una RACE CONDITION, donde el padre # ha muerto muy rápido. Ojo, requiere INIT = 1, # cosa que no se cumple con las zonas Solaris. ppid = ppid if ppid != 1 else -999 valores = [ 'imap://email@example.com/atari/Hatari', 'imap://firstname.lastname@example.org/atari/coldfire', ] while ppid == os.getppid() : if (not len(buzones_notificados)) or os.path.exists(nombre) : time.sleep(1) continue f = open(nombre+'.new', 'w') for i in buzones_notificados.copy() : buzones_notificados.remove(i) f.write('imap://email@example.com/'+encode(i)+'\n') f.flush() os.fsync(f.fileno()) f.close() os.rename(nombre+'.new', nombre) valores = valores[1:]+[valores] if __name__ == '__main__' : main()
We get the patch of the notification file (line 8).
We will keep pending notification in a Python set, getting duplicate deletion for free.
Lines 91-93 launch a thread to take care of IMAP4 communication.
Main thread enters a loop. If there is nothing to notify or the notification file is still there, wait for a while a repeat (lines 108-110).
Lines 102-105, and 119... just forget it. It is dead code from a previous version, to be deleted :-).
The IMAP4 thread starts at line 47. We connect to the IMAP4 server (note the hardcoded values) and send several commands. We don't care about the result, we assume everything is correct. Then we read from the socket and parse STATUS lines by hand, taking care of IMAP4 literals (Dovecot sends folder names as IMAP4 literals if they are not ASCII, encoded as UTF-8). Each notification is added to the pending set, to be notified by the main thread. If this thread dies for whatever reason (for instance, the IMAP4 connection is severed) it will be catched in lines 83-88 and restarted after a short delay.
Note that I am ignoring changes in the Inbox folder (line 78), because Thunderbird will be watching it already and you probably already have it opened in the GUI. I ignore Trash folder too just because I think it is the right thing to do. Both ignores are arbitrary and your mileage could be different. Performance is not an issue.
This code is not perfect, but it is good enough for my current needs. Just be aware of the caveats described in section There are dragons out there.
Code can be greatly improved with minimal effort. Some day, maybe.
If you are interested in building something really usable by regular users, please email me.
Is that working?
Then buy me a drink.
20181213 Update: Compatibility with Thunderbird 60 in IMAP4 NOTIFY addon for Thunderbird! (Thunderbird 60 compatible!).