Using Mailman 2 in a machine without a mail server

Mailman is a mailing list software written in Python. It includes mailing list operation, management and a web interface.

Usually Mailman is running in a machine having a web server and a mail server. In this article I will describe how to operate a Mailman environment where the mail server resides in a different machine. Why? Because I already have a mail server running. Taking care of configuration, operation and spam filtering of an additional service is a burden.

There are two interfaces between Mailman 2 and the mail server:

1. Outgoing: Fortunately you can configure the mail server used for outgoing emails editing the configuration file mm_cfg.py and modifying the variable SMTPHOST.

   Problem solved.

2. Incoming: There is a small script from Mailman 2 that the mail server runs for each email received. This script injects the email in a Mailman 2 queue to be processed by the Mailman machinery.

   We need to replace that script to transport the emails to the Mailman 2 incoming queue running in a different machine.

For Postfix (my mail server) you usually have something like this in your aliases file:

hacking:                "|/home/mailman/mail/mailman post hacking"
hacking-admin:          "|/home/mailman/mail/mailman admin hacking"
hacking-bounces:        "|/home/mailman/mail/mailman bounces hacking"
hacking-confirm:        "|/home/mailman/mail/mailman confirm hacking"
hacking-join:           "|/home/mailman/mail/mailman join hacking"
hacking-leave:          "|/home/mailman/mail/mailman leave hacking"
hacking-owner:          "|/home/mailman/mail/mailman owner hacking"
hacking-request:        "|/home/mailman/mail/mailman request hacking"
hacking-subscribe:      "|/home/mailman/mail/mailman subscribe hacking"
hacking-unsubscribe:    "|/home/mailman/mail/mailman unsubscribe hacking"

This configuration will run /home/mailman/mail/mailman script (my Mailman 2 is installed in /home/mailman/) when a mail is received in those email addresses. This requires that Mailman 2 and Postfix are running in the same machine.

I am going to change that. I will use Pyro, a Python remote procedure call library. Why? Because it is easy to use, it requires no infraestructure and it is mostly Python transparent.

This is the code I use in the machine running my Postfix:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57

 #!/opt/local/bin/python3

 # (c) 2016 jcea@jcea.es - https://www.jcea.es/
 # Código liberado como Dominio Público.
 # Haz con él lo que quieras.

 import sys
 import io

 # Podríamos hacer una importación condicional, solo cuando hace falta, pero
 # cualquier ahorro es ridículo comparado con cargar Pyro4 y hacer el RPC.
 import posix
 import traceback

 def main():
     import Pyro4

     # Default serpent serializer is problematic. Check:
     # https://pyro4.readthedocs.io/en/stable/tipstricks.html#binary-data-transfer-file-transfer
     # https://pyro4.readthedocs.io/en/stable/clientcode.html#serialization
     Pyro4.config.SERIALIZER = 'marshal'

     mailman = Pyro4.Proxy('PYRO:mailman@10.200.0.101:37191')
     mailman._pyroTimeout = 60

     accion, lista = sys.argv[1], sys.argv[2]

     # Transparent (binary) encoding
     #input_stream = io.TextIOWrapper(sys.stdin.buffer, encoding='latin-1')
     #msg = input_stream.read().encode('latin-1')  # Bytes
     msg = sys.stdin.buffer.read()  # Bytes

     status, stdout, stderr = mailman.do(lista, accion, msg)

     sys.stdout.write(stdout)
     sys.stderr.write(stderr)

     return status

 if __name__ == '__main__':
     try:
         status = main()

     except Exception as e:
         traceback.print_exc()

         #import Pyro4
         #print("Pyro traceback:")
         #print("".join(Pyro4.util.getPyroTraceback()))

         # http://www.postfix.org/pipe.8.html
         sys.exit(posix.EX_TEMPFAIL)  # sysexits.h

     # XXX: ¿Los números de estado del cliente y del servidor PYRO son los
     # mismos? Si son el mismo sistema operativo, sí. Si no, no hay
     # garantías. ¿O sí? ¿Están estandarizados?
     sys.exit(status)

This code will be executed by Postfix when a new email arrives. Execution starts at line 40. The only thing we do is to call main(). If anything goes wrong we print some details and we indicate Postfix to try again soon (line 52). If everything was fine, we give to Postfix the return code returned by the remote enqueueing process (line 57).

The heart is routine main() (lines 15-38). Here we import Pyro (line 16) inside the routine because we want an import failure (Pyro not installed, for example) to be a temporal failure to be notified, not a catastrofic mail loss.

Interestingly, default Pyro serializer makes transporting binary data non trivial, so I use an alternative builtin marshal serializer (line 21). We indicate where the service is located (line 23) and the connection timeout (line 24).

Next step is to get the details of the Postfix request (line 26) and the content of the email (line 31).

Then we do the remote procedure call (line 33). If everything is fine, we send to stdout and stderr the content the remote enqueueing process generated and we return to the caller the return code. The idea here is that the RPC should be transparent and the sysadmin should feel that the enqueueing behaves and provides the same diagnostics that a local version.

This code will reside in /home/postfix/z-mailman.py.

Now we change Postfix aliases configuration file to this:

hacking:                "|/home/postfix/z-mailman.py post hacking"
hacking-admin:          "|/home/postfix/z-mailman.py admin hacking"
hacking-bounces:        "|/home/postfix/z-mailman.py bounces hacking"
hacking-confirm:        "|/home/postfix/z-mailman.py confirm hacking"
hacking-join:           "|/home/postfix/z-mailman.py join hacking"
hacking-leave:          "|/home/postfix/z-mailman.py leave hacking"
hacking-owner:          "|/home/postfix/z-mailman.py owner hacking"
hacking-request:        "|/home/postfix/z-mailman.py request hacking"
hacking-subscribe:      "|/home/postfix/z-mailman.py subscribe hacking"
hacking-unsubscribe:    "|/home/postfix/z-mailman.py unsubscribe hacking"

In the machine we have Mailman 2 we need to run the other end of the Pyro remote procedure call. The code is quite simple too:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

 #!/usr/local/bin/python3

 # (c) 2016 jcea@jcea.es - https://www.jcea.es/
 # Código liberado como Dominio Público.
 # Haz con él lo que quieras.

 import subprocess
 import Pyro4

 class AccessDenied(RuntimeError):
     pass

 @Pyro4.expose
 @Pyro4.behavior(instance_mode='single')
 class mailman:
     @staticmethod
     def do(lista, accion, mensaje):
         process = subprocess.run(['/home/mailman/mail/mailman', accion, lista],
                                  input=mensaje,
                                  stdout=subprocess.PIPE,
                                  stderr=subprocess.PIPE,
                                  timeout=45)
         return (process.returncode,
                 process.stdout.decode('latin-1'),
                 process.stderr.decode('latin-1'))

 class daemon(Pyro4.Daemon):
     def validateHandshake(self, conn, data):
         ip = conn.sock.getpeername()[0]
         if ip != '10.200.0.7':
             raise AccessDenied('No permitimos el acceso a "%s"' %ip)
         return super().validateHandshake(conn, data)

 #daemon = Pyro4.Daemon(host='10.200.0.101', port=37191)
 daemon = daemon(host='10.200.0.101', port=37191)
 uri = daemon.register(mailman, 'mailman', force=True)
 print('uri =', uri)
 daemon.requestLoop()

Code execution starts at line 35. We bind the network details (line 35), register the service we export (line 36) and just sit there processing requests (line 38).

The requests are processed in class mailman (lines 13-25). We basically just call the regular Mailman 2 enqueueing script (lines 18-22). Note the use of a timeout and that timeout is shorter than the request timeout used in the Postfix machine. Note also how we return stdout and stderr to the calling process. We are careful about transfering binary data.

Lines 27-32 are interesting because we are implementing an access control based on IP address. We don't want anybody to be able to inject arbitrary emails in our mailing lists. We could use a shared secret, but since the data travels in the open, it is pointless. [1]

This code in the Mailman 2 machine must be always running. I am using a SmartOS native zone (a Solaris derivative), so I declare a SMF manifest:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85

 <?xml version="1.0"?>
 <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1">
 <!--
     Copyright 2004 Sun Microsystems, Inc.  All rights reserved.
     Use is subject to license terms.

     ident       "@(#)http-apache2.xml   1.2     04/11/11 SMI"
 -->

 <service_bundle type='manifest' name='jcea:mailman-Pyro4'>

 <service
         name='jcea/mailman-Pyro4'
         type='service'
         version='1'>

         <!--
           Because we may have multiple instances of network/http
           provided by different implementations, we keep dependencies
           and methods within the instance.
         -->

         <instance name='default' enabled='false'>
                 <dependency name='loopback'
                     grouping='require_all'
                     restart_on='error'
                     type='service'>
                         <service_fmri value='svc:/network/loopback:default'/>
                 </dependency>

                 <dependency name='physical'
                     grouping='optional_all'
                     restart_on='error'
                     type='service'>
                         <service_fmri value='svc:/network/physical:default'/>
                 </dependency>

                 <dependency name='fs-local'
                         grouping='require_all'
                          restart_on='none'
                          type='service'>
                          <service_fmri
                                  value='svc:/system/filesystem/local' />
                 </dependency>

                 <exec_method
                         type='method'
                         name='start'
                         exec='/home/mailmanPyro4.py 2&gt;&amp;1 &amp;'
                         timeout_seconds='60'>
                             <method_context>
                                 <method_credential user='nobody' group='nobody' />
                                 <method_environment>
                                         <envvar name='PYTHONIOENCODING' value='UTF-8' />
                                         <envvar name='PYTHONUNBUFFERED' value='1' />
                                 </method_environment>
                             </method_context>
                 </exec_method>

                 <property_group name='startd' type='framework'>
                         <!-- sub-process core dumps shouldn't restart
                                 session -->
                         <propval name='ignore_error' type='astring'
                                 value='core,signal' />
                 </property_group>

         </instance>

         <stability value='Evolving' />

         <template>
                 <common_name>
                         <loctext xml:lang='C'>
                                 Pasarela Pyro4 a Mailman
                         </loctext>
                 </common_name>
                 <documentation>
                         <manpage title='XXXX' section='1M' />
                         <doc_link name='example.org'
                                 uri='http:/example.org' />
                 </documentation>
         </template>
 </service>

 </service_bundle>

To avoid that anybody running in the local machine can use Mailman 2 enqueueing script to inject arbitrary emails in the mailing lists, Mailman 2 enqueueing script verifies the username and group of the calling process. In my installation I have configured Mailman 2 to require an username and group of nobody.

I also disable Python buffering (line 55) and instruct Python to use UTF-8 (line 54) for Input/Output independently of the Operating System default encoding.

After importing this manifest, the Operating System will take care of launching the service at boot time, restarting it if it dies, logging, etc.:

[root@babylon5 home]# svcs mailman-Pyro4
STATE          STIME    FMRI
online         Oct_28   svc:/jcea/mailman-Pyro4:default
[root@babylon5 home]# svcs -l mailman-Pyro4
fmri         svc:/jcea/mailman-Pyro4:default
name         Pasarela Pyro4 a Mailman
enabled      true
state        online
next_state   none
state_time   Sat Oct 29 19:22:12 2016
logfile      /var/svc/log/jcea-mailman-Pyro4:default.log
restarter    svc:/system/svc/restarter:default
contract_id  142
dependency   require_all/error svc:/network/loopback:default (online)
dependency   optional_all/error svc:/network/physical:default (online)
dependency   require_all/none svc:/system/filesystem/local (online)