Mailman MySQL Adaptor Implemented: Database Driven Open Source Email Lists
With a lot of help from someone on the Mailman Developers email list I was able to get Mailman to work with a MySQL database!
This is awesome because Mailman is the leading free mailing list choice. I would have switched to sympa, but it would have meant resetting our server - as cPanel does not work with sympa (as it cannot use the postfix program to handle mail). Mailman seems to have slightly better features than Sympa, and I'm hoping that when Mailman 3 comes out (in 2-3 years??? It is in alpha now), that we can switch to that.
Typically Mailman uses text files to track list members.
I did this for a Virtual Private Server running Centos, cPanel, and Mailman 2.1.13.
I mostly followed these instructions
But then I ran into a bunch of problems as Python was not able to import the MySQLdb module. The problem was that the module was installed in the wrong directory.
--------------------
Here is my solution:
--------------------
I found a solution!
Mark Sapiro deserves all the credit for helping me out a TON
=)
Here is my documentation in the hope that it will help someone in the future.
---------------------------
Mark writes:
The basic problem is that the python library or at least the one
containing the site-packages/ into which MySQLdb was installed is
/usr/lib64/python2.4 rather than /usr/lib/python2.4.
Mailman starts a lot of Python processes with the -S option which
bypasses importing the site module at startup and this is what
normally puts the path to site-packages in sys.path. Mailman attempts
to do this itself in its own paths module which everything imports,
but it assumes the path begins with /usr/lib/pythonv.v so it never got
/usr/lib64/python2.4/site-packages into sys.path.
The bit added to extend.py does this when it replaces the
MemberAdaptor, so MysqlMemberships can successfully import MySQLdb.
--------------------------
Aaron adds:
We changed extend.py to be:
import sys
if '/usr/lib64/python2.4/site-packages' not in sys.path:
sys.path.append('/usr/lib64/python2.4/site-packages')
from Mailman.MysqlMemberships import MysqlMemberships
def extend(list):
list._memberadaptor = MysqlMemberships(list)
--------------------------
To track down the fact that the module's directory was wrong we had to fix a bug in Mailman's error logging:
There is a bug in the error logging code in
/usr/local/cpanel/3rdparty/mailman/scripts/driver
Edit that file. Find
def print_environment(logfp=None):
if logfp is None:
logfp = sys.__stderr__
try:
import os
except ImportError:
os = None
# Write some information about our Python executable to the log
file.
print>> logfp, '[----- Python Information -----]'
print>> logfp, 'sys.version =', sys.version
print>> logfp, 'sys.executable =', sys.executable
print>> logfp, 'sys.prefix =', sys.prefix
print>> logfp, 'sys.exec_prefix =', sys.exec_prefix
print>> logfp, 'sys.path =', sys.exec_prefix
print>> logfp, 'sys.platform =', sys.platform
Change the next to last line of that from
print>> logfp, 'sys.path =', sys.exec_prefix
to
print>> logfp, 'sys.path =', sys.path
and then see what's reported for sys.path in the traceback.
---------------------------
Finally because my Cpanel installation of mailman was creating mailing lists that had "." in their name, and MySQL does not create tables with a period in the name I modified the code for MysqlMemberships.py
A global replace of
self.__mlist.internal_name()
with
self.__mlist.internal_name().replace('.', '_')
- Login to post comments
Exporting Data For Existing Lists
Exporting data from lists into MySQL was tricky.
Python combines the data about the subscribers with that about the list and stores them in a "pickle" file for each list.
You can get the information by running dumpdb, a mailman program.
I ran "python dumpdb config.pck > dump.txt" to dump the pickle into a text file.
Then to convert the text file into JSON, I had to delete all of the bounce information because it was in a format that wasn't compatible with JSON. A depickled file is text, so it is easy to delete this information.
I used the SimpleJSON python library.
I used this script (which works fine if you don't have any bounce data), but then commented out part of it and set
unpickled_data = [copied and pasted the string from dump.txt here]
Used a python script from: http://stackoverflow.com/questions/3040872/pythons-cpickle-deserialization-from-php
# pickle2json.py
import sys, optparse, cPickle, os
try:
import json
except:
import simplejson as json
# Setup the arguments this script can accept from the command line
parser = optparse.OptionParser()
parser.add_option('-p','--pickled_data_path',dest="pickled_data_path",type="string",help="Path to the file containing pickled data.")
parser.add_option('-j','--json_data_path',dest="json_data_path",type="string",help="Path to where the json data should be saved.")
opts,args=parser.parse_args()
# Load in the pickled data from either a file or the standard input stream
if opts.pickled_data_path:
unpickled_data = cPickle.loads(open(opts.pickled_data_path).read())
else:
unpickled_data = cPickle.loads(sys.stdin.read())
# Output the json version of the data either to another file or to the standard output
if opts.json_data_path:
open(opts.json_data_path, 'w').write(json.dumps(unpickled_data))
else:
print unpickled_data
On second thought
I do NOT recommend using this code or modifying mailman to use MySQL. Wait for Mailman 3.0 to come out.
I did these modifications, but our vps has been plagued with problems that crash mailman as a result of us not using the standardized cPanel installation. If you are running mailman on your own server, without cPanel or similar management software - you'll hopefully have better luck.