Worklog | Work, notes, and projects. Thursday, September 1, 2005
 
March 02, 2006

Pine and Perdition

Testing Pine with Perdition — I have my .pinerc stored on the IMAP server, and I simply replaced all the hostnames that point to KE (mailer.earlham.edu) with ones that point to mailproxy.earlham.edu (Perdition and Sendmail running on SIPALA).

It works flawlessly. I get my mail folders on KE, and I send mail through SIPALA.

Posted by Rowan Littell at 06:56 PM
February 24, 2006

Cyrus, Perdition, LDAP

I got Cyrus working again on Solaris, as per entries last spring. Not terribly difficult. The harder part was getting Perdition, the IMAP/POP3 proxy, working. There are some source code errors that made the build on Solaris tricky.

  • Don’t build Perdition with the daemon map option. It requires the mkdtemp(3) library call, which isn’t available on Solaris.
  • Don’t build Perdition static like the documentation tells you to for Solaris. It will always try to load a dynamic library, but static builds don’t provide dynamic libs.
  • There is an error in the LDAP initialization routines. #if 0 the code that calls ldap_initialize and just let it call ldap_init. The author said the fix would be in the next release sometime last fall, but no such release seems to be forthcoming.
  • Installation: Perdition is set to listen on the third network interface, which is set to mailproxy.earlham.edu, a different IP address from the first interface. That way the LDAP attribute can be the hostname of the local box without having to mess around with alternate ports for Cyrus. Cyrus listens on the bge0 interface, while Perdition listens on bge2 and then contacts bge0 if LDAP tells it to.

Once installed, there have to be individual Perdition processes for each protocol: IMAP, POP3, IMAPS, and POP3S. They each use different config files and PID files. Once up and running, it does work flawlessly, and changing the mailHost attribute in LDAP points Perdition to the right place.

Posted by Rowan Littell at 01:47 PM
December 13, 2005

Perdition notes

To make the Cyrus IMAP transition easier, I’ll be using Perdition to migrate batches of users from KE to SIPALA. We can set the mailHost attribute in the LDAP user entry, and mail will be delivered to the proper server. Perdition can look up the attribute to connect to the right POP or IMAP server on retrieval.

The Perdition LDAP URI will be:

ldaps://directory.earlham.edu/ou=People,dc=earlham,dc=edu?mailHost,uid?sub?(uid=%32s)?!bindname=ldap-search%20etc%20etc,!x-bindpw=secret

Posted by Rowan Littell at 10:03 AM
November 08, 2005

Intrusion Prevention Systems

Looking at IPS boxes recently. The idea being that they’d block virus/worm/etc. propagation, clamp down on spyware, stop some other attacks, and possibly throttle P2P (although the shaper is doing fine in that regard).

  • TippingPoint: I keep hearing a lot of good things about them, particularly that they’re very safe and effective with their initial configuration. Also has a watch-only mode.
  • FortiGate: some people have them, but I can’t find any good info on their capabilities yet.
  • McAfee IntruShield: some like them, and they seem decent. But I’ve never had warm fuzzies about McAfee.
  • Checkpoint InterSpect: they’ve been a decent firewall company for a while, so this is probably decent as well. Heard nothing outstanding about them.
  • Juniper: has been a fairly good player in the specialized net devices (load balancing, VPN, firewall, etc.) for a while, so they’re probably good. Heard they’re not quite as nice as TippingPoint, though, and they might play more happily in a place where they had Cisco buddies to work with.
  • Cisco: seems to have something off the PIX line, but I’ve never been a fan of PIXen.
Posted by Rowan Littell at 08:11 AM
June 07, 2005

Calendar groups

There’s a slight drawback in Sun ONE calendar: you can’t define calendar groups that people can, in whatever way, subscribe to. However, I have found that managing one’s individual groups is as simple as setting an LDAP attribute.

It would be really nice if we could create a group of calendars and just tell people to add that group. Changes to the group could then easily propagate into people’s view of it. Like, say, a group with all the employees in Department A, and as people come and go in the department somebody modifies that group.

No such luck.

However, it’s relatively straightforward, through some LDAP manipulation, to add and modify groups on someone’s individual account. It needs to be done for all users who have that group, but it can be done by the administrator. The group stuff is set in an attribute called icsSet, one for each group. The syntax is:

icsSet: name=Group Name$calendar=username1;username2;username3:subcalendar$tzmode=specify$tz=$mergeInDayView=true$description=

It’s just a set of fields separated by $:

  • name: the name of the group. May contain spaces.
  • calendar: a list of calendars, separated by semicolons. The calendars are referenced by the standard Sun ONE scheme.
  • tzmode: timezone information. Set to specify.
  • tz: the timezone. May be left empty or set to a particular named timezone.
  • mergeInDayView: unknown. Set to true.
  • description: theoretically a description could go here, but the groups created by the web interface don’t have this set and when it is set it doesn’t show up in the web interface. Leave it empty.
Posted by Rowan Littell at 03:47 PM
May 19, 2005

Working Sendmail M4

Attached to this post is a working M4 file for SITH’s sendmail installation.

Sanitized, here is the M4 config that is working on SITH. This should do it all.

sith.earlham.edu.mc

Posted by Rowan Littell at 04:31 PM

Cyrus, LDAP, and Sendmail aliases

I got the aliases problem sussed out. It wasn’t a Sendmail or Cyrus problem but an LDAP problem.

Sendmail was happily looking up alias@domain, rewriting it to alias, and not getting any farther — and then Cyrus said it didn’t know anything about alias.

The problem was that it was looking up alias@domain in LDAP, saying, “yup, that’s local, so it’s just alias,” and then trying to look up alias in LDAP and failing (since only alias@domain is listed as a mail attribute). The fix is to put alias as another mail attribute, and it looks it up just fine.

Previous alias LDAP entry:

dn: cn=ALIAS, ou=Aliases, dc=earlham, dc=edu
objectClass: top
objectClass: groupOfUniqueNames
objectClass: mailrecipient
objectClass: mailGroup
cn: ALIAS
mail: ALIAS@earlham.edu
mailHost: sith.earlham.edu
mgrpRFC822MailMember: recipient

The new one adds one mail attribute:

dn: cn=ALIAS, ou=Aliases, dc=earlham, dc=edu
objectClass: top
objectClass: groupOfUniqueNames
objectClass: mailrecipient
objectClass: mailGroup
cn: ALIAS
mail: ALIAS@earlham.edu
mail: ALIAS
mailHost: sith.earlham.edu
mgrpRFC822MailMember: recipient

This lets sendmail search on the LHS part of the alias@domain and find a match.

Reference this entry for how I thought it should have been done last fall (I was closer than I ended up doing it).

Posted by Rowan Littell at 03:34 PM

Samba 3 groups quirkiness

Apparently Samba 3 still relies on unix-side password and group files for group membership information, not just LDAP.

In particular, the Domain Admins group needs to have its members in /etc/group, otherwise those folks don’t get admin rights. Aaron sussed this out. See the following:

Posted by Rowan Littell at 11:20 AM

SITH, Cyrus, and Sendmail

The test setup on SITH seems to be almost entirely working.

The following all work properly:

  • Cyrus IMAP (access & authentication)
  • Timsieved (access & authentication)
  • Sieve forwarding and vacation scripts
  • Sendmail delivery to Cyrus via cyrus2 mailer
  • Sendmail access & authentication (including AUTH forcing and access DB checks)
  • Sendmail LDAP routing
  • Startup and shutdown of Cyrus, saslauthd, and sendmail
  • Install of sendmail cf and maps

I don’t have alias expansion working yet. If I point an alias in LDAP to SITH, sendmail and Cyrus don’t expand the alias once we get there. Will have to look at that more.

Posted by Rowan Littell at 11:03 AM
May 18, 2005

Sendmail hacking

I figured out the rulesets for the authentication force a while back, but they weren’t working on Solaris. I finally traced the problem to incorrect Berkeley DB version for the access map.

I ended up rebuilding Sendmail with the proper include flags to access the new Berkeley DB install for everything (makemap hadn’t compiled earlier), reinstalled the new Sendmail (and makemap), remade access with the new makemap, and it all seems to work now.

I’m currently writing a minimal Makefile for Sendmail control like *BSD has in /etc/mail, and we should be good.

Posted by Rowan Littell at 05:06 PM
May 16, 2005

Cyrus delivery testing

Just ran a small message through a few thousand recipients on the Cyrus test box.

The test box is a SunFire V120. The number of recipients was 5132. The total delivery took about 3 minutes, during which the load average hit 17 (and sendmail refused a few connections). The original message was 1 K, and a single delivered message was about 2.5 K. Disk space used after delivery was approximately 133 K.

I think my V240 will be plenty beefy, particularly with the Xserve RAID.

Posted by Rowan Littell at 03:57 PM
May 05, 2005

Sendmail cf hacking - fixed

Ah, here we go. Turns out that the auth_type macro doesn’t quite respond to the R+ rule like I thought it did.

I had been trying to do it with just one rule that checked to see whether auth_type had been set. I’ve had to expand that to two rules to check to see whether auth_type is either LOGIN or PLAIN (the two auth types I allow). I’m sure there’s a slicker way to do this, but I’m still a little green at raw cf hacking.

So, for background:

The 64 dollar question is: “How do I keep port 25 open to the world, but require SMTP AUTH for anything outside of a particular network or host list?”

We want KE (or future Cyrus box, or BARIS, for that matter), to be available to clients on and off campus for use — off campus use must be authenticated. But we want to have this on port 25, since there are a number of broken clients out there that can’t deal with SMTP on any other port (or make it very difficult — Qualcomm, are you listening?). But I don’t want to have to set up TAIKA to use authentication when shoveling in the daily mail load, since there’s no good reason to require auth, and doing so would put a small but unnecessary strain on the auth services.

So ideally, we want Sendmail to say, “Ah, incoming from TAIKA — no need for authentication” and “Oh, incoming from 192.168.2.4 — better require auth.” Both on port 25.

It turns out that mucking with the LOCAL_RULESETS can get this accomplished. Here’s what I finally put in the mc file:

LOCAL_CONFIG
Knoauthmap hash -o /etc/mail/noauthmap

SLocal_check_mail
R$*                     $: $1 $| $>"local_auth_plain" $1
R$* $| $#$*             $#$2
R$* $| $*               $: $1 $| $>"local_auth_login" $1
R$* $| $#$*             $#$2
R$* $| $*               $: $1 $| $>"noauth_map" $1
R$* $| $#$*             $#$2

Slocal_auth_plain
R$*                     $: $&{auth_type}
RPLAIN                  $# RELAY

Slocal_auth_login
R$*                     $: $&{auth_type}
RLOGIN                  $# RELAY

Snoauth_map
R$*                     $: $&{client_addr}
R$+                     $: $(noauthmap $1 $)
RRELAY                  $# RELAY
R$*                     $#error $@ 5.7.0 $: "530 Authentication required"

Now put the IP addresses of those hosts that you want to allow to send without authentication in the /etc/mail/noauthmap file and don’t forget to makemap it:

10.0.5.3     RELAY

Then 10.0.5.3 can send without requiring authentication. Everything else gets the “authentication required” error — unless, of course, they’ve authenticated properly, in which case they’re allowed relay.

Posted by Rowan Littell at 05:08 PM

Sendmail cf hacking

Working on hacking the Sendmail cf file to preferentially require authentication on port 25.

I want to require authentication on port 25 for all client addresses coming in to KE except TAIKA — thus allowing people off campus to continue relaying after authenticating but not requiring TAIKA/Baleen to authenticate.

I’m getting somewhere with local rule sets — I’ve got it looking up the client IP in a map and if it’s there allowing it to relay, but I don’t yet have it to the point of being able to bail out and allow relay before that if the client has authenticated. I clearly need some work on the first part of the rule set, but I’m not sure quite how to rehabilitate it yet.

Posted by Rowan Littell at 04:56 PM
May 03, 2005

Comment spam observation

It seems that the comment spam floods in MT are primarily caused by web crawlers downloading the comments that are there rather than attempts at posting new ones.

We had another brief meltdown from comment spam (happens every couple of days now), and I checked the Apache log file immediately afterward. Most of the mt-comments.cgi requests immediately before the incident were GET requests for specific comments, usually with a referrer off site.

I don’t think we’ve had any new comment spam since installing the Scode plugin.

So the trick now is to remove the comment spam from blogs where it’s gotten to. Nasty here, since most of these blogs are long dead class projects which could probably be safely deleted. But can’t do that, so I’ll have to just delete the comment spam.

Posted by Rowan Littell at 02:18 PM
April 01, 2005

LDAP routing load on Sun ONE

Since I was having problems with the OpenLDAP proxy cache, I decided to stress test Sun ONE with the kind of traffic it would get from Sendmail. Turns out that it’s barely noticeable.

I updated all our LDAP records for the proper mail routing information and then wrote a script that selects a username or list name at random, one of our LDAP-routable domains, combines them, and asks for a few attributes. About 10% of the time it generates a random nonexistent address and asks for that. Then it waits for a random amount of time between 0 and 1 seconds (or 0.5 seconds on another test) before asking for another.

It maybe adds about 0.01 to 0.05 to the average 1 minute load average. Still we’re sitting at over 50% idle most of the time and below a load average of 0.50 except when I do the NIS dumps every 15 minutes (and it goes up to 0.60 to 0.70).

I think the proxy cache is not needed.

Posted by Rowan Littell at 11:19 AM
March 31, 2005

LDAP routing stuff

A variety of notes on LDAP routing in Sendmail…

  • LDAPROUTE_EQUIVALENT (or the corresponding file of domains) searches the LDAP server for the domain that you’re masquerading as. cf/README even says this. Don’t forget it and forget to set a masquerading domain if you’re using it.
  • You can have multiple LDAPROUTE_DOMAINs, if you like. They’ll use the same LDAP lookup syntax, though. But at least they’ll look for different domains in the LDAP server.
  • If you’ve got aliases in LDAP, an MX will treat them as aliases and try to expand them, even if you think you want to have it pass them off to a mailHost for that host to expand. At least as far as I can determine. So you’ll have to put aliases in different parts of the tree for different servers to access, depending on who’s supposed to be doing the expansions. Maybe the doco and suggested tree for aliases already does this, but we relish being non-standard here.
  • When it works right, LDAP routing is really really handy.
  • There’s something screwy going on with the OpenLDAP proxy cache that the first time it returns an uncached response Sendmail doesn’t like it. After it’s cached, all is well. Weird.
Posted by Rowan Littell at 08:04 PM
March 30, 2005

Greylisting

Apparently CS installed milter-greylist version 1.6 last week and has been enjoying it greatly. So I’ll be heading that way with SAgate rather than trying to build a MIMEDefang/perl greylist that does what I want it to.

The main catch is that v1.6 doesn’t support as rich an exception language as I’d like to deal with multiple domains in SAgate. v2.0, which is still in beta, has a much nicer ACL language that can properly deal with positive and negative regular expression match requirements, mix up recipient and IP address ACLs, etc. It looks like v2.0 is stable enough to run with, but I’ll keep my eye on it. This milter also has the nice feature of being able to sync with other peers in an MX cluster — maybe not necessary here, but I’d kind of like to build SAgate with load balancing cluster capabilities, if possible (and I think I can, between MySQL, milter-greylist, and some rsyncing of the quarantine directories).

Posted by Rowan Littell at 09:02 PM

Racks and UPSen

Well, a little bit of spring break, and then catching up. Monday I spent most of the day researching APC racks and UPSen for the machine room expansion.

It looks like the top of the line NetShelter racks are pretty much standard, and that’s actually the model that we already have, so getting four more of them won’t be a problem. The Smart-UPS RT seems to be the UPS we want to go for — it’s a double conversion online rather than line interactive, which is a step up from any of the other Smart-UPS models. The Symmetra line is very attractive with its redundancy options, but at a higher cost, more rack space, and lower kVA output, I think we can be better served by the Smart-UPS (particularly with a generator and an automatic transfer switch, which comes in relatively inexpensively).

One of the tricky bits is that it’s apparently ill-advised (to the point of not being possible) to connect a UPS unit with greater than 5kVA to the power source with a plug — you have to hardwire it. For general ease, I therefore limited the discussion to the 5kVA Smart-UPS units, which at one per rack would give us a total of 20kVA power in the rack system.

Posted by Rowan Littell at 08:55 PM
March 18, 2005

Sun Messaging Server

Spent some time this week with a spare V120 and Sun Java System Messaging Server.

I swear, the iPlanet/ONE/Java System family of messaging server products are based on Cyrus, but I can’t find anything that actually comes right out and says so. It’s just that the message store directory structure is too close to be a coincidence. That, and the fact that other Sun enterprise systems are based on open source software — like Directory Server being derived from the UMich LDAP server.

I installed Messaging Server 6 on SITH after decommissioning it from use as the old WebDB server. Since this is a test box and the setup of the whole system is somewhat interdependent on specific versions of Sun enterprise software, I didn’t try to get it talking with our LDAP directory and instead used the bundled one on SITH. The install, as far as that goes, is pretty straightforward. The main drawback is, as I say, its interdependencies on other software and its approach of assuming your a totally standard Sun enterprise-only shop. This will make it play less well with others, in particular Mailman, SquirrelMail, and possibly SAgate.

As you might expect from the underlying structure, the raw performance of Messaging Server is pretty zippy. I wasn’t able to perform rigorous tests, but I see no significant difference between it and Cyrus.

In any case, it sounds like the latest word is to strongly consider Cyrus. I’ll be doing some stress tests on Cyrus after break in order to determine what kind of hardware we’ll want to use on it.

Posted by Rowan Littell at 10:10 AM
March 03, 2005

Sendmail queue groups

Started playing with Sendmail queue groups this morning, and I’ve got a queue group set up on KE for the local mailer.

The following needs to be added to the mc file:

QUEUE_GROUP(`qlocal', `P=/var/spool/mqueue/qlocal, F=f, I=1m')
define(`LOCAL_MAILER_QGRP', `qlocal')

The directory /var/spool/mqueue/qlocal needs to be created (and it must be below /var/spool/mqueue). Restart sendmail, and it’s happy. You can see the contents of that queue by specifying -qGqlocal for mailq. Run that queue the same way with sendmail to process it by hand.

The high power way to do this would be to play with the filesystems that the queue directories are on.

Posted by Rowan Littell at 11:07 AM
March 02, 2005

OpenLDAP as proxy cache for Sun ONE

Expanding on yesterday’s work with the proxy cache in OpenLDAP, I have a fully functional system for caching the Sun ONE entries for Sendmail LDAP routing.

This config file gives the functional basics for a working proxy cache system for this setup:

And this file has the necessary schema to get the object classes and attributes from Sun ONE (iPlanet) into OpenLDAP:

The schema is very bare bones and does not implement all of the Netscape mail attributes and object classes. That’s ok, because I’m overloading the Netscape stuff for our own purposes in Sendmail routing rather than for Netscape/iPlanet/Sun ONE Messaging Server. See the entries from last September (aliases and routing) for details on how I’m doing this.

The purpose of this proxy cache is so that the MX that’s a Sendmail LDAP routing client doesn’t have to hit the master LDAP server constantly to figure out how to deal with known addresses. Instead, it should hit it the first time the address appears, and then keep it in local cache for a while (in the config file above, 30 minutes) before having to ask the master again.

Posted by Rowan Littell at 09:59 AM
March 01, 2005

LDAP proxy cache

Going back to the old OpenLDAP proxy cache stuff, this time for the SAgate spam gateway, so that LDAP mail routing doesn’t hit ASHTI too hard.

The trick is that OpenLDAP doesn’t have a definition for the mailRecipient object class, and it’s tricky to find one out there. Details to be inserted later, but the basic idea is to just get one in there that requires cn and allows mail, mailHost, and mailRoutingAddress. Once these are in, OpenLDAP will happily procy-cache the query that Sendmail makes back to Sun ONE.

Posted by Rowan Littell at 04:03 PM
February 28, 2005

Sendmail, Cyrus, and LMTP

It appears that when you tell Sendmail to use LMTP to deliver to mailboxes, it doesn’t reject unknown recipients, instead letting LMTP do that.

Understandable, since with Cyrus, at least, Sendmail may have no idea who is an actual user. But this means that we’re suddenly having to bounce nonexistent users rather than SMTP-rejecting them. In my architecture, this isn’t a big deal, but I’d rather avoid it.

Apparently, though, there’s a work-around for this. This page has instructions for using the cyrus smmapd process and Sendmail 8.13.x’s socketmap feature to verify that a user exists and their mailbox is writable before letting Sendmail let go of the message to LMTP.

I’m compiling 8.13.1 on the test box now to see how well it works.

Some more info: This post has some useful tips.

FreeBSD’s 8.13.1 port includes patches for Cyrus and socket map, if you enable them. The doco is sparse all around, but we’ll get this working.

Actually, the FreeBSD doco and the Cyrus patches to the port are quite simple. I don’t know how well that’ll carry over into Solaris, if we go that route.

Posted by Rowan Littell at 04:15 PM
February 18, 2005

Cyrus SQUAT

Running the squatter indexing process on the Cyrus IMAP mailboxes seems to significantly improve some of the tests run yesterday.

In particular, the second and third SquirrelMail tests have their times cut by about a third. Most of the other times didn’t change significantly.

Posted by Rowan Littell at 10:07 AM
February 17, 2005

Cyrus Benchmarking

I got Cyrus working to a good approximation and then went about comparing it on the same system with both UW-IMAP and Dovecot. It doesn’t completely blow the others out of the water, but it is distinctly faster in most cases.

The server is a VMware instance on a Dell PE 2650 running FreeBSD 4.7 with 256MB of RAM and a 3.1GHz CPU. The UFS filesystem has softupdates turned on. As before, I ran clearcache prior to each test to clear the RAM cache. The numbers follow (and again, the zeroes in the Dovecot tests represent the ORDEREDSUBJECT thread capability that it does not have):

Servermailbox formattesttime
cyruscyrus1.capa0.006436
cyruscyrus1.fetch0.088901
cyruscyrus1.pine0.259319
cyruscyrus1.select0.057946
cyruscyrus1.sort0.324073
cyruscyrus1.squirrelmail0.817137
cyruscyrus1.thread0.27596
cyruscyrus2.sort0.198544
cyruscyrus2.squirrelmail1.00655
cyruscyrus2.thread0.20403
cyruscyrus3.sort0.12817
cyruscyrus3.squirrelmail0.161446
uw-imapmbox1.capa0.001587
uw-imapmbox1.fetch0.319003
uw-imapmbox1.pine0.297432
uw-imapmbox1.select0.198513
uw-imapmbox1.sort0.289168
uw-imapmbox1.squirrelmail1.607335
uw-imapmbox1.thread0.930278
uw-imapmbox2.sort0.294195
uw-imapmbox2.squirrelmail0.570358
uw-imapmbox2.thread0.378559
uw-imapmbox3.sort0.236569
uw-imapmbox3.squirrelmail0.250143
dovecotmbox1.capa0.117913
dovecotmbox1.fetch0.865082
dovecotmbox1.pine0.883572
dovecotmbox1.select0.108543
dovecotmbox1.sort0.365751
dovecotmbox1.squirrelmail1.20554
dovecotmbox1.thread0.598424
dovecotmbox2.sort0.276563
dovecotmbox2.squirrelmail1.110137
dovecotmbox2.thread0.0
dovecotmbox3.sort0.362111
dovecotmbox3.squirrelmail0.12792
dovecotmaildir1.capa0.089035
dovecotmaildir1.fetch0.359677
dovecotmaildir1.pine0.522552
dovecotmaildir1.select0.156107
dovecotmaildir1.sort0.542121
dovecotmaildir1.squirrelmail2.110486
dovecotmaildir1.thread1.322453
dovecotmaildir2.sort0.385708
dovecotmaildir2.squirrelmail0.393788
dovecotmaildir2.thread0.0
dovecotmaildir3.sort0.250563
dovecotmaildir3.squirrelmail0.175929

And again, with thanks to gnuplot:

Imaptest2

Follow the red line for Cyrus. It’s the fastest in all but four tests:

  • 1.capability, which simply logs in and requests the CAPABILITY for the server.
  • 1.sort, which selects INBOX and then does a SORT (REVERSE DATE) where UW-IMAP just barely nudges it out.
  • 2.squirrelmail, which does some EXAMINEs, EXPUNGEs, UID SORT (ARRIVAL), a FETCH of some header info for the first twenty UIDs, an EXAMINE, and then disconnect.
  • 3.squirrelmail, which does a LIST, LSUB, LIST, EXAMINE, and STATUS.

Both Dovecot in maildir mode and UW-IMAP beat it out by a significant amount on the second SquirrelMail test. Interestingly, the 1.squirrelmail test, which is identical to 2.squirrelmail with the substitution of a UID THREAD REFERENCES for the UID SORT shows Cyrus winning hands down. The second SquirrelMail test is also the only one where Cyrus took longer than 1 second to complete.

I think this shows Cyrus in a pretty good light. It’s fastest at most tasks. I’d worry more about it not being the fastest in two of the three SuirrelMail tests except that the third test is so close as to not necessarily be relevant.

I’m also pretty impressed with its administration. We’ve all heard how nasty and complicated Cyrus is to manage, but that’s not at all what I found: setting it up was a matter of installing OpenLDAP, Berkeley DB 3, Cyrus SASL, and Cyrus IMAP, and then doing a little careful reading for the three config files: imapd.conf, cyrus.conf, and saslauthd.conf. With some intelligent values in there, it just goes right along. I can’t say it’s easier than UW-IMAP, but that’s hardly saying anything. It’s not a whole lot worse than either Dovecot or Courier.

Posted by Rowan Littell at 09:47 AM
February 16, 2005

Cyrus IMAP

Spent most of the day after this morning’s explorations playing with Cyrus IMAP.

I’m not entirely sure what all the fuss is about. Yes, Cyrus is a little more complex than any of the others, and yes, SASL is a slight pain, but I was able to get a test instance of Cyrus up and running on a fairly fresh FreeBSD VMWare instance in a couple of hours (I already had OpenLDAP installed). I’ve even got it authenticating to our LDAP server, and the great part is that the whole system, Sendmail and all, doesn’t care about Unix users — if the Cyrus mailbox exists, that’s all it cares about. So a Cyrus box doesn’t have to be in NIS or have the passwd file synced.

The setup I’ve been testing has the alternate path separator, so that folders are “foo/bar/baz” rather than “foo.bar.baz”, which may make converting from UW-IMAP a little easier.

The main glitch will be transferring all the mail. I don’t know that there’s any way to directly copy messages into the Cyrus mail store and preserve the message flags (copying them in is easy, if followed by a reconstruct call, but they all appear as new messages). The approach at this point would probably be to temporarily reset the user’s password, do an IMAP to IMAP copy from the old server to the new (including all subfolders), and then set the password back to the original.

There are some IMAP copy/sync tools out there, but they all seem to have problems or at least areas where they wouldn’t fit our mentality. I’ll probably end up coding something in perl if we go this way.

First off, though, is to get UW-IMAP (and possibly Dovecot or Courier) installed on this test server to compare some real numbers (running the tests referenced earlier produced extremely good looking numbers, but we’re also running on a much more powerful server and I wasn’t clearing the memory cache between each test).

Posted by Rowan Littell at 05:04 PM

IMAP Performance

I’ve been looking at IMAP performance over the last few days, and I’ve come up with some pretty heretical results.

The platform for these tests was FreeBSD 4.5/i386, running on a Dell PE 2400. The filesystem was UFS with softupdates enabled. RAM was 512MB, and clearcache was run prior to every test. I tested UW-IMAP using mbox format, Dovecot using both mbox and maildir formats, and Courier-IMAP using maildir.

The results (the two tests that have 0 times are for Dovecot on the THREAD ORDEREDSUBJECT command, which is not implemented in Dovecot):

Servermailbox formattesttime
dovecotmaildir1.capa0.053203
dovecotmaildir1.fetch0.156899
dovecotmaildir1.pine0.234601
dovecotmaildir1.select0.002402
dovecotmaildir1.sort0.140133
dovecotmaildir1.squirrelmail2.692953
dovecotmaildir1.thread0.272252
dovecotmaildir2.sort0.046875
dovecotmaildir2.squirrelmail0.183964
dovecotmaildir2.thread0.0
dovecotmaildir3.sort0.012698
dovecotmaildir3.squirrelmail0.007863
dovecotmbox1.capa0.002634
dovecotmbox1.fetch3.066798
dovecotmbox1.pine0.844669
dovecotmbox1.select0.002329
dovecotmbox1.sort0.053704
dovecotmbox1.squirrelmail2.689958
dovecotmbox1.thread0.206899
dovecotmbox2.sort0.046262
dovecotmbox2.squirrelmail2.488003
dovecotmbox2.thread0.0
dovecotmbox3.sort0.014379
dovecotmbox3.squirrelmail0.00873
couriermaildir1.capa0.002704
couriermaildir1.fetch0.268682
couriermaildir1.pine0.158809
couriermaildir1.select0.032384
couriermaildir1.sort1.258794
couriermaildir1.squirrelmail1.587882
couriermaildir1.thread1.232035
couriermaildir2.sort1.219862
couriermaildir2.squirrelmail0.364796
couriermaildir2.thread1.131387
couriermaildir3.sort0.107762
couriermaildir3.squirrelmail0.04076
uw-imapmbox1.capa0.002472
uw-imapmbox1.fetch0.363805
uw-imapmbox1.pine0.171299
uw-imapmbox1.select0.142298
uw-imapmbox1.sort0.253514
uw-imapmbox1.squirrelmail1.16735
uw-imapmbox1.thread0.595428
uw-imapmbox2.sort0.266605
uw-imapmbox2.squirrelmail0.62801
uw-imapmbox2.thread0.35498
uw-imapmbox3.sort0.163758
uw-imapmbox3.squirrelmail0.116656

Or, as a graph:

Imaptest

The PINE and SquirrelMail tests are taken from a typical short session dump of the corresponding mail clients. The PINE test is a simple opening of the inbox, fetching the header information for a range of messages, fetching the header information for another range, and finally fetching a message. The first two SquirrelMail tests are variants on the form of open mailbox, expunge, sort by some key, and then fetch the header information for a message range. The third just lists subscribed mailboxes and gets a status for INBOX.

This seems to suggest that while maildirs are faster for certain operations, mbox is quite acceptable overall, in particular the UW-IMAP implementation (the Dovecot version seems to fall over). This is counter to the prevailing wisdom that maildirs scream. On the other hand, it seems that the functions used most often by mail clients typically involve a fair amount of sorting, something that maildirs are particularly bad at (except sorting by arrival).

Not included in this set of tests were numbers for creating a new mailbox, copying a message from INBOX to the new mailbox, expunging, copying the message back, expunging, and then deleting the new mailbox. Preliminary tests indicate that Dovecot’s maildir and UW-IMAP’s mbox implementations are about the same speed, Courier is slightly slower, and Dovecot’s mbox implementation completely falls over. This is perhaps a slightly overblown scenario, but without the mailbox creation and deletion it’s not far off from actual usage: there are a number of mail clients that will do a search of the INBOX for certain things, copy those messages to another mailbox (like, say, the spam box), and then expunge — possibly over and over.

I am still working on testing Cyrus, and I’d like to test Sun ONE as well, if I can get a server up and running for it.

Posted by Rowan Littell at 09:31 AM

Greylisting

Ran across some possibly decent greylisting sources last night.

milter-greylist looks like a good, fast greylisting implementation. The main question there is whether I can make it fit into the right spot with respect to MIMEDefang. Also, libspf2 would be useful in tuning the greylist. Ought to look into that even if I want to implement a MIMEDefang-internal greylist.

Posted by Rowan Littell at 08:06 AM
February 10, 2005

Sympa and Mailman

E-mail system upgrades are on the slate for this spring, and part of that involves upgrading the mailing list software. I’ve been reading up on the latest versions of Sympa and Mailman as the top contenders.

Sympa has some nice features over Mailman v2.0, but Mailman v2.1 (which is current) gains a lot of those. I still somewhat dislike Mailman’s binary-everything worldview (making command line fixes difficult), but they’ve added a bunch of utility programs for dealing with binary files. Sympa has some interesting LDAP integration features, but I suspect that those aren’t terribly high priority.

At this point, Mailman is in the lead, but we still need to finish looking at Sympa.

Posted by Rowan Littell at 03:35 PM