I upgraded Sendmail on TAIKA to version 8.13.6 to patch a race condition vulnerability.

I downloaded tar archives of the latest ports trees for sendmail and sendmail-ldap, unpacked them in /usr/ports/mail, and then built a new sendmail-ldap port. As little has changed since the prior version, there were no problems.

Sendmail on KE was updated this morning to perform LDAP routing. A new version of the sendmail package, compiled with LDAP support, replaced the previous version. The alias map was replaced with calls to LDAP, as done on both TAIKA and BARIS. This allows for a smoother transition to SIPALA as we migrate users and update the LDAP mailHost attribute.

I upgraded milter-greylist from version 2.0b5 to 2.0.2 this morning. A new configure option, —with-libspf2_10 is required for the version of the SPF library we have. Installing requires changing the program name from milter-greylist to greylist, as the rc.d script I wrote for FreeBSD 5 uses the latter.

KE has undergone two major changes: migration of Mailman and introduction of Baleen.

• Mailman has moved to BARIS (see notes there), along with an upgrade to Mailman.

• Baleen, a mail scanning system based on the concept of an e-mail gateway or firewall, has been introduced (running on TAIKA) as a replacement for SpamAssassin running on KE, offloading most of the heavy processing that KE had been experiencing.

BARIS has had a number of alterations to it, although its role is still entirely e-mail related.

• In February, 2005, we migrated Mailman mailing lists to BARIS, upgrading the version of Mailman in the process and introducing searchable archives for all mailing lists.

• At the same time we migrated Mailman, we added LDAP mail routing functionality, moving all mailing list aliases into LDAP, where KE reads them every hour and BARIS and TAIKA access them immediately. New and deleted Mailman lists are also updated every hour.

• BARIS is currently running SquirrelMail version 1.4.4.

BARIS is being a test case for a new sednmail milter: milter-sender.

The new filter performs a reverse check on the sending address’s primary mail server to verify that the sending address is legitimate and that the domain is willing to accept return mail. No actual return mail is sent, however if the tests fail then the incoming message is rejected. Other sanity checks are also performed. The primary result of this filter ought to be a decrease in the amount of spam coming from fake addresses, primarily at large domains such as Hotmail and Yahoo!. While most of this could be done in MIMEDefang, having a separate milter with this as its single function decreases complexity of the individual pieces. If testing procedes favorably on BARIS, we should be able to install this filter on KE as well.

Update: This milter does actually send a return message, contrary to the implications on the author’s web site (it is never stated that it doesn’t send this message, it is just somewhat implied). It also is not stable on BARIS, dying after a few days and unable to restart. For these reasons we won’t be using this milter.

MIR is no longer acting as a mail exchanger for earlham.edu e-mail.

After bringing up BARIS as a mail exchanger with MIMEDefang and other sendmail filtering capabilities, we have disabled this capability on MIR. MIR has never been able to properly use the milter features of sendmail to allow us to use MIMEDefang or other filters for decreasing spam. This is due primarily to the difficulty of building a good RPM package for a milter-enabled sendmail on MIR. Since BARIS is the same architecture and operating system and KE, building the appropriate sendmail package has been quite easy.

BARIS is now a mail exchanger for the earlham.edu domain.

BARIS is running Sendmail 8.12.10 with the same MIMEDefang milter as KE. With this, we should be able to both host a mail exchanger and have it run extremely strict spam and virus checks at the same time. Unlike KE, BARIS is running the open source ClamAV antivirus program. As this program is able to run in a daemon mode, it is scanning for viruses much more quickly than Vexira on KE.

BARIS is also set up to use STARTTLS and AUTH, allowing it to be used as a secure mail relay for mail clients. Like KE, it is accessible on both the standard SMTP port (25) and the submission port (587), with the latter requiring authentication.

Monday, January 5, we switched WebMail services to a new server.

This new server is dedicated to running the WebMail interface, taking the load of this off of KE, the e-mail content server. E-mail remains stored on KE, but WebMail access is now handled by BARIS.

I've installed up-imapproxy to test it with the SquirrelMail test instance.

Standard compile on FreeBSD 4.5. I put the daemon in /usr/local/libexec, the config file in /usr/local/etc and the stats program (pimpstat) in /usr/local/sbin. I wrote a basic startup script for /usr/local/etc/rc.d. The testing configuration listens on port 1143 and connects to mailer.earlham.edu on port 143. SquirrelMail test is now pointing to this instead of the regular IMAP server.

The proxy seems to work, including password changes. I can't tell, at the moment, whether it's faster or not. I suspect the delays I'm seeing are the PHP rendering and the netlag between campus and home.

I installed PHP Accelerator on KE in the hopes of keeping SquirrelMail from stomping on the CPU too much. So far it seems to be working well.

Installing the accelerator involves downloading the source file (I used php_accelerator-1.3.3r2_php-4.3.0_freebsd_i386-4.5), installing the shared library, editing the php.ini file, and restarting Apache.

I saved the shared library as /usr/local/lib/php_accelerator_1.3.3r2.so.

I added the following lines to php.ini:

• zend_extension = /usr/local/lib/php_accelerator_1.3.3r2.so
• phpa.cache_dir = /tmp/phpa

I created the directory /tmp/phpa to store the cache files. I changed the owner to www and made it mode 0700.

SquirrelMail seems to be working well, and the load average seems to be hovering in the 1-3 range at the moment.

I installed a patched Sendmail package last week, correcting the recent vulnerability.

Sendmail.org provided a simple patch for all 8.12.x Sendmail sources that corrected this vulnerability. I added the patch to the FreeBSD package directory on the build system and created a new sendmail package (sendmail-sasl-8.12.6_4ecs) for installation on KE.

Probably the most widespread worm ever, Sobig.F continues to inundate the Internet. Fortunately for us, we've been blocking the worm at our e-mail server since Tuesday morning at 5:40 when we saw our first occurrence. Nevertheless, the worm has had a significant impact on our network.

We have been blocking viruses and worms at the e-mail server for approximately one year and keeping detailed statistics during that time. On average, we process between 10,000 and 20,000 e-mail messages per day. Monday (8/18/2003), we processed 15,577 messages, 58 of which were viruses that were blocked.

Tuesday, Sobig.F was released and we saw a significant increase in e-mail and virus activity. We processed 22,015 messages on Tuesday. 3,725 of these messages were blocked viruses, of which the vast majority were the Sobig.F worm (3,682).

On Wednesday, the worm activity intensified. We processed 32,030 total messages. More than a third of these messages (12,005) were viruses; 11,886 of them were Sobig.F. This made Wednesday, August 20, 2003, the fourth busiest day ever in the history of Earlham e-mail.

Thursday kept up the activity, with 28,800 total messages, 11,118 of which were Sobig.F. During Wednesday and Thursday, we were receiving Sobig.F messages at the rate of approximately one every seven seconds. As of mid-morning on Friday, the rate seems to be remaining at the same level as the previous days.

In addition to the statistics kept by the mail server itself, the worm's impact can be seen by the PacketShaper on our Internet connection. The following graph shows the inbound e-mail traffic (including POP and IMAP retrievals) on our Internet connection for the period of the two weeks prior to Friday, August 22, 2003. It shows a noticeable increase in traffic starting on August 19.

Update (Sep 11, 2003):

We started keeping detailed records of the number and kinds of viruses dropped at the mail server in February of this year. This graph shows both the total number of messages processed each day and the number of viruses dropped. The number of messages shows a strong weekly cycle, while viruses have only made up a small portion of the traffic until recently. This week we have dropped more viruses than the peak number of messages processed in some previous weeks.

Apparently Japanese language Internet Explorer (I believe) is unhappy with SquirrelMail 1.4.0, so I made the old version (1.2.9) available under the /squirrelold URL.

Some students in Japan complained that they were getting blank pages upon initial connection to the SquirrelMail login page. This corresponded with the introduction of 1.4.0, so after determining that it was the Japanese browser and that I couldn't really debug it at present, I enabled the /squirrelold URL (primarily accessible from the root webmail server page). Reports are that this works.

We upgraded SquirrelMail to 1.4.0 on Monday morning.

Ian Kelly did most of the work getting the new version ready to go and making sure plugins were compatible. On Monday we found a bug in the HTML code for the mailbox list which made Squirrel unusable on Netscape 4.7. A patch had been submitted to the developers list but was not in CVS, so I copied it to our installation. We may need to watch for that when we upgrade.

I set the Vexira updater daemon to update itself every two hours.

With a recent release of a new virus that got through during the time between updates, I decided that having more frequent updates on the mail server was important. The Windows 2000 updates on MIR are still daily.

I installed 1 GB RAM and updated versions of MIMEDefang, SpamAssassin, and PHP on KE today.

• RAM Addition: I installed 1 GB of RAM, increasing the total to 2 GB and using up all four DIMM slots. 512 MB of the new RAM is being used as a memory filesystem mounted on /var/spool/MIMEDefang. This is speeding up mail submissions considerably, since all MIMEDefang work and Vexira antivirus scanning is now essentially done in RAM. The 512 MB size of the filesystem is fixed due to limitiations in FreeBSD, however this should be plenty for these purposes (by comparison, the old spool directory had approximately 600 MB free and much of the space used there is for the /var/log directory).

• I upgraded MIMEDefang to version 2.33. I apparently hadn’t installed the latest version of my mimedefang-filter script, so I copied that from my workstation (to get native Vexira support).

• SpamAssassin is now at version 2.54, which should be much more accurate than the year-old previous version. This also includes the Bayes extensions, which are currently turned off.

• I upgraded to PHP 4.3.1 (the latest version in the FreeBSD ports tree). This required the addition of the pth package. I also changed the default configuration to have register_globals be ON (to keep SquirrelMail happy).

I downloaded and installed Sendmail RPMs from RedHat's support site. I installed the base, cf, and doc packages. Everything's fine.

Another security flaw found in Sendmail, as per this patch announcement.

I built a new Sendmail 8.12.6 package (sendmail-sasl-8.12.6_3ecs) and installed it on KE. I used the generic 8.12 patch file in the patch tarball referenced in the page above.

Sendmail upgraded on KE, same issue as MIR.

KE is special: we're using the sendmail-sasl port from the FreeBSD ports tree because we want to provide both SMTP AUTH and STARTTLS (which are not present in the default sendmail, particularly for FreeBSD 4.5). I have built a new package on my workstation, labeled sendmail-sasl-8.12.6_2ecs. This package include the 8.12 patch from sendmail.org. It still calls itself 8.12.6, however it is a fully patched version.

It seems to be working properly.

To apply the patch to the FreeBSD ports tree, I downloaded the patch (above) and saved it as /usr/ports/net/sendmail/files/patch-ab. The ports make system automatically applies patches with that filename scheme. Searching the source files after make for a post-patch modification (like "Dropped invalid comments from header address" in sendmail/headers.c) shows that the patch worked. This string is also in the sendmail binary (/usr/local/sbin/sendmail - use the strings command to look for it).

I upgraded Sendmail on MIR to take care of CERT Advisory 2003-07.

I used RH6.2 RPMS from RedHat support. The one oddity about this is that this removes support for SMTP AUTH on MIR. This should not be a big deal, since this is primarily a high order mail exchanger and not a system where we expect people to point their mail clients in sending mail.

Note: I have to edit /usr/lib/sendmail-cf/cf/Makefile to change CFDIR to an absolute pathname in order to get the rcs.mgr install script to work properly.

TWIG URLs have been redirected to SquirrelMail.

On webmail.earlham.edu, I set /webmail and /twig to redirect permanent to /squirrel. See TWIG Removal. I have not removed the TWIG software from KE, nor have I changed the PostgreSQL prefs database in any way yet.

We're currently in the restore phase of operations - restoring around 30 Gb of mail from the dump image earlier this morning. No glitches so far at all.

Details:

• 4:30 - 5:00: set downtime in Nagios so it wouldn't complain about KE/RT being down.
• 5:00: began level 0 dump of /home (/dev/aacd1s1e) to PAX:/home/r0p1/ke-home-dump.20030221 over private net.
• 5:00 - 7:00: compiled MIMEDefang 2.30 on my workstation and transferred it to /tmp on KE, monitored dump process, had breakfast, etc...
• 7:10: dump finished
• 7:20 - 8:00: arrive, reboot DBA (crashed, again)
  install Sendmail 8.12.6:
  • use package that I created on my workstation earlier
  • modify /etc/passwd and /etc/group to change smtp user and group to smmsp
  • modify /usr/local/etc/mimedefang/mimedefang.conf so that it runs as user smmsp instead of smtp
  • replace /etc/mail/Makefile with one taken from FreeBSD 4.6.2 and modified to reflect proper location of Sendmail cf M4 directory (/usr/local/share)
  • add lines to /etc/rc.conf for the new 8.12.x split queue structure:

    sendmail_enable="YES"
    sendmail_flags="-L sm-mta -bd -q30m"
    sendmail_submit_enable="YES"
    sendmail_submit_flags="-L sm-mta -bd -q30m -ODaemonPortOptions=Addr=localhost"
    sendmail_outbound_enable="YES"
    sendmail_outbound_flags="-L sm-queue -q30m"
    sendmail_msp_queue_enable="YES"
    sendmail_msp_queue_flags="-L sm-msp-queue -Ac -q1m"
    

  
  install MIMEDefang 2.30:
  • extract source in /tmp
  • "make install"
  
  Swap 18 Gb data disks for 72 Gb data disks, configure as RAID 5 container via Dell PERC BIOS, and then use /stand/sysinstall to label and newfs the disks
• 8:00 - current: restoring dump image from PAX

We have three 18 Gb disks for Dell PowerEdge servers going begging. Maybe they'll find a home in MIR. I won't do anything with them until I know that the new disks are happy, though.

TWIG is being removed on 2/24/03.

I'll change the link to a "removed" page with pointers to SquirrelMail. At some later date we can drop the TWIG PostgreSQL database (still have to keep pgsql for the RT database).