Sendmail on KE was updated this morning to perform LDAP routing. A new version of the sendmail package, compiled with LDAP support, replaced the previous version. The alias map was replaced with calls to LDAP, as done on both TAIKA and BARIS. This allows for a smoother transition to SIPALA as we migrate users and update the LDAP mailHost attribute.

KE has undergone two major changes: migration of Mailman and introduction of Baleen.

• Mailman has moved to BARIS (see notes there), along with an upgrade to Mailman.

• Baleen, a mail scanning system based on the concept of an e-mail gateway or firewall, has been introduced (running on TAIKA) as a replacement for SpamAssassin running on KE, offloading most of the heavy processing that KE had been experiencing.

Monday, January 5, we switched WebMail services to a new server.

This new server is dedicated to running the WebMail interface, taking the load of this off of KE, the e-mail content server. E-mail remains stored on KE, but WebMail access is now handled by BARIS.

I've installed up-imapproxy to test it with the SquirrelMail test instance.

Standard compile on FreeBSD 4.5. I put the daemon in /usr/local/libexec, the config file in /usr/local/etc and the stats program (pimpstat) in /usr/local/sbin. I wrote a basic startup script for /usr/local/etc/rc.d. The testing configuration listens on port 1143 and connects to mailer.earlham.edu on port 143. SquirrelMail test is now pointing to this instead of the regular IMAP server.

The proxy seems to work, including password changes. I can't tell, at the moment, whether it's faster or not. I suspect the delays I'm seeing are the PHP rendering and the netlag between campus and home.

I installed PHP Accelerator on KE in the hopes of keeping SquirrelMail from stomping on the CPU too much. So far it seems to be working well.

Installing the accelerator involves downloading the source file (I used php_accelerator-1.3.3r2_php-4.3.0_freebsd_i386-4.5), installing the shared library, editing the php.ini file, and restarting Apache.

I saved the shared library as /usr/local/lib/php_accelerator_1.3.3r2.so.

I added the following lines to php.ini:

• zend_extension = /usr/local/lib/php_accelerator_1.3.3r2.so
• phpa.cache_dir = /tmp/phpa

I created the directory /tmp/phpa to store the cache files. I changed the owner to www and made it mode 0700.

SquirrelMail seems to be working well, and the load average seems to be hovering in the 1-3 range at the moment.

I installed a patched Sendmail package last week, correcting the recent vulnerability.

Sendmail.org provided a simple patch for all 8.12.x Sendmail sources that corrected this vulnerability. I added the patch to the FreeBSD package directory on the build system and created a new sendmail package (sendmail-sasl-8.12.6_4ecs) for installation on KE.

OpenSSH has been upgraded on all the FreeBSD servers to the latest openssh-portable package, correcting the vulnerabilities discovered last week.

FreeBSD 4.4 packages were installed on HEIWA, KE, and SHANTI. A FreeBSD 4.7 package was installed on PAX. These are all openssh-portable-3.6.1p2_3. They install into /usr/local and require the following changes to /etc/rc.conf:

sshd_program="/usr/local/sbin/sshd"

sshd_flags="-f /etc/ssh/sshd_config"

PAX is the only server that currently required these changes, as the others had previously been upgraded to OpenSSH-portable. PAX also required minor changes in the /etc/ssh/sshd_config file.

Apparently Japanese language Internet Explorer (I believe) is unhappy with SquirrelMail 1.4.0, so I made the old version (1.2.9) available under the /squirrelold URL.

Some students in Japan complained that they were getting blank pages upon initial connection to the SquirrelMail login page. This corresponded with the introduction of 1.4.0, so after determining that it was the Japanese browser and that I couldn't really debug it at present, I enabled the /squirrelold URL (primarily accessible from the root webmail server page). Reports are that this works.

We upgraded SquirrelMail to 1.4.0 on Monday morning.

Ian Kelly did most of the work getting the new version ready to go and making sure plugins were compatible. On Monday we found a bug in the HTML code for the mailbox list which made Squirrel unusable on Netscape 4.7. A patch had been submitted to the developers list but was not in CVS, so I copied it to our installation. We may need to watch for that when we upgrade.

I set the Vexira updater daemon to update itself every two hours.

With a recent release of a new virus that got through during the time between updates, I decided that having more frequent updates on the mail server was important. The Windows 2000 updates on MIR are still daily.

I downgraded the version of OpenLDAP on KE to cure a segfault problem in the PAM LDAP module.

The PAM LDAP module periodically caused a segfault in any program that uses it when it was paired with the 2.0.21 version of OpenLDAP. I grabbed the package from HEIWA and force-removed and downgraded it on KE, and the segfault problem seems to have gone away.

The known way to reproduce the problem was using sudo -v. This would always segfault with 2.0.21, and it never segfaults with 2.0.14.

Since we’re not using OpenLDAP on these servers for anything besides its library for the PAM module, I believe this is safe enough.

I have compiled and installed the pam_ldap module on KE to help us with the authentication difficulties.

FreeRADIUS was failing under the load of authentications this morning, since it was running in single threaded mode on SHANTI. To get around this, I found and compiled the FreeBSD package for the pam_ldap PAM module. This shifts the bulk of our network authentications (e-mail) from RADIUS to directly querying LDAP.

Unfortunately, it seems that the SSL portions of pam_ldap aren’t happy on KE, even though it worked fine on my workstation. Nevertheless, I believe we have a relatively stable authentication system at the moment.

Update

SSL is working fine now. It required the setting host directory.earlham.edu rather than the IP address so that it could verify the certificate. I tested this on HEIWA, and now it, too, is using pam_ldap in place of RADIUS.

I installed 1 GB RAM and updated versions of MIMEDefang, SpamAssassin, and PHP on KE today.

• RAM Addition: I installed 1 GB of RAM, increasing the total to 2 GB and using up all four DIMM slots. 512 MB of the new RAM is being used as a memory filesystem mounted on /var/spool/MIMEDefang. This is speeding up mail submissions considerably, since all MIMEDefang work and Vexira antivirus scanning is now essentially done in RAM. The 512 MB size of the filesystem is fixed due to limitiations in FreeBSD, however this should be plenty for these purposes (by comparison, the old spool directory had approximately 600 MB free and much of the space used there is for the /var/log directory).

• I upgraded MIMEDefang to version 2.33. I apparently hadn’t installed the latest version of my mimedefang-filter script, so I copied that from my workstation (to get native Vexira support).

• SpamAssassin is now at version 2.54, which should be much more accurate than the year-old previous version. This also includes the Bayes extensions, which are currently turned off.

• I upgraded to PHP 4.3.1 (the latest version in the FreeBSD ports tree). This required the addition of the pth package. I also changed the default configuration to have register_globals be ON (to keep SquirrelMail happy).

Another security flaw found in Sendmail, as per this patch announcement.

I built a new Sendmail 8.12.6 package (sendmail-sasl-8.12.6_3ecs) and installed it on KE. I used the generic 8.12 patch file in the patch tarball referenced in the page above.

Sendmail upgraded on KE, same issue as MIR.

KE is special: we're using the sendmail-sasl port from the FreeBSD ports tree because we want to provide both SMTP AUTH and STARTTLS (which are not present in the default sendmail, particularly for FreeBSD 4.5). I have built a new package on my workstation, labeled sendmail-sasl-8.12.6_2ecs. This package include the 8.12 patch from sendmail.org. It still calls itself 8.12.6, however it is a fully patched version.

It seems to be working properly.

To apply the patch to the FreeBSD ports tree, I downloaded the patch (above) and saved it as /usr/ports/net/sendmail/files/patch-ab. The ports make system automatically applies patches with that filename scheme. Searching the source files after make for a post-patch modification (like "Dropped invalid comments from header address" in sendmail/headers.c) shows that the patch worked. This string is also in the sendmail binary (/usr/local/sbin/sendmail - use the strings command to look for it).

TWIG URLs have been redirected to SquirrelMail.

On webmail.earlham.edu, I set /webmail and /twig to redirect permanent to /squirrel. See TWIG Removal. I have not removed the TWIG software from KE, nor have I changed the PostgreSQL prefs database in any way yet.

We're currently in the restore phase of operations - restoring around 30 Gb of mail from the dump image earlier this morning. No glitches so far at all.

Details:

• 4:30 - 5:00: set downtime in Nagios so it wouldn't complain about KE/RT being down.
• 5:00: began level 0 dump of /home (/dev/aacd1s1e) to PAX:/home/r0p1/ke-home-dump.20030221 over private net.
• 5:00 - 7:00: compiled MIMEDefang 2.30 on my workstation and transferred it to /tmp on KE, monitored dump process, had breakfast, etc...
• 7:10: dump finished
• 7:20 - 8:00: arrive, reboot DBA (crashed, again)
  install Sendmail 8.12.6:
  • use package that I created on my workstation earlier
  • modify /etc/passwd and /etc/group to change smtp user and group to smmsp
  • modify /usr/local/etc/mimedefang/mimedefang.conf so that it runs as user smmsp instead of smtp
  • replace /etc/mail/Makefile with one taken from FreeBSD 4.6.2 and modified to reflect proper location of Sendmail cf M4 directory (/usr/local/share)
  • add lines to /etc/rc.conf for the new 8.12.x split queue structure:

    sendmail_enable="YES"
    sendmail_flags="-L sm-mta -bd -q30m"
    sendmail_submit_enable="YES"
    sendmail_submit_flags="-L sm-mta -bd -q30m -ODaemonPortOptions=Addr=localhost"
    sendmail_outbound_enable="YES"
    sendmail_outbound_flags="-L sm-queue -q30m"
    sendmail_msp_queue_enable="YES"
    sendmail_msp_queue_flags="-L sm-msp-queue -Ac -q1m"
    

  
  install MIMEDefang 2.30:
  • extract source in /tmp
  • "make install"
  
  Swap 18 Gb data disks for 72 Gb data disks, configure as RAID 5 container via Dell PERC BIOS, and then use /stand/sysinstall to label and newfs the disks
• 8:00 - current: restoring dump image from PAX

We have three 18 Gb disks for Dell PowerEdge servers going begging. Maybe they'll find a home in MIR. I won't do anything with them until I know that the new disks are happy, though.

Data drives to be replaced on KE, 2/21/2003.

I will be replacing the data drives on KE (three 18 Gb drives in RAID 5) with three 72 Gb drives (RAID 5). I'll start a level 0 dump at 5 (or as soon after that as the nightly backup will allow), saving the dump file to PAX:/home/r0p1. This should take approximately two hours, judging by AMANDA reports. Then power off system, replace drives, configure drives, format drives, and restore dump file. Happy ETA is 10 AM, I'm publicizing noon as definite ETA.

TWIG is being removed on 2/24/03.

I'll change the link to a "removed" page with pointers to SquirrelMail. At some later date we can drop the TWIG PostgreSQL database (still have to keep pgsql for the RT database).

Upgraded Samba on all systems except MIR on Friday morning.

PACO and ROJ are using Sunfreeware.com packages (requires the popt package). All others are using FreeBSD packages built on my workstation.

On installation on SHANTI, it somehow overwrote all individual entries in the smbpasswd file such that passwords were null and accounts were disabled. Restored from previous night's backup.