Links

Archives

General update

Several projects eating time in the last couple of weeks.

Cyclades console server

It died. I’ll be getting tech support on it to tell me what to do next. In the mean time, we have:

Caprica name server

This will be a replacement for Eirene, since it’s actually in the machine room and in the racks. Right now it’s been delegated the priv.earlham.edu subnet and runs a DHCP server for the private network. The idea here is that the private net /etc/hosts file was getting unwieldly and it’d be good to have that in DNS. Since the console server died, I decided to use the ethernet ports on the Sun ALOM as alternate consoles; having them speak DHCP was the easiest. The main trick there is that they require infinite lease times.

Timezone updates

As Indiana begins to observe DST, timezone info must be updated. All the Solaris boxes have patches for this (113225-03 for Solaris 9, 109809-03 for Solaris 8). This does the right thing for the Indiana time zone. On the FreeBSD systems, I’ve simply been setting the localtime to EST5EDT; the main problem we’ll see is in timestamps of files (off by an hour isn’t a huge concern for most of these systems).

A lot of Cyrus work

This week has been mostly a continuation of Cyrus IMAP work. The system itself is ready for production use, but there are a number of helper scripts and utilities that need to be developed before we can start the migration.

• Tuesday I discovered that / and /usr were both at 99% full, even before thinking about going production, and after agonizing about this, I decided that the best thing to do was to repartition and re-Jumpstart. To make it slightly easier on myself (although building Cyrus and friends really isn’t that hard, as my SSL troubles last week showed), I made Solaris packages out of most of the software: Cyrus SASL, Cyrus IMAP, Perdition (and vanessa libs), and some perl libs — Cyrus::IMAP, and Mail::IMAP. Sendmail is still an after-build, and there are a number of other perl libs that I’ve since added to support the migration, user management, and quota scripts.

• Top of the list of things yet to do, of course, is the migration utility itself. I’ve got one almost completely ready — it temporarily resets the user’s password, copies all the subscribed folders from KE to SIPALA, gets the user’s quota from the self service quota system, sets the LDAP mailHost attribute to sipala, and then resets the password back to the original. Still to complete is the part that grabs the forward and vacation settings, if present. I’d also like to extend it to general sieve copying and some other Cyrus admin work, for future use in migrating users from one Cyrus box to another, but that’s future.

• As implied, there’s a quota system tie-in. I’ve been working on an XML-RPC interface to the quota system, or at least a subset of it, to use on SIPALA rather than the direct PostgreSQL connection. I think the XML-RPC is more portable, and it means I don’t have to install PostgreSQL libraries on SIPALA. It also means we could use MySQL as the database for the quota system if we wanted to. The part of the XML-RPC server that supports the migration script is done, and some parts that support the cron update scripts are done, but there’s more work there. I also need to tool the web interface for the quota system to take you to the mail host that your mail is on, rather than assuming everyone’s on the same mail host.

• Also, as implied, there’s a WebDB tie in. WebDB is where forwarding and vacations are set, so it needs to be made aware of sieve. I’ve considered putting that info into LDAP and letting both KE and SIPALA generate the necessary stuff from there, but I think that’s actually not the best way. I think WebDB needs to be looking at the mailHost attribute itself and then either talking KE’s own peculiar language or sieve as appropriate.

Pine and Perdition

Testing Pine with Perdition — I have my .pinerc stored on the IMAP server, and I simply replaced all the hostnames that point to KE (mailer.earlham.edu) with ones that point to mailproxy.earlham.edu (Perdition and Sendmail running on SIPALA).

It works flawlessly. I get my mail folders on KE, and I send mail through SIPALA.

SIPALA Notes

Most of work this week has been working on SIPALA and various pieces of the new mail system. At this point, just about everything is working. I think the main things to do are VxFS/VxVM installation for the message store and considering a milter for antivirus checking.

Further build notes:

• OpenSSL 0.9.7 is required, as Sendmail doesn’t do the right thing with signature generation, or something with 0.9.6i. Since everything depends on OpenSSL, this required a rebuild of everything. Teaches me how to rebuild, in any case.

• OpenLDAP (v2.3.19): built using Forte C

• Cyrus SASL (v2.1.21): built using Forte C, and with CFLAGS=-DOPENSSL_DISABLE_OLD_DES_SUPPORT (otherwise we get conflicting encryption routines).
  • ./configure —enable-login —with-bdb-incdir=/usr/local/BerkeleyDB.4.2/include —with-bdb-libdir=/usr/local/BerkeleyDB.4.2/lib —with-ipctype=doors —with-openssl=/usr/local/ssl —with-ldap=/usr/local —enable-ldapdb CFLAGS=-DOPENSSL_DISABLE_OLD_DES_SUPPORT

• Cyrus IMAP (v2.3.1): built using GCC (Forte C doesn’t work)
  • ./configure —with-cyrus-user=cyrus —with-cyrus-group=cyrus —with-bdb=/usr/local/BerkeleyDB.4.2 —with-ldap=/usr/local —with-openssl=/usr/local/ssl —without-snmp —disable-krb5afspts
  • Change “pod2man” to “/usr/perl5/5.6.1/bin/pod2man” in man/Makefile
  • Set SASL_LIB=-L/usr/local/lib -R/usr/local/lib -lsasl2 and SASL_INC=-I/usr/local/include in perl/Makefile and perl/sieve/Makefile after the make process bombs in the perl directory (the Makefiles haven’t been generated before that). Then remake. Perl objects will be built with Forte C; this derives from system perl being built with standard Sun tools, and it does, in fact, work properly.

• Perdition (v1.17): built using GCC (and the vanessa libraries)
  • ./configure —disable-ldap-doc —with-ssl-includes=/usr/local/ssl/include —with-ssl-libraries=/usr/local/ssl/lib —with-ldap-includes=/usr/local/include —with-ldap-libraries=/usr/local/lib —disable-daemon-map
  • Daemon map needs to be disabled, since it uses the mkdtemp() routine, which is not present on Solaris 9.

• Sendmail (v8.13.3): built using GCC

Operational and configuration notes:

• Sendmail requires that /usr/local/lib/sasl2 be symlinked to /usr/lib/sasl2. Setting the environment pointer in the .mc file makes no difference; Sendmail still looks in the latter for the SASL information.

• /etc/nsswitch.conf: remove “nis” from the alias info, otherwise Sendmail tries to look up aliases in NIS, which won’t work.

• Sendmail .mc file needs to use “” notation for the LDAP specs, since they include commas (simple `’ notation doesn’t work for specs that include commas).

• Sendmail .mc files need to be processed with GNU m4 (/usr/local/bin/m4). /usr/ccs/bin/m4 doesn’t properly set the config directory macro from the command line.

• If I recall correctly, the only addition to the Sendmail cf directory is the ckuser_cyrus.m4 file in cf/feature.

Cyrus, Perdition, LDAP

I got Cyrus working again on Solaris, as per entries last spring. Not terribly difficult. The harder part was getting Perdition, the IMAP/POP3 proxy, working. There are some source code errors that made the build on Solaris tricky.

• Don’t build Perdition with the daemon map option. It requires the mkdtemp(3) library call, which isn’t available on Solaris.

• Don’t build Perdition static like the documentation tells you to for Solaris. It will always try to load a dynamic library, but static builds don’t provide dynamic libs.

• There is an error in the LDAP initialization routines. #if 0 the code that calls ldap_initialize and just let it call ldap_init. The author said the fix would be in the next release sometime last fall, but no such release seems to be forthcoming.

• Installation: Perdition is set to listen on the third network interface, which is set to mailproxy.earlham.edu, a different IP address from the first interface. That way the LDAP attribute can be the hostname of the local box without having to mess around with alternate ports for Cyrus. Cyrus listens on the bge0 interface, while Perdition listens on bge2 and then contacts bge0 if LDAP tells it to.

Once installed, there have to be individual Perdition processes for each protocol: IMAP, POP3, IMAPS, and POP3S. They each use different config files and PID files. Once up and running, it does work flawlessly, and changing the mailHost attribute in LDAP points Perdition to the right place.

SULA, Big RAM

Got SULA installed, mostly, and Aaron’s taking it over now. It’s got 8GB of RAM, which entailed turning on the PAE extensions for FreeBSD’s kernel. And that entailed making a static kernel, which entailed compiling in the IPFilter stuff. All is well now, and MySQL is up and running for development work.

Off the top of my head, MySQL will be used for:

• Moodle (PostgreSQL just isn’t working out there, since some of the developers can’t seem to write database-neutral SQL code).

• Cascade content management server

• MovableType

There will probably be other things coming down the pike that could benefit from a MySQL install.

Lumana and Jumpstart tweaks

There were still a few bugs in the SVM script that I was tweaking the other day, but I got them worked out, jumpstarted Lumana, and things are pretty good now.

The new SVM script creates MD device names based on the raw devicce name of the slice and only deals with mirrored setups:

cNtXdYsZ becomes dXZ, throwing out the controller number (we’re assuming that all install disks are on the same controller) and the disk LUN number (we’re assuming that there’s only the one). So a mirror of c1t0d0s7 and c1t1d0s7 has submirror devices of d7 and d17. The meta device container for the two is taken from 7 plus the target number of the second disk: here, 7+1, and then the slice number is the ones place. So the container for this would be d87.

I did this for Lumana, because it came with 4 disks, and I wanted to mirror the second pair as well. They’re on c1t2d0sN and c1t2d0sN, so they’re made of meta devices d30 and d40, with a container of d100 (only the first slice, 0, is used).

Need to add NetBackup and possibly the Forte compiler set to Lumana, and I’ll be done with it.

Jumpstart and V440

The Solaris 9 version we had on ROJ was too ancient to support the new V440, so when it tried to boot in Jumpstart, it failed. Updating the Solaris version in the Jumpstart area has always been a pain; something about the script seemed to always bomb out on ROJ, requiring editing of the script and running it from somewhere other than the CD. Fortunately, Doug Hughes clued me in to the fact that all I had to do was copy the Solaris_9 directory contents from the DVD into the expected place in the Jumpstart area, and all would be good.

Next up, though, is making my volume manager Jumpstart script work with four disks. It’s always assumed two disks for a simple mirroring setup, but the V440 has four, and I’d like to add those in to the mix at Jumpstart time, too. I think I got it working now — now that I’ve removed a couple of lines of stale code that were causing a test bug and leaving the system in a pretty sad state. To test it all tomorrow.

SANs and LUNs and devices, oh my...

Once you tell the ITMPT driver in Solaris that it can access more than the first LUN on a fibre channel device, mapping out the SAN makes a lot more sense.

I’ve got the SAN mapped out by Solaris and VMware device names over in the TWiki. I’m not entirely certain what happens to the device mappings if we add another FC target device to the SAN (like, say, the original Xserve RAID), but I suspect that if we just add it to the next available ports on the switches it’ll be ok.

Wildcard SSL certificate

We purchased an wildcard SSL certificate last week for use across our SSL services. Installation has been pretty painless. Apache is easy, sendmail and UW-IMAP are pretty straightforward. The main holdouts are EZproxy and the LDAP servers.

• EZproxy: sicne the cert is for *.earlham.edu, and EZproxy, when doing ProxyByHostname, redirects to login.proxy.earlham.edu, the client gets a cert mismatch warning. I’m talking to the folks at Useful Utilities to see if there’s any way around that while keeping ProxyByHostname.

• LDAP: Sun ONE Directory Server is based on Netscape tech, so it expects the certs and keys to be in the PKCS12 Berkeley DB format. I’ve got the utilities off a Directory Server 5.2 disc to convert them, installed them on ASHTI, and done initial conversions. I have not installed the certs and tested them yet, though.

Still to complete is ProFTPd, which should also be straightforward.

Server installs, continued...

All the new servers, etc. have arrived and are in the racks.

I’ve been hip deep in fibre channel stuff this week. I think I’m leaning away from LUN masking, simply because any changes to the LUN masking setup require a reboot of the RAID controller on the Xserve RAID boxen, disrupting any other hosts that are connected to volumes on that controller.

Installing the new V480, I had to power down EYEWI to move its DLT drive. I took the opportunity to replace the heat sink with the bad fan with the heat sink I pulled out of the new V210 that had an extra CPU in it (long story: Oracle licensing and educational discounts on preset V210 configs).

Toys!

Santa brought me several cartloads of toys this year. Still trying to get all of them unboxed, racked, and inspected, much less set up and configured. That’s for the rest of winter.

• SunFire V210: replacement for PACO, Banner development

• SunFire V440: new SCT Luminis server

• Dell PE 1850: MySQL server

• Dell PE 1850: replacement for HEIWA, web server

• Dell PE 850: content management server

• SAN equipment: Xserve RAID, 2 Qlogic SANBox 5200s, a bucket of cables, FC HBAs, and more drives to populate another Xserve RAID.

Everything but the new Xserve RAID is racked, and I’m making sure that things are in good condition. I’ve still got a few more pieces of equipment that have yet to come in, too.

FLEXlm

I re-purposed one of the FreeBSD VMware instances yesterday to serve as a FLEXlm license server for Maple 10.

I gave it a minimal set of resources and then reloaded the Linux emulation packages onto the OS. FLEXlm seems relatively straightforward to manage, despite it being a “legacy” product that no company seems to want to admit having ownership of. In any case, it works, and I closed of access to the system from off campus.

The trickiest part, actually, was with some weird bug between VMware and FreeBSD where it would hang and spike the VMware server CPU at the point in the boot process that it was probing the parallel port. I disabled both parallel and serial ports in the VMware BIOS, and after a while it seemed to boot up again. I don’t know if I solved the problem, though.

Bridge online

Got the OpenBSD bridge online today, passing (and blocking, as appropriate) traffic on the main pipe. Tuning Argus turned out to be slightly tricky, but it’s going well now.

Argus post processing takes a lot of memory, and it doesn’t do anything useful when it fails — just dumps core (it’d be nice if it checked the return code from malloc and failed gracefully with an error, but no…).

Found a couple of IP addresses still spewing worm traffic and alerted the right people.

Bridging

Finally got around to putting OpenBSD on the firewall box that we got a while back. As I remembered, setting up bridging is pretty simple, assuming you can figure out which of the 6 interfaces on the box you’re using. Argus is installed and running, but I haven’t had much chance to play with it yet. Next week I’ll be looking into that more and then connecting it to the main pipe.

Mirrored LDAP

After jumpstarting SITH into another DirectoryServer configuration, I got LDAP set up to mirror between it and ASHTI. It was surprisingly easy.

There’s a new user in the administrators section (same place as the MacOS X config user and the LDAP search user) that is allowed to connect with replication data. Setting up the replication agreements was straightforward (but don’t try to use SSL with our certs — it fails). Replication updates happen right away, and work properly.

Thanks to round robin DNS entries, the load evened out pretty quickly between the two, and now we’re sitting at an average of about .4 to .5 on each. Still a little high for my tastes, but it’s usable. Maybe a new LDAP server pair should be in the future; we’ll see how well this pair does for now.

LDAP index notes

A few notes about rebuilding LDAP indices on Sun ONE.

• The proper command is /usr/sbin/directoryserver -s instance db2index-task -D managerDN -w - -n userRoot -v Don’t use the db2index.pl script, since you have to be in the right directory for it to find the shared libs and all.

• Progress of the index can be found in the instance’s error log, the most logical place for it.

• I don’t know whether the database can be in read-only mode. Best to try it without, so set nsslapd-readonly to off in cn=userRoot,cn=ldbm database,cn=plugins,cn=config (see this entry).

• It really does take a little while.

Sendmail cf hacking - improvements

I made some improvements to the previous local rule sets that have it use the access map and allow for classed IP subnets.

I changed the noauth_map ruleset so that it goes as follows:

Snoauth_map
R$*                     $: $&{client_addr}
R$+                     $: $>A <$1> <?> <+ Connect> <$1>
R<RELAY> $*             $@ RELAY                relayable client IP address
R$*                     $#error $@ 5.7.0 $: "530 Local authentication required."

This takes the client IP address, puts it through the A ruleset (which looks for an IP match — or classed subnet — in the access database and returns the RHS of that match). If the result of the A access lookup is RELAY, then we allow an unauthenticated connection, otherwise we fail to the standard authentication required error.

I installed the full ruleset on both BARIS and KE this evening, and we’ll see how things progress tomorrow. This should force all incoming general mail through TAIKA — or make the spammers stop using KE and BARIS.

Baleen semi-production

I changed the MX for our domain to TAIKA and removed the other MXs this morning.

We had a bit of a bull ride after that as I tried to get the mail systems under control from the Monday morning onslaught of faculty list messages. The Baleen filter was mostly fine, although I had to rewrite the greylist configuration generator slightly — it wasn’t happy with the regex syntax I’d used for exceptions.

The main other problem was solved by making BARIS also do Sendmail LDAP routing. I had to tweak the default LDAP spac for BARIS to take out the v3 syntax flag and point the root at dc=earlham,dc=edu rather than into the Aliases OU. List mail works fine now and goes directly to KE from BARIS, bypassing TAIKA and Baleen as it should (and I will love this when we have Cyrus’ single instance message store).

I have a few testers in Baleen now, and everything is working well. Most of the spam seems to be stopped outright by greylisting and DNSBLs.

Cyrus on Solaris

I’ve got Cyrus IMAP running on Solaris now.

First, get the right Jumpstart configuration up so that you can get the build environment (and then don’t forget to add Sun’s C compiler). And also don’t forget to add Berkeley DB to the Jumpstart config for later…

Then, the following:

• Install OpenLDAP 2.2.x: SASL and IMAP really want this, even though Solaris has it’s own LDAP libs. For some reason they won’t work right, at least at first blush.

• Install Cyrus SASL. The configure arguments are “—enable-login —with-ldap=/usr/local —with-openssl=/usr/local/ssl —with-ipctype=doors”. Don’t forget to use crle to add /usr/ucblib to the shared lib search path.

• Install Cyrus IMAP. The configure arguments are “—with-bdb=/usr/local/BerkeleyDB.3.3 —with-ldap=/usr/local —with-cyrus-user=cyrus —with-cyrus-group=cyrus —with-openssl=/usr/local/ssl —with-sasl=/usr/local”.

• Create the config directory (/var/imap) and partition directory (/home/spool/imap), owned by the cyrus user and group, and with 0750 permissions.

• In the IMAP build directory, run tools/mkimap as the cyrus user to populate the config and partition directories.

• Start saslauthd and the IMAP constellation, and things ought to work.

Next up, making Sendmail 8.13.3 compile and run on Solaris instead of the native Sendmail.

Baleen racked and testing

After turning off MIR, I installed TAIKA in the rack for Baleen and initiated more thorough testing of Baleen.

I used the ethernet cable from MIR for the gigabit link on TAIKA as well as the power cables from MIR.

I’ve got procmail sending duplicates of everything it gets for a few people through Baleen on TAIKA and filing the results in another folder so we can see how well it works. So far we seem to be doing quite well.

I got a logo for Baleen, and I’ve been doing a little polishing off of the web interface as well as some minor behind-the-scenes work. I’m planning on letting this run in preliminary testing mode for a week or so as I work on other stuff, finish some details, and monitor the system. Then we’ll go into production testing, where I turn it on for the whole domain but in log-only mode except for volunteer users. Finally I’ll convert it to full-on mode and celebrate.

SAgate: name change and fully testing

Well, SAgate has a new and catchier name: Baleen (as in whale). I’ve also gotten the bugs worked out of the system and have it running is general test mode (accessible with a special domain address, but not yet handling all of our domain’s traffic as the default MX).

The main bug was that SQL preferences weren’t taking in SpamAssassin properly. I don’t know what the right incantation for the SA perl code ended up being, but I found a way to run spamd with full SQL access, referencing full address usernames (and not trying to setuid), so I modified the MIMEDefang code to run spamc and submit the message to spamd, and this seems to work. This means we’ve got an extra fork going on for spamc, and if I get a chance I may look into making the MD code a spamd client itself. One nice thing about using spamd is that I can make config changes to SA without having to take down MD. Nice decoupling. Now I need to delete the cruft code from the MD filter.

I also had to download the DCC clients. I was under the mistaken impression that I had some perl modules that were DCC clients already, or that it was somehow included with SA (like the URL block lists are). No such luck, but the FreeBSD port is workable and available. I may try to see if I can run some daemon mode DCC service so SA doesn’t have to fork off a DCC client itself for each message, but that’s less of a worry.

More SAgate

Gee, I wonder what I’ve been working on… SAgate is close to ready.

The following things still need at least an eyeball, if not some work:

• PHP user interface review (including “logout” links and a password change feature).

• General MIMEDefang filter review, particularly in the MIME rewriting area for tagged spam.

I’ve gotten the database all set up, installed phpMyAdmin and graphdefang to decent degrees. Milter-greylist is installed and works with the ports version of libspf2, even if that’s an older version. I had to make a few code tweaks in the config file generation for that and the startup script, but it’s all happy now and doing the proper thing with the config file. Oh, nice — not only does it tell you the time for the original greylist delay in the tempfail code, but it tells you the remaining time in future attempts (of course, this can be turned off, as I recall, but it’s nice for testing, at least).

SAgate build continues

We’re getting closer with SAgate.

MySQL, MIMEDefang, and SpamAssassin are all there and looking like they’re working properly. The SAgate MIMEDefang script is happy. I’ll need to populate the database with some defaults, make sure it’s handling SpamAssassin ok, and then take a look at the PHP interface.

LDAP routing stuff

A variety of notes on LDAP routing in Sendmail…

• LDAPROUTE_EQUIVALENT (or the corresponding file of domains) searches the LDAP server for the domain that you’re masquerading as. cf/README even says this. Don’t forget it and forget to set a masquerading domain if you’re using it.

• You can have multiple LDAPROUTE_DOMAINs, if you like. They’ll use the same LDAP lookup syntax, though. But at least they’ll look for different domains in the LDAP server.

• If you’ve got aliases in LDAP, an MX will treat them as aliases and try to expand them, even if you think you want to have it pass them off to a mailHost for that host to expand. At least as far as I can determine. So you’ll have to put aliases in different parts of the tree for different servers to access, depending on who’s supposed to be doing the expansions. Maybe the doco and suggested tree for aliases already does this, but we relish being non-standard here.

• When it works right, LDAP routing is really really handy.

• There’s something screwy going on with the OpenLDAP proxy cache that the first time it returns an uncached response Sendmail doesn’t like it. After it’s cached, all is well. Weird.

SAgate on the rise

This morning I delved into finally preparing the actual server for SAgate.

I’ve finally made the transition to FreeBSD 5.3, and it seems to be a very nice step forward in the OS. The basics are the same, but it’s got the new tech that we’ve been waiting for.

Mostly I downloaded and built ports this morning, but I was at least able to get Sendmail LDAP routing working as far as I had on Karelia and get a few other ports ready for more of my attention. I seem to need a slight bit of a refresher on LDAP routing, though, as it’s not wanting to accept addresses in LDAP unless I masquerade as earlham.edu, which I’d rather not (for the sake of multiple domains in SAgate, and also for the fact that an MX shouldn’t have to masquerade as a domain that it’s MXing for, should it?).

But dang, that server’s loud. I need to get it out of my office as soon as I can. At least it’s not as loud as the VMware box I had in there for a couple of days, though.