Recently in Email Category

We lost power to part of the Hamilton Hall server room on Friday afternoon just before 3:00 pm.  The main server wasn't affected, but the main storage array was, which means that mail, web and workstations were unavailable until the problem was corrected.  Web sites were back up by half past three, but other services were spotty until about four o'clock.

There was no damage to the files on the storage array, though some mail may have been returned to senders as undeliverable.

Most workstations will need to be rebooted (Alt-Ctrl-F1 then Alt-Ctrl-Del); some will need to be restarted (hold power button for ten seconds to turn off then turn back on).

Any jobs running on bayes, gosset or freesurface will have been lost as those servers were connected to the part of the power system which failed.

http://www.rhpcs.mcmaster.ca/Most of us are able to spot the "Webmail Upgrade" or "Webmail Account Warning" phishing spam these days, so I don't normally warn people about it.

But a spate of such spam has hit @math.mcmaster.ca accounts right on the heels of some server upgrades and mail problems, so I'm going to emphasize that RHPCS will never

• ask you for your password
• send a message without signing off with the name of a specific RHPCS staff member
• commit more than two outrageous solecisms per message

Email messages addressed directly to @math.mcmaster.ca addresses sent from off-campus sources (e.g. gmail.com, another university or from home without a VPN connection) were not deliverable between 5:00 pm Tuesday and 4:30 pm Wednesday.

Some of the undeliverable messages will have been queued off campus and delivered once @math.mcmaster.ca was accessible to external mail servers again.  Other messages will have bounced, in which case the sender will most likely have received a delivery-failure warning.

Messages sent from the following sources were not effected ...

• mathmail.mcmaster.ca web mail
• univmail.mcmaster.ca or muss.mcmaster.ca web mail
• mail clients (Outlook, Thunderbird, Mail) used on campus
• mail clients used from off-campus with a VPN connection

Most mail from off campus addressed to @math.mcmaster.ca addresses is not reaching the server; the problem started when we moved the mail server from the ABB server room to the HH server room yesterday evening.

I'm working with UTS to resolve the problem and expect to have it sorted out this afternoon.

Note that ...

• mail forwarded by @mcmaster.ca to @math.mcaster.ca is arriving;
• mail forwarded by unvimail is arriving;
• all mail originating from univmail, muss, other campus mails servers, or VPN-connected clients is arriving.

Some of you will be having trouble getting to your mail via imap clients or webmail until later on this evening: I neglected to redirect the mathmail.mcmaster.ca to the new network location of the server. My apologies.

Note that anyone using the addresses mail.math.mcmaster.ca or ms.mcmaster.ca won't see these problems - though mathmail.mcmaster.ca is the preferred address.

SquirrelMail, used for mathmail.mcmaster.ca webmail, is miserably slow even though the server is more or less idling. I'll be looking at this today or Monday.

Mail (via imap clients) is no longer restricted.

In order to reduce the load on the storage server while to digests the replacement disk, I've turned of IMAP access to mail boxes.

You can read your mail via pine or http://mathmail.mcmaster.ca.

I'll be turning things on periodically to test the ability of the storage server to accept the mail load. Access from on-campus locations will be turned on before access from off campus.

You may notice that some mail is arriving later than expected or in the wrong order. That's because mail which could not be delivered earlier when the server was busy or down was held upstream for a few hours before delivery was attempted again.

We've had complaints of pine (and other command-line applications) being painfully slow in the past few weeks. It appears that this is almost always the result of logging in from off campus via VPN. Our own testing has shown that the slowness disappears when one shuts off VPN and connects directly to ms.mcmaster.ca.

We don't know yet whether this is a result of a change in the UTS VPN service or load or whether it might be due to an interaction between the new server and the UTS service ... though the former seems far more likely than the latter.

UTS is aware of the problem.

Some email messages delivered following the end of yesterday's network interruption will have confusing timestamps: the messages are stamped with the date and time of their arrival on our servers, not the date and time at which they were sent. And even though our network connection has been up since 7:30 a.m., some messages will arrive up to several hours later as the upstream servers only attempt delivery periodically.

McMaster's internet connection was down from mid-Monday afternoon until 7:30 this morning. Apparently a fibre-optic cable belonging to out upstream provider was cut.

Email sent to your @math.mcmaster.ca or @mcmaster.ca addresses during the downtime will either have made it through when the network came up or will have been bounced back to the sender with a warning message.

Spammers are constantly probing mail systems for guessable passwords and then using those mail systems to send spam to thousands of addresses. This has happened with mathmail.mcmaster.ca recently and so we have made these changes:

1. accounts with weak passwords will be locked very quickly;


2. the number of recipients will be limited for webmail.

Preventing use of our system for sending spam is very important: if too much spam originates from our network, mcmaster.ca might well end up in email blacklists and all email from McMaster to other networks would be blocked (this has happened before).

Password Scanning
We will be scanning for easily-crackable passwords more frequently and allowing only two days between emailing a warning and locking an account.

Restrictions to Bulk Mailings
You will only be able to send a message to 50 people at a time and no more than two of such messages in a five minute period.

Alternatives to Webmail
The webmail interface at mathmail.mcmaster.ca is really quite rudimentary and was never intended to be the primary interface. If you want to send out bulk email to classes, I highly recommend setting up a mail client such as Thunderbird, Evolution, OSX Mail, or Outlook Express / Windows Live Mail; we have setup instructions for our mail service.

And of course there is good old pine: the command-line interface seems awkward at first, but it's a very powerful email program.

There is no limit to the number of recipient you may specify with either pine or a mail client.

Access to http://mathmail.mcmaster.ca from off-campus network locations is blocked for the time being as we investigate a possible security problem.

If you connect to the campus VPN service, you will be able to use webmail. You can also use IMAP/POP clients or ssh to ms.mcmaster.ca and run pine.

... and mail was unresponsive until 2:25. It's back up now.

Webmail has been accessible on via on-campus or VPN network connections since Oct. 5th because it was used by spammers to send out thousands of spam messages. The spammers appear to have got in using some easily cracked passwords.

I will be doing two things before opening up access to webmail again:

1. asking users with easily crackable passwords to change their passwords


2. tweaking webmail so that brute-force cracking is harder to do

Webmail will probably be opened on Tuesday, Oct. 12th.

Access to http://mathmail.mcmaster.ca from off-campus network locations is blocked for the time being as we investigate a possible security problem.

If you connect to the campus VPN service, you will be able to use webmail. You can also use IMAP/POP clients or ssh to ms.mcmaster.ca and run pine.

Access to mail via webmail and mail clients will wink out for up to ten minutes now and again Thursday evening as we bring a new file server up.

Mail is spotty this evening as we work out some mail-load issues on our new server. Web mail at mathmail.mcmaster.ca is particularly slow and pine on ms is laggy. Mail clients (Thunderbird, OS X Mail, Outlook, etc.) are fine for the most part.

There will be occasionally periods (of a few seconds or minutes) when inbound and outbound mail will be stopped.

Yikes. Our spam filter wasn't running for a few hours this evening - you may have a spate of spam to deal with. Consider it a taste of what you normally miss :)

The new mail server is using a new self-signed security certificate. Your mail client or browser may throw up a warning to the effect the effect that the certificate cannot be verified or has changed. In either case, you may accept/confirm/authorsize the security exception.

Thunderbird on the ms workstations has been updated; this is a minor stability/security upgrade. You should close and restart Thunderbird if your session predates this message.

There will be a few brief interruptions to incoming and outgoing mail this morning.

Some time after the file server recovered on Saturday morning, there was a hiccough with the non-crashed server accessing home directories starting with a - l. This was easily corrected, but unfortunately was not caught not until after I returned from camping. As of 10:30 Monday, all home directories are accessible and mail is starting to flow to accounts starting with a - l.

Many mathserv users received a message with the subject "Please confirm your message" today, purportedly from support@math.mcmaster.ca. This appears to be a spam or phishing attack, though a broken one.

As ever, you can be pretty sure that something which purports to come from a "support team" or unnamed administrator is not from RHPCS; messages from an RHPCS analyst will always have a proper name.

The links in these messages can be deceiving because they appear to be on www.math.mcmaster.ca, but they HTML actually points to a different server. For safety, you can always choose to copy-and-paste a URL in a message into your browser, rather than clicking on the link.

The departmental server was having trouble processing mail this afternoon. While I was working on the problem, some spam got through unfiltered and there were some temporary outages to outbound mail.

A fake system administration message with the subject "Math & Stats Webmail Alert" was sent to many Math & Stats accounts today; the text begins ...

Attn: Faculty/Staff/Students,

This message is from McMaster University Technology Service Desk to all
Faculty, Staff and Students of the Department of Mathematics & Statistics
using the Math & Stats Webmail accounts.

This is a phishing ploy - i.e. scammers are trying to trick you into emailing them your username and password. This is a cleverer ploy than usual in that it references the department name and UTS. But there are still several clues that this message came neither from the RHPCS sysadmins nor from UTS ...

We are still running on one server instead of two and workstation and website access is still slow. I hope to return half of the load to the second server on Tuesday morning. Note that there may be brief interruptions to mail client access between now and tomorrow morning.

Accounts starting with the letters m to z have home directories on the failed server; these home directories have been recovered on the other server using backups. If your account is in this range, you may have lost mail or file changes from early Monday morning.

More specifically, mail received for these accounts and file changes made between the time of the backups (ca. 1:30 am) and the time of the server failure (ca. 2:30 am) are not reflected on the recovered home directories being used.

Once I have the time to analyze the failed server, I should be able to recover any missing messages or files.

We've discovered a bug in our SpamAssassin configuration - quite a wide-spread one, we've since found out - which resulted in email dated 2010 receiving an elevated spaminess score. This has been corrected. Only a very few messages on mathserv were actually affected by the problem and I will notify the intended recipients on Monday.

We're seeing a spate of malware spam with subjects like "A new settings file" or "The settings for the username@math.mcmaster.ca mailbox were changed" (a good portion are making it past the spam filter). This messages should be deleted and ignored.

Remember that any real message from one of the sysadmins asking you to do something about your account will always be signed with the name of particular person, not a generic "sysadmin team" tag.

UTS announced that mail sent to @mcmaster.ca addresses yesterday morning between 8:00 and 10:00 may not have been delivered. Mail sent to directly to @math.mcmaster.ca addresses were not affected. Mail redirected to @math.mcmaster.ca from @mcmaster.ca may well have been affected.

Here is the original UTS announcement ...

Dozens of math mail accounts have received messages with the subject "Conflicker.B Infection Alert" purporting to be from Microsoft and carrying an attachment with a supposed disinfection tool; most but not all instances are being caught be the spam filter. This message is itself a worm vector and should be deleted.

We've a spate of email messages with subjects like "Conflicker.B Infection Alert" and "Important Notification!" purporting to be from "System Administrator" or "Microsoft Windows Agent". These messages are examples of phishing, the intent of which is to get you to run a program which will infect your computer or visit a web site and divulge your password. Needless to say, you should ignore these messages and neither click on the attachments nor visit the web sites.

We've found that double-clicking on mail attachments in Thunderbird doesn't work under Gnome, though it does under KDE. But there is a simple workaround and also a fix - a wacky one, but an easy one.

I will be turning off email access via webmail and IMAP/POP clients for ca. 15 min. at 9 PM this evening in order to do some maintenance.

Email will be unavailable for a few minutes at a stretch several times on Monday because of some email maintenance I'm working on.

We currently have about 6 Gb of deleted mail stored on mathserv (that's about 6% of all mail storage). The indexing and updating of the Trash/Deleted folders is an unnecessary drain on the server. Starting next Thursday, Trash/Deleted folders greater than 10 Mb will be removed each week. You will be notified two days before if your Trash/Deleted folder is slated for purging.

A number of us have seem a dramatic spike in the amount of spam in our inboxes over the past two days - this is a result not so much of more spam hitting the server, but rather of more of the spam making it past the UTS and RHPCS filters. We are looking into the problem.

It would be helpful if you could file the missed spam into the folder rhpcs/spam-missed rather than deleting it.

There has been at least one recent case of the UTS mail gateway (through which much mail bound for mathserv also travels) flagging scanned documents sent from the HH-303 printer/scanner as spam - check your spam folder if a document doesn't arrive immediately.

Betimes, the check-mail process started by a mail client (Outlook, Thunderbird, etc.) doesn't close down properly. These processes take up memory and resources and interfere with new connections. I've now got the server periodically killing mail processes more than one hour old. The good news is that this clears up some bad behaviour with Apple Mail; the bad news is that this _might_ result in deleted mail reappearing.

Let me know if you start to notice any behaviour like this.

I'm sure that many of you are seeing a spam spike in the past two weeks. This is due to a increase in the amount of spam being sent, a change in some of the spam, and my removing some filter tweaks in order to rationalize them. I'll be looking at this some time this week.

The mail server has continued to accepted mail addressed to @icarus.mcmaster.ca some six years after icarus, the former mail server, was decommissioned. About 20% of the spam sent to Math & Stats addresses is actually sent to icarus; it appears that very, very little real mail is addressed to icarus anymore.

As of 3:30 pm today, mail sent to an account @icarus.mcmaster.ca will bounce.

If you are worried that someone may be trying to send you legitimate mail, let me know and I will send you a list of addresses which have tried to reach you @icarus.

The file-server problem is corrected as of 10:15 am and all services are operational again.
The mail filters were not operating properly for many people from midnight to 10am today, so you find a strong flavour of spam in your inbox (200+ messages in under 12 hours, in my case).

I will be turning off email client access (incl. webmail) for a few minutes at a time this morning.

Mail was trickling through the incoming queue between noon and 3:45 pm. The jam has been cleared and mail is flowing again. You may see a spike of spam because spam filtering was turned off 30 seconds (during which period hundreds of messages were delivered).

The self-signed mail-server security certificate has been updated. Go ahead and accept it if your mail clients asks.

You can read the departmental computing updates in Apple Mail, Thunderbird and some other mail clients if you subscribe to the RSS feed at
http://www.math.mcmaster.ca/blogs/computing_news/index.rdf

Email was not being delivered from ca. 12:30 pm to 3:45 pm today. Mail is flowing again; no messages should have been lost. Since some mail clients were failing silently - mine, for example - so the problem was not immediately apparent.

A reminder that any mail from "the McMaster admin team" or some such which asks you to go to a web site and change your password or to email your password is to be ignored. Any mail from the RHPCS sysadmins will come from a personal account with a real name, not from "the team". A recent example of the sort of message making the rounds follows.

Each night, accumulated spam is compressed and archived. The archives are now stored outside of the normal mail folder so that mail clients won't try to read and index them. The command "access-archived-mail" gives you access to these folders. For more info, see http://www.math.mcmaster.ca/computing/email/?page=spam

We have implemented a simplified vacation-message procedure has been for accounts that use spam filtering (and so cannot use the stock vacation command). In summary:

cd ~
mkdir .vacation
\rm .vacation/cache
vi .vacation/message
touch .vacation/on

For more information, see this email FAQ.

We have introduced a new spam-filtering configuration for the departmental server. We're still using SpamAssassin to identify spam and procmail to filter mail, but we have added improved backscatter filtering, a duplicate-message filter, a rationalized folder scheme and automatic daily archiving.

To use the new setup, run the command activate-mail-filter on mathserv or your linux workstation.

More information on the Computing Resources website.

The backscatter filtering announced two weeks ago has been turned off for a week now due to problems with false positives. We will be making changes to the spam filtering mechanism in the next week or so which will result in your spam and bounce folders being renamed and backscatter filtering working again.

A number of people have been blasted with spam backscatter since I first mentioned the recent storm two weeks ago.

I've modified the spam filter so that these messages (which notify you of the failed delivery of a message you never sent) are treated as spam. Note that you must have spam filtering turned on in order for the filtering to work.

A handful people using RHPCS-managed email servers have been hit by backscatter in the past week: dozens of messages reporting that email from them has been rejected by various mail servers for various reasons ("no such address", "message contains a virus", "message is spam"). This backscatter is result of a spambot somewhere on the internet using a valid McMaster email address in the forged "From:" header.

In general, a backscatter storm will result in a batch of messages arriving over the course of an hour or two and then end.

I've activated tagging of these messages using SpamAssassin but have not turned on filtering until I have a chance to see what a real tagged backscatter storm looks like. If you find yourself on the receiving end of one, please let me know before you delete all of the bounce messages.

In addition to the system-wide delivery delays this morning, some people (maybe a dozen) will have seen other odd mail behaviour. The problems were tracked back to two sources which were corrected at ca. 10:30 am and 2:30 pm. No inbound mail appears to have been permanently affected and stalled outbound mail should be flowing again.

Incoming messages were held for ca. 30 minutes before being delivered for much of Friday morning due to an experiment with greylisting (a method of blocking spam which relies on spam programs giving up on delivery much more quickly than real mail programs). We have discontinued that experiment for now.

There was a problem delivering some email messages to mathserv (or to addresses forwarded from mathserv) Sunday and Monday. I've addressed what I believe to be the source of the problem and mail is flowing freely again and most of the undelivered messages should appear shortly.

Mail and web sites were inaccessible for about five minutes at 6:10 pm while I was working on the problem.

Spam filtering was interrupted between 4:45 pm and 7:15 pm today due to a SpamAssassin crash; the failure was unrelated to the server problems of the weekend. I will be keeping an eye on the SA daemon.

Spam filtering wasn't working between 5:30 pm Saturday and 12:15 pm Sunday due to a misconfiguration. It was dreadful, I realize - my inbox alone was hit with more than 400 spam messages in that period.

Some people have not yet turned on spam filtering. Here are the instructions from http://www.math.mcmaster.ca/mathcomputing/email/?page=spam:

All you need to do in order to start using SpamAssassin is to put the following lines in a file called .procmailrc in your home directory:

### spam assassin
SPAMTO=Spambox # keep in Spambox
#SPAMTO=/dev/null # remove leadng # to discard
INCLUDERC=/usr/local/etc/procmail/spam
### end spam assassin

If you are confident that only spam and no important real email is reaching your Spambox folder, you can comment the Spambox line out and uncomment the /dev/null line to send the spam directly to the bit bucket.

Email will be inaccessible via imap clients (Outlook, Thunderbird, Mail.app) on and off this afternoon and tomorrow morning while I debug a server performance problem.

Based on the increased amount of spam in my own inbox and a cursory examination of a few other inboxes, it appears that the new spamassassin setup needs tuning. You should see some improvement in the next week or so.

Math & Stats webmail is once again available at mail.math.mcmaster.ca.

The webmail interface will be upgraded in the second or third week of april. You can expect that this interface will be occasionally unavailable.

Your username - and thus your email address - might be awkward: changx79 or longwowo or something. Remember that people can also email you at firstname.lastname@math.mcmaster.ca: Xianbin.Chang@math.mcmaster.ca and William.Longwood@math.mcmaster.ca are much nicer.

Most email applications will let you set firstname.lastname@math.mcmaster.ca for your From and ReplyTo fields, too.

You can email uts@mcmaster.ca and request firstname.lastname@mcmaster.ca as an alias for your username@mcmaster.ca account, too.

We seem to be approaching another peak in the spam-wave cycle: I'm up to maybe a dozen spam messages in my inbox each day instead of maybe three.

If you haven't yet turned on spam filtering for your account, you probably should.

Please ignore any email warning you that your account has been compromised and asking y ou to run an attached program; it's a ruse to get you to run an attachment which may very well compromise your Windows computer.

Mail from RHPCS will always be identified as coming from me or from one of the other sysadmins personally and we would never ask you to run anything from an attachment.

I've mentioned the spike in spam that we've seen on mathserv in the past few months before. You may take some consolation in knowing that it's not just us. This New York Times article notes that spam volume has doubled since last year and that 90% of all email on the Internet is now spam.

I really, really recommend turning spam filtering on for your account if you have not already done so.

Yes, it's been a bad couple of days for spam. Updating the spamassassin database will help to some degree, but we're considering other options for the near (but not immediate) future.

If you've not activated the spam filter for your Math & Stats account, you really should do so. Even on a bad spam day like yesterday when I got 40 spam messages in my mailbox, more than 600 were caught by SpamAssassin.

You can help the spam filter by training with spam that it missed.

Back in August, I introduced a way for you to help train SpamAssassin by setting aside spam that slips past the filter and lands in your inbox. After a particularly spammish yesterday, with twenty spam messages staring at me over my morning coffee, I found myself wondering if it's really worth the bother. Short answer: yes it is.

I've got details below, but to those who are going to get on with their lives at this point, I will just say please do file those messages into Spambox-learn - it makes a difference.

In addition to setting aside missed spam for nightly SpamAssassin training you can file false positives in a mail folder called Spambox-falsepositives; these messages will be used to train SpamAssassin to recognize "ham" (that is, not-spam).

Short version:

Spammers adjust their tactics to try to get around spam filters. Save any spam which evads the filters in a folder called Spambox-learn so that SpamAssassin on mathserv can use the messages to update its bayesian spam database each night.

Long version: