December 2008 Archives

The file-server problem is corrected as of 10:15 am and all services are operational again.
The mail filters were not operating properly for many people from midnight to 10am today, so you find a strong flavour of spam in your inbox (200+ messages in under 12 hours, in my case).

Email, login and workstation services are disabled while we work on a problem with the file server disk arrays. Most web sites are still up, though some personal sites are inaccessible.

The main file server was spinning its wheels ferociously from ca. midnight to 9:15 am today. And as soon as it was sorted out, the main mail/web server went a little nuts until it was rebooted. As of 9:50 am, everything seems OK. We are investigating.

I brought the compute servers back on line this morning. Workstations seem to be OK, but they are running without local scratch space over the holidays (so that disks errors from unexpected power loss doesn't prevent booting); this would reduce workstation responsiveness under normal conditions, but should not when so few systems are in use.

Since the university is closed, I will be leaving shortly. I will be available via email much of the day, though.

Starting Monday, December 22nd, there will be reduced computer support from RHPCS. From Christmas through New Year, the university is closed and there will be no support. Though if there is a dire problem and you send email to sysadmin@rhpcs.mcamster.ca, I don't promise that someone won't respond.

The standard ms workstations are in fact rebooting when given the shutdown signal. In order to shut your system down, use the "turn off computer" option, wait until you see the Dell logo or the line "Sending kill signal to all processes" and then press the power button.

I recommend that you turn computers and monitors off over the holiday given the risk of power outages and the power spikes that often accompany them. It would not be completely paranoid to unplug your equipment if it plugs directly to a wall jack.

Of course, computers which are turned off cannot be accessed remotely and will not be backed up. Note that most graduate students and post-docs have home directories on the server, which should not lose power.

Facility Services has just announced that there will be a power outage this evening from 5 pm to 7 pm. The main servers and the network will remain up on backup/emergency power, so if you are outside of HH or BSB, web and email will work.

I will be shutting down non-essential servers and office Macintosh and linux systems at 4:50 pm.

I will be turning off email client access (incl. webmail) for a few minutes at a time this morning.

The workstations are able to connect to the file server as of 4:30 pm. All major services are now fully operational as far as my testing shows. Things we be slow this evening while the file systems are being rebuilt, though. Send us email if you see any problems.

Mail services are back on line and ssh logins are no longer read-only. There will be a delay with workstation access while a file-system problem is corrected.

While the primary file server is being upgraded, the following are up:

The announced system downtime has been pushed forward a bit and the systems will go down at 1 pm. Web service will come back shortly thereafter and other systems about an hour later.

While all systems are down due to the network upgrade this Sunday I will be upgrading hardware and software on our primary file server. The file server, email access and workstations will remain down for about an hour after the network comes back up; most web sites will be accessible immediately.

UTS advises that "Cogent, our primary Commercial Internet provider is experiencing routing issues for customers in the Toronto area. Sites normally reached via ORION (such as other universities, google.com, yahoo.ca, cnn.com, etc) are still reachable."

UTS has announced that they will be doing network upgrades to Hamilton Hall between 8 am and noon this Sunday; assume that internal and external network access and workstations will be down for this period.

The ms-workstations went pretty much unresponsive for about two minutes mid-morning and for about ten minutes late this afternoon. These hiccoughs are related to the recent weekend crashes and my attempts to ameliorate things. You may see similar, brief problems again this week, though I am, of course, trying to keep interruptions to a minimum. Your patience as we try to sort out this server problem is appreciated.

If your workstation stops responding or gives strange errors this week, please wait five minutes before rebooting - it will very likely come back to life with all applications and windows still open.

The workstations and mail are functional again as of 10:30 am (other services where up earlier or didn't go down at all).

Our primary file server face-planted early Sunday morning. Email is down but web service is restored as of 9:20 am. All services should be up by 10:00 am.
Efforts to determine the elusive cause will be intensified this week.

Workstations went wonky for a few minutes late this afternoon. Mea culpa - I introduced an network error which affected most workstations while fixing another problem.