MySQL Service will be down for an extended period (up to 24hr) Wednesday January 3rd 2018

Wednesday January 3rd 2018 CSAIL MySQL service on mysql.csail.mit.edu will have a long maintenance window. Service will be unavailable from start of day and last up to 24hr.

Updates will appear here.

UPDATE

2018-01-03 17:20 - Database import has been running much more slowly than tested and is still on going. Staff will be montioring progress into the evening it is possible down time will run into tomorrow morning (this is possible remotely so weather should have no impact)

2018-01-03 22:00 - Database import complete, but errors relating to old passwords

2018-01-03 23:00 - There is (predictably) no way to auto convert old password hashes to new format, nor any reasonable way for users to reset passwords of accounts in this state through the web interface post migration reverting to old server

Details on reversion

Database accounts created on an even older version or MySql carried over in a previous upgrade use a weak password hashing algorithm which has been removed. Approximately 1/2 of the database accounts are in this state. It's unclear why this didn't show up in what we though was a full dry run we did prior to scheduling this, but it didn't.

We've reverted and are back on the old server.

We will be identifying which affected database accounts belong to which CSAIL users and notifying them the need to change passwords so the passwords are rehashed with a newer algorithm which is supported by both the old and new servers. After this we'll need to set a deadline for password changes and reschedule another upgrade.

IMPACT

Many user web pages or other services reliant on this database service will also be degraded or unavailable. When service returns, not user action is required existing configurations will begin working again without modification.

MITIGATION

TIG will attempt to maintain read-only database service for as much of this window as possible. If you can adapt your usage to accept readonly conditions your impact will be minimized.

DETAILS

The mysql server's software and hardware are among the oldest operated by TIG. This upgrade will move to modern software and significantly newer and faster hardware. Due to the age of the MySQL server software it is not possible to use live replication to do a lower impact transition.

We will need to freeze the current database, switching to read only mode, export ('dump') the databases to text files, copy those backups across the network to the new server, then import in the new database format.

Given the number of databases involved the export and import process will take the bulk of the time. Pre-flight testing with nightly back up data supports this.

-- JonProulx - 15 Dec 2017
Topic revision: 04 Jan 2018, JonProulx
 

MIT Computer Science and Artificial Intelligence Laboratory

 

  • About CSAIL
  • Research
  • News + Events
  • Resources
  • People

This site is powered by Foswiki MIT: Massachusetts Institute of Technology