MySQL Service will be down for an extended period (up to 24hr) Tuesday May 29th 2018

Tuesday May 29th 2018 CSAIL MySQL service on mysql.csail.mit.edu will have a long maintenance window. Service will be unavailable from start of day and last up to 24hr. A previous attempt at upgrade ran from approx 06:00 to 22:00 hours.

UPDATES

2018-05-29 19:30 Upgrade complete see IMPACTS below for possible lingering effects on some older accounts.

IMPACT

Many user web pages or other services reliant on this database service will also be degraded or unavailable. When service returns, not user action is required existing configurations will begin working again without modification.

Some very old user acocunts may be unable to authenticate on the new server, do to changes in supported password hashes (this was hte reason previous upgrade was reverted). TIG has attempted to contact all known humans resonsible for these database users in advance. Not all had identifiable responsible parties and not all those which did responded. If your DB user lands in this predicament simply log into the new https://mysql.csail.mit.edu and change that user's password (or contact help@csail.mit.edu and ask us to do so)

MITIGATION

TIG will attempt to maintain read-only database service for as much of this window as possible. If you can adapt your usage to accept readonly conditions your impact will be minimized.

To ensure you are not bitten by the password hash updates you can login to the current https://mysql.csail.mit.edu and change your DB user's password. This will update the hashing algorithm to one new enough to work on both the old and new systems.

DETAILS

The mysql server's software and hardware are among the oldest operated by TIG. This upgrade will move to modern software and significantly newer and faster hardware. Due to the age of the MySQL server software it is not possible to use live replication to do a lower impact transition.

We will need to freeze the current database, switching to read only mode, export ('dump') the databases to text files, copy those backups across the network to the new server, then import in the new database format.

Given the number of databases involved the export and import process will take the bulk of the time. Pre-flight testing with nightly back up data supports this. Previous import ran until 22:00 the import was complete and successful.

You may rememeber our previous attempt

Wednesday January 3rd 2018 CSAIL MySQL service on mysql.csail.mit.edu will have a long maintenance window. Service will be unavailable from start of day and last up to 24hr.

Updates will appear here.

UPDATE (from January attempt)

2018-01-03 17:20 - Database import has been running much more slowly than tested and is still on going. Staff will be montioring progress into the evening it is possible down time will run into tomorrow morning (this is possible remotely so weather should have no impact)

2018-01-03 22:00 - Database import complete, but errors relating to old passwords

2018-01-03 23:00 - There is (predictably) no way to auto convert old password hashes to new format, nor any reasonable way for users to reset passwords of accounts in this state through the web interface post migration reverting to old server

Details on reversion

Database accounts created on an even older version or MySql carried over in a previous upgrade use a weak password hashing algorithm which has been removed. Approximately 1/2 of the database accounts are in this state. It's unclear why this didn't show up in what we though was a full dry run we did prior to scheduling this, but it didn't.

We've reverted and are back on the old server.

We will be identifying which affected database accounts belong to which CSAIL users and notifying them the need to change passwords so the passwords are rehashed with a newer algorithm which is supported by both the old and new servers. After this we'll need to set a deadline for password changes and reschedule another upgrade.

-- JonProulx - May 22 2018
Topic revision: 29 May 2018, JonProulx
 

MIT Computer Science and Artificial Intelligence Laboratory

 

  • About CSAIL
  • Research
  • News + Events
  • Resources
  • People

This site is powered by Foswiki MIT: Massachusetts Institute of Technology