Script level upgrade for database 'master' failed because upgrade step 'sqlagent100_msdb_upgrade.sql'

As huge a database grows, that tedious are regular DBA activities become. Besides, a warehouse database, can grow with no bounds to challenge. 

There was a patching on one of our databases, SP3 for SQL Server 2008 R2, which was failing due to the error below. As the error says, MASTER the heart of SQL Server 

“Script level upgrade for database 'master' failed because upgrade step 'sqlagent100_msdb_upgrade.sql' encountered error 2627, state 1, severity 14. This is a serious error condition which might interfere with regular operation and the database will be taken offline. If the error happened during upgrade of the 'master' database, it will prevent the entire SQL Server instance from starting. Examine the previous errorlog entries for errors, take the appropriate corrective actions and re-start the database so that the script upgrade steps run to completion.

Cannot recover the master database. SQL Server is unable to run. Restore master from a full backup, repair it, or rebuild it. For more information about how to rebuild the master database, see SQL Server Books Online.”

The Error 2627, State 1, severity 14 leads us to Primary Key Violation.

So, we decided to find where exactly the error is being thrown. The sqlagent100_msdb_upgrade.sql script is found at the location:

 "C:\Program Files\Microsoft SQL Server\MSSQL10_50\MSSQL\Upgrade". 

After a Service Pack upgrade, the very next SQL Service start would run a set of .sql scripts, which is when it fails. 

So, per our research we tried this work around as given in the KB Article :

1.       Enable trace flag 902 on the instance of SQL Server 2008 R2. To do this, follow these steps:

a.       Open SQL Server Configuration Manager.

b.       In SQL Server Configuration Manager, click SQL Server Services.

c.        Double-click the SQL Serverservice.

d.       In the SQL Server Properties dialog box, click the Advanced tab.

e.       On click the Advanced tab, locate the Startup Parameters item.

f.         Add ;-T902 to the end of the existing string value, and then click OK.

2.       Right-click the SQL Serverservice, and then click Start.

3.       If the SQL Server Agent service is running, right-click the SQL Server Agent service, and then click Stop.

4.       Open SQL Server Management Studio, and then connect to the instance of SQL Server 2008 R2.

5.       Run the following statements:

EXEC sp_configure 'show advanced', 1;
RECONFIGURE;
EXEC sp_configure 'allow updates', 0;
RECONFIGURE;
EXEC sp_configure 'Agent XPs', 1;
RECONFIGURE;
GO

6.       In SQL Server Configuration Manager, right-click the SQL Serverservice, and then click Stop.

7.       Remove trace flag 902 on the instance of SQL Server 2008 R2. To do this, delete ;-T902 from the string value that you updated in step 1f.

8.       Right-click the SQL Serverservice, and then click Start.

9.       Right-click the SQL Server Agent service, and then click Start.

No use, soon after the trace flag is removed, we still get the error. T902 is just to disable the running of upgrade scripts. In the SQL Error Logs, we could get hold of the table where we got the primary key violation, which is sysssispackages.

The error was because, there was meta data for a few 2005 SSIS packages in the table sysdtspackages90, which was being moved to the table sysssispackages, which already had that meta data. Hence, the primary key violation. 

Resolution: Truncate the table sysdtspackages90, if you feel it’s no more needed, or all 2005 packages are upgraded, and restart the services without Trace Flag T902.

We found that the backward compatibility table sysdtspackages90 is no more existing on the SP3.

Note: 1. Take a backup of the table sysdtspackages90 before you take an action on this.

2. When the .sql scripts are being applied during service startup, all databases need to be in consistent state, even User Databases. So wait until all the databases are recovered to FULLEST. Else, you would see an error while logging in, saying Server is in script upgrade mode.