Microsoft has released an emergency fix for a year 2022 bug that is breaking email delivery on on-premise Microsoft Exchange servers.
As the year 2022 rolled in and the clock struck midnight, Exchange admins worldwide discovered that their servers were no longer delivering email. After investigating, they found that mail was getting stuck in the queue, and the Windows event log showed one of the following errors.
Log Name: Application Source: FIPFS Logged: 1/1/2022 1:03:42 AM Event ID: 5300 Level: Error Computer: server1.contoso.com
Description: The FIP-FS "Microsoft" Scan Engine failed to load. PID: 23092, Error Code: 0x80004005. Error Description: Can't convert "2201010001" to long.Log Name: Application Source: FIPFS Logged: 1/1/2022 11:47:16 AM Event ID: 1106 Level: Error Computer: server1.contoso.com Description: The FIP-FS Scan Process failed initialization. Error: 0x80004005. Error Details: Unspecified error.These errors are caused by Microsoft Exchange checking the version of the FIP-FS antivirus scanning engine and attempting to store the date in a signed int32 variable.
However, this variable can store only a maximum value of 2,147,483,647, which is less than the new date value of 2,201,010,001 for January 1st, 2022, at midnight.
Due to this, when Microsoft Exchange attempts to check the AV scanning version, it would generate a bug and cause the malware engine to crash.
“The version checking performed against the signature file is causing the malware engine to crash, resulting in messages being stuck in transport queues,” Microsoft explained in a blog post.
Microsoft releases temporary fix
Microsoft has released a temporary fix requiring customer action while working on an update that automatically fixes the issue.
This fix comes in the form of a PowerShell script named ‘Reset-ScanEngineVersion.ps1.’ When executed, the script will stop the Microsoft Filtering Management and Microsoft Exchange Transport services, delete older AV engine files, download the new AV engine, and start the services again.
To use the automated script to apply the fix, you can follow these steps on each on-premise Microsoft Exchange server in your organization:
- Download the Reset-ScanEngineVersion.ps1 script from https://aka.ms/ResetScanEngineVersion.
- Open an elevated Exchange Management Shell.
- Change the execution policy for PowerShell scripts by running Set-ExecutionPolicy -ExecutionPolicy RemoteSigned.
- Run the script.
- If you had previously disabled the scanning engine, enable it again using the Enable-AntimalwareScanning.ps1 script.
Microsoft warns that this process may take some time, depending on the organization’s size.
Microsoft has also provided steps that admins can use to update the scanning engine manually.
After running the script, Microsoft says that email will start delivering again, but it may take some time to complete depending on the amount of email that was stuck in the queue.
Microsoft also explains that the new AV scanning engine will be version number 2112330001, which references a date that does not exist and that admins should not be concerned.
“The newly updated scanning engine is fully supported by Microsoft. While we need to work on this sequence longer term, the scanning engine version was not rolled back, rather it was rolled forward into this new sequence,” explained Microsoft.
“The scanning engine will continue to receive updates in this new sequence.”
Update 1/3/22: Changed to correct maximum value of int32 variable.
Source: www.bleepingcomputer.com