Possible issue delivering email to Office 365 mailboxes

Microsoft have closed their service issue now, and our mail queues show mail being delivered successfully. All mail and notifications which were sent during the period of this incident have now been delivered. Should you still have any specific issues please contact our support team.

Further to the previous post, Microsoft have confirmed that some fixes have been deployed. However, the incident is still open and so we will continue to monitor the situation.

We are now seeing our mail queues clear indicating that mail is now getting through to Microsoft accounts. However, Microsoft are still reporting the incident as open so users may still experience a delay in emails and system notifications arriving in Microsoft accounts from OpenCRM. We are continuing to monitor and will provide further updates when we have them.

Microsoft have provided a further update to this problem. They are providing updates approximately once an hour. Hopefully as per below, we will be able to provide some timescales for resolution soon.

"We're progressing with the testing and development stage of our fix, to ensure its efficacy prior to deployment. We expect to be able to provide an estimate time of completion by the time of the next scheduled update."

A further update from Microsoft:

Current status: We’ve identified that a recent service update, applied to a section of infrastructure responsible for enforcing IP address anti-spam rules, contains a change which is inadvertently causing impact. We’re developing and validating a fix to revert to the most recent stable service version to remediate impact. In parallel, we’re continuing to manually add reported affected IP addresses to an allowed list to alleviate impact in the short term.

Microsoft's latest update is as follows. We have already followed the process outlined below and contacted Microsoft for further help. Currently they advise to wait for further updates to the incident.

Any emails / notifications which are sent from OpenCRM are currently queued in our mailservers and will be delivered to their destination once this issue is resolved.

"Current status: We’ve identified that specific IP addresses are being unexpectedly limited by our anti-spam procedures, causing inbound external email delivery to become throttled and delayed. We're reviewing if there have been any recent changes to our anti-spam rules to understand why the IP addresses are being limited. In the meantime, we're manually adding reported affected IP addresses to an allowed list to provide immediate relief.

Scope of impact: Impact is specific to some users who are served through the affected infrastructure."

We have identified and confirmed that the issue delivering emails to Microsoft 365 accounts is due to an ongoing service issue with Microsoft 365. We are exploring any possible solutions and will provide updates here.

We are currently investigating a possible issue with OpenCRM delivering emails and system notifications to Microsoft Office 365 mailboxes

To note this is an issue with Microsoft, not OpenCRM, but we are currently investigating the impact for OpenCRM users and any possible mitigation we can do. Details of Microsoft's service status post below.

Exchange Online

Details

Title: Some users may encounter delays receiving external email messages in Exchange Online

User impact: Users may encounter delays receiving external email messages in Exchange Online.

More info: Affected users may see a "451 4.7.500 Server busy" error message.

Current status: We're analyzing sample throttling IPs from simple messages to confirm whether the issue with the portion of SQL infrastructure is causing impact, before we begin formulating a remediation plan

Scope of impact: Impact is specific to some users who are served through the affected infrastructure.

Next update by: Wednesday, October 11, 2023, at 11:00 AM UTC

Are you experiencing this issue? https://admin.microsoft.com/AdminPortal/home#/servicehealth/:/alerts/E= X680695?shdlinksource=3DIncidentMail&flyout=3Dimpactreport