Historic Incident - NDR notices on Auto-forwarded messages

Resolved
Resolved

Final status: Microsoft has successfully reprocessed all of the impacted messages to mitigate this issue.

Scope of impact: The NDUS Office 365 tenant was affected by this event, and any user expecting to receive an auto-forwarded message sent between Wednesday, May 26, 2021, at 7:30 AM UTC and 1:30 PM UTC, was impacted.

Start time: Wednesday, May 26, 2021, 2:30 AM (7:30 AM UTC)

End time: Monday, May 31, 2021, 9:15 AM (2:15 PM UTC)

Current status: Microsoft is continuing their efforts to reprocess and deliver the affected messages.

Current status: Microsoft's new script is running a little slower than expected and they estimate that approximately 98 percent of the messaged have been delivered. Microsoft now anticipate that all messages will be replayed by Sunday, May 30, 2021, 12:30 PM CDT (5:30 PM UTC)

Current status: Microsoft's method of replaying the impacted events is taking longer than expected, but is progressing. Users should see relief as it proceeds. Microsoft expect to have a timeline of complete mitigation by the next scheduled update.

Current status: Microsoft has corrected configurations in an attempt to replay messages, but their efforts thus far to rerun the command for delivering the remaining messages have continued to be unsuccessful. Microsoft has identified an alternate method of replaying the remaining affected messages, which requires scripts to individually iterate through the messages, but it will take some time to complete.

More info: A recent impact caused auto-forwarded messages to return NDRs. This event was communicated under EX258320 and was resolved by redeploying a spam rule, which had been found to be causing an impact.

While Microsoft has focused on long-term mitigation, these messages can be successfully received when manually forwarded.

Current status: Microsoft is continuing to provision the feature that will enable message replay for the remaining subset of affected infrastructure, though this is taking longer than originally anticipated. Microsoft is monitoring this process to ensure it completes as expected and then they can promptly run the command to deliver the remaining messages.

Recovering

Current status: Microsoft has removed the exception for most of the targeted infrastructure and reran the command to enable the delivery of the formerly blocked messages. While they have confirmed that most users can now see messages auto-forwarded between 7:30 AM and 1:30 PM UTC on May 26, 2021, they are still encountering an exception preventing the delivery of messages in a small subset of infrastructure. To address this issue, they are provisioning a feature that will enable message replay, and will rerun the command in this environment once complete.

Current status: Microsoft has finished running commands throughout the affected infrastructure. However, the emails are not being delivered as they anticipated due to an exception that is blocking the relay. They are verifying that it is safe to remove the exception before re-running the commands that will deliver the remaining messages to users.

Next update by: Thursday, May 27, 2021, 1:30 PM (6:30 PM UTC)

Resolved

NDUS is aware that an unscheduled incident has occurred and systems are now available.

Updates from Microsoft Current status: A new rule is being deployed which should resolve this issue. This should complete in approximately 15 minutes.

Original notice from Microsoft Can't forward external email via the auto-forwarding feature EX258320, Exchange Online, Last updated: May 26, 2021 7:33 AM Estimated start time: May 26, 2021 5:03 AM

User impact Users may receive a Non-Delivery Report (NDR) for external email when forwarded via auto-forwarding feature.

More info: Users may see the following error when attempting to send external email via the auto-forwarding feature: "550 5.7.520 Access denied, Your organization does not allow external forwarding. Please contact your administrator for further assistance. AS(7555)"

Scope of impact: This affects any email being auto-forwarded externally via the feature.

Began at:

Affected components
  • Office 365
    • Outlook