Microsoft BPOS outages cause customers to question Exchange
Connecting state and local government leaders
Microsoft's cloud-based Business Productivity Online Suite network underwent multiple outages last week, according to complaints from customers.
Microsoft's cloud-based Business Productivity Online Suite (BPOS) network underwent multiple outages last, according to complaints from customers using the company's hosted e-mail service.
Hints that something was awry were noted by ZDNet, which starting receiving complaints from angry customers on May 12. Customers using Microsoft's BPOS services described the outages in posts to a Microsoft Online Services forum, ZDNet noted.
"Cause of #bpos mail delays mitigated; message queues are draining; watch the dashboard for updates," Microsoft stated in a post on Twitter this afternoon. A Microsoft spokeswoman confirmed the outages, saying the company will issue a formal statement.
The outages appear to have impacted Exchange Online, with some customers reporting message delays as well as lost e-mails. One Microsoft partner said in a phone interview that messages were delayed up to six hours.
"It's been resolved in that there are not currently delays but it's been in and out since [May 10]," the partner said. "I'm hoping that it [the outages] won't come back up again but I don't know for certain that it is permanently resolved."
Users of Exchange Online fumed on the online forum. "We're a worldwide corporation using this. If it doesn't improve, we may have to go back to in-house Exchange," said one poster.
Said another: "I migrated our company to Exchange Online from in-house Exchange 2003 last October, and I'm sorry to say that I regret everything I ever said about how this would be better. It has been far worse in terms of both performance and reliability. I hate to be so harsh, but I am deeply frustrated. ... We're actively looking at migration paths back to in-house e-mail."
The outages come as Microsoft is looking to promote the next generation of Exchange Online in its Office 365 suite, which was released for beta testing last month. According to the Microsoft spokeswoman, the outages "only affected BPOS and did not affect Office 365 whatsoever."
UPDATE: Late on May 12, Dave Thompson, corporate vice president of Microsoft Online Services, posted an explanation for the BPOS outages on the Microsoft Online Services Team Blog. According to Thompson, the intermittent access reported by BPOS users was the result of issues relating to "malformed email traffic" that occurred on May 10 and 12.
Thompson wrote:
"On Tuesday at 9:30am PDT, the BPOS-S Exchange service experienced an issue with one of the hub components due to malformed email traffic on the service. Exchange has the built-in capability to handle such traffic, but encountered an obscure case where that capability did not work correctly. The result was a growing backlog of email. By 12:00am PDT, the malformed traffic was isolated and the mail queues cleared. The delays encountered by customers varied, on the order of 6-9 hours. Short term mitigation was implemented and a fix was under development.
"At 9:10am PDT today, service monitoring again detected malformed email traffic on the service. The problem was resolved at 10:03am, but users experienced up to 45 minute email delays during this time. A second, but related issue was detected via monitoring at 11:35am PDT, resulting in email stuck in some end users' outboxes. The issue was remediated at 12:04pm PDT. During this time, more than 1.5 million messages had queued on the service awaiting delivery. The backlog was 90% clear by 4:12 PM, but because of this large backlog of email, customers may have experienced delays of as long as 3 hours. We are implementing a comprehensive fix to both problems."
Thompson's post also noted that an unrelated DNS issue on May 12 "prevented users from accessing Outlook Web Access hosted in the Americas, and partially impacted some functionality of Microsoft Outlook and Microsoft Exchange ActiveSync devices." That issue was resolved a few hours later, Thompson said.