Temporary Connectivity Issues affecting some OpenCRM users

Resolved
Resolved
After 1 day, 2 hours, and 8 minutes

This issue is now fully resolved and all customers are operating normally on our primary servers. A full outage report will be posted separately.

Avatar for
Recovering
After 6 hours and 25 minutes

We have fixed the core issue and all customer systems should now be back online.

As you would expect, we will be monitoring this closely in the days and weeks to come.

We will also be working with our providers at the data centre to establish the exact cause of the issue and put in place safeguards to prevent it happening again. Our incident report will be published here when it is complete.

Avatar for
{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 5 hours and 49 minutes

We have restored connectivity to further customer systems and are continuing to move those who are still affected.

Avatar for
{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 4 hours and 50 minutes

We have restored connectivity to another group of customer systems and are continuing to move those who are still affected. We will continue to monitor the situation.

Thank you again for your patience.

Avatar for
{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 4 hours and 23 minutes

We are continuing to work with our data centre to restore connectivity to all customers.

Although we cannot provide any set time frames for full resolution, we are doing everything we can to get everyone back online.

Avatar for
{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 3 hours and 56 minutes

We are continuing to work with our data centre to implement new servers.

Thank you for your continued patience.

Avatar for
{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 3 hours and 35 minutes

Our team are working to put new servers in place to ensure all customers are able to get back online as quickly as possible.

We are also continuing to monitor those that have already been brought online.

Avatar for
{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 3 hours and 6 minutes

We are seeing stable connectivity on all new servers. Customers are being moved over to these as quickly as we can.

We want to thank you so much for your patience as we address these issues. Rest assured we will be conducting a full investigation with our data centre when the issue has been fully resolved.

Avatar for
{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 2 hours and 38 minutes

We are continuing to put new servers in place and monitor those that have already been brought online.

Thank you for your understanding.

Avatar for
{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 2 hours and 13 minutes

We are working with our data centre to bring some new servers online to restore connectivity to those customers affected.

Although we cannot provide a time frame for this issue to be fully resolved, please know that we (and the good people at AWS) are doing everything in our power to do so.

Avatar for
Identified
After 1 hour and 46 minutes

This issue at our data centres has been identified and we are currently working toward a solution.

Thank you again for your patience, we know that this has been disruptive for those customers affected.

Avatar for
{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 1 hour and 23 minutes

Thank you for your patience. Our engineers are working with those at our data centre to resolve the connectivity issues some of our customers are experiencing.

We will continue to post updates here. Thank you again for your understanding.

Avatar for
{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 1 hour and 1 minute

A small number of customers may currently be experiencing intermittent connectivity issues to their OpenCRM systems.

We apologise for any disruption this is causing to our customers. Rest assured that we are investigating this as a priority.

Avatar for
{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 32 minutes

We are currently continuing to investigate connectivity issues affecting a subset of customers. Our engineers are working on this as a priority and we will post updates here with any further information.

Avatar for
{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 8 minutes

Affected systems have been failed over to secondary web servers while we investigate further.

Avatar for
Investigating

A small number of customers may currently be experiencing intermittent connectivity issues to their OpenCRM systems.

We are currently investigating this issue. We will post another update within the next 15 minutes.

Avatar for
Began at:

Affected components
  • Web Servers