APRIL 4 UPDATE - 4:25PM PT
We’ve got even more good news today as well. After bringing the West US datacenter back online and monitoring it closely, we can confirm it is no longer seeing the issue. This means that our NorthCentral US, SouthCentral US, and West US databases are back online and functioning as expected.
While we’re considering this issue resolved, we will continue to monitor our all of our datacenters to ensure this does not impact any other locations or players. If you see any remaining cases of warping or hitching, please share a video of it happening and a include a link to the match here on Waypoint.
Thank you again for your patience as we worked through this over the past two weeks. It was a tough one to track down, solve, and apply in the live environment, so we truly appreciate you hanging in there with us.
APRIL 3 UPDATE - 7PM PT
Today comes with good news! Let’s jump straight into it by location so you can see how things have progressed.
- NorthCentral US: Rollback completed yesterday. We’re seeing servers continue to perform exceptionally at this location. - SouthCentral US: Rollback completed today. While we anticipated this to take until Saturday, our partners found and expedited an alternative solution which has resolved the issue at this datacenter. - West US: Rollback completed today. This finished as planned today, but because it completed right before peak hours we opted to bring this location online tomorrow morning. In doing so, we ensure West US players can jump in without any issues tonight and allow the datacenter time for a more smooth return (and scaling) tomorrow.We’ll follow up with another update once the West US datacenter comes back online and we can look into its performance.
APRIL 2 UPDATE - 8PM PT
The rollback continued today and progressed enough for us to turn our NorthCentral US datacenter back on. Unfortunately, it has not completed in our other datacenters. Here’s a status of each of our impacted locations:
-
NorthCentral US: Rollback completed. The location has been brought back online and we’re already seeing servers perform well. - SouthCentral US: Rollback in progress. Our partners anticipate that this datacenter will finish rolling back on Saturday. While only a fraction of the matches hosted in this location were encountering the issue, you may still experience lingering funkiness until everything is complete. - West US: Rollback in progress. Our partners anticipate that this datacenter will finish rolling back tomorrow. Once it is completed, we’ll bring it back online and monitor the performance of its matches.We realize that the timing has shifted back since our initial estimates, so we wanted to provide increased visibility on today’s findings. As a heads up, tomorrow our team will be focused on the following for each location:
-
NorthCentral US: Actively monitor the matches hosted here and ensure it continues to perform well. - SouthCentral US: Our focus will be to monitor the rollback with partner teams and ensure we’re both prepared for it to complete by Saturday. - West US: Validate that the rollback completes on the WestUS datacenter, bring it back online, and monitor it closely (as mentioned above)Hopefully breaking things down in this manner helps shed more light on the issue and answers some of the questions you may have.
Please know that we’re working hard to bring everything back to normal and will provide daily updates until we’re able to do so.
We can’t thank you enough for your patience and understanding as we work though this with multiple external teams.
APRIL 1 UPDATE - 5:30PM PT
We worked with partner teams today to ensure the rollback continued throughout the day. Our new estimate is that it will be completed within the next 24 hours. While this has taken much longer than we initially anticipated on Friday, we’re hoping that it will bring a resolution to these issues.We’ll continue to monitor the impacted datacenters and provide another update tomorrow on how the rollback has progressed.
APRIL 1 UPDATE - 12PM PT
Thanks for the continued reports over the weekend everyone. We noticed that our rollback has taken longer than anticipated and has not reached all of the impacted datacenters.We plan on updating this thread again before the end of the day with our progress. Please stay tuned for more information.
MARCH 29 UPDATE - 5:30PM PT
After jumping on the investigation again today with our partners, we believe we were able to identify the root cause. As a result of these findings, we are going to begin rolling back a change that was made at the impacted datacenters. To help set expectations, we anticipate this rollback may take approximately 24 hours to complete and may mean you’ll still see some lingering issues as the transition occurs. That said, please feel free to jump on this weekend and let us know if you’re experience has improved – your feedback will be extremely valuable as we keep an eye on the situation. Thank you for your patience and understanding throughout this process. We’ll follow up with another update on Monday that will speak to whether or not this change has resolved the issue.
MARCH 28 UPDATE - 6:00PM PT
Hey everyone,
We’ve been keeping a close eye on increasing player reports of server issues in Halo Wars 2. After some initial investigating, we were able to identify and resolve a specific issue with last week’s maintenance, but the issue is persisting. The Pub team is engaging with other teams in the studio who have already been investigating this and we are also working with our partners to alleviate this issue. We will continue to investigate and provide updates as soon as they are available.
Thank you for your patience.