Workspaces Status Update

Hi everyone,

I wanted to touch base and provide some insight into an ongoing issue that I know has been causing some pain over the last few days, specifically the 'Not Authorized' error showing up in Workspaces.

First, let me assure you we are well aware that this is happening, and we're working hard to reduce the impact, and hopefully make this go away entirely; we don't like this either, but unfortunately we're not in a position to be able to immediately stop the problem, for reasons enumerated below.

Due to the increasing popularity of Workspaces, for the last few months we've been approaching the capacity of the existing Workspaces storage infrastructure. To address this, we investigated a variety of solutions, and as we're swiftly approaching the upper limits of the current system, we are now in the process of re-architecting Workspaces onto a "version 2" storage architecture that will set us up for much more elastic capacity, better performance, and more redundancy going forward. As developers, scale and capacity problems are often referred to as "good problems to have", and that's true; we're hugely honored by the response of the Treehouse community to Workspaces.

In preparation for going live with the new storage infrastructure, on Monday, we began pre-synchronizing existing Workspaces onto the new platform, which has increased load on the already near-capacity "version 1" systems. This was expected, and from the start of this process, we've been aggressively rate-limiting and adjusting CPU and I/O priorities of the synchronizing processes to prevent I/O saturation and prevent an increase in latency.

This has been working fairly well, but the increased load has revealed some unexpected behaviors in one of the key backend Workspace services that we didn't anticipate. Initially, the increase in Workspace start-errors was attributed to the previously mentioned higher system load, but experimentation with increasing the degree of throttling applied to the synchronization process has had very little effect in reducing the occurrence of errors. Once that was eliminated, we began to dig a little deeper, which we've been doing over the last couple of days.

As a result, we've just deployed several mitigations that we believe should help to further reduce the impact of this ongoing process. We will continue to closely monitor Workspaces performance, and take additional steps if necessary to ensure a reasonable quality of service, as we believe the learning process should be as pain-free and pleasant as possible, and we're saddened to witness the frustration the current situation has been causing.

As for the status going forward: we're currently a little less than 50% through the initial synchronization, with an expected completion between Saturday evening and Sunday morning; once that's completed, we'll be continuously running smaller, less impactful re-synchronizations to keep the new storage infrastructure up to date leading up to our final cut-over window in the next couple of weeks (we'll give a day or two of warning ahead of this, so keep your ears open).

Thank you for your patience and understanding; we're really excited about the new system, and we hope that once it's actively serving Workspaces you'll agree that the improved experience will prove worth the transitional difficulties.

If you have any questions, please feel free to ask below, and I'll do my best to address them.

Regards,

Nathan W
Treehouse Site Ops