tag:status.revisionassistant.com,2005:/historyRevision Assistant Status - Incident History2023-08-27T14:00:04-04:00Revision Assistanttag:status.revisionassistant.com,2005:Incident/181419752023-08-27T14:00:04-04:002023-08-27T14:00:04-04:00Revision Assistant Scheduled Maintenance<p><small>Aug <var data-var='date'>27</var>, <var data-var='time'>14:00</var> EDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Aug <var data-var='date'>27</var>, <var data-var='time'>10:00</var> EDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Aug <var data-var='date'>14</var>, <var data-var='time'>10:36</var> EDT</small><br><strong>Scheduled</strong> - Revisions Assistant will take an 4 hour maintenance window to perform essential database maintenance on August 27 between 10am and 2pm EST, and the service will be unavailable during this time. We regret any inconvenience this may cause.</p>tag:status.revisionassistant.com,2005:Incident/96044992022-03-26T16:05:23-04:002022-03-26T16:05:38-04:00Database Upgrade<p><small>Mar <var data-var='date'>26</var>, <var data-var='time'>16:05</var> EDT</small><br><strong>Update</strong> - Scheduled maintenance has completed as planned, Revision Assistant is back up and ready to go.</p><p><small>Mar <var data-var='date'>26</var>, <var data-var='time'>16:05</var> EDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>26</var>, <var data-var='time'>16:00</var> EDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>14:54</var> EDT</small><br><strong>Scheduled</strong> - We're going to do a minor upgrade the databases behind Revision Assistant. The upgrades mean the site will be down for a short period of time during the weekend and then making sure that everything is back to normal. But once we're back we'll look and feel and act just the same.</p>tag:status.revisionassistant.com,2005:Incident/65708452021-03-19T11:43:00-04:002021-03-19T11:43:00-04:00Signal checks failed to process<p><small>Mar <var data-var='date'>19</var>, <var data-var='time'>11:43</var> EDT</small><br><strong>Resolved</strong> - This morning our signal check workers weren't starting up properly, which meant that students would see the "Signal Check is taking too long to process" message when submitting their work. This has been fixed and signal checks should be processing normally now. Apologies for the disruption.<br /><br />Essays written against Expansion Pack prompts were scoring properly during this time, so some students may not have seen any problems.</p>tag:status.revisionassistant.com,2005:Incident/55813222020-11-16T21:03:42-05:002020-11-16T21:03:42-05:00Signal checks and proofreads failing to process<p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>21:03</var> EST</small><br><strong>Resolved</strong> - Signal checks have been processing as normal for some time, we'll continue to monitor to ensure your essays get feedback as you'd expect.</p><p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>18:21</var> EST</small><br><strong>Monitoring</strong> - The fix has been implemented and working for some time with stability. We'll continue to monitor, and later this evening we'll ensure all the essays submitted today have scoring and feedback.</p><p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>16:49</var> EST</small><br><strong>Identified</strong> - Signal checks are not processing right now, we're having some problems with the infrastructure around processing essays for scoring and feedback. We think we've identified a fix and are working on implementing it now.</p>tag:status.revisionassistant.com,2005:Incident/32580482019-11-26T15:23:49-05:002019-11-26T15:23:49-05:00Signal checks and scores for some prompts failing<p><small>Nov <var data-var='date'>26</var>, <var data-var='time'>15:23</var> EST</small><br><strong>Resolved</strong> - Signal checks and scores are processing normally now, and the backlog of work has been cleared out. Thanks for your patience!</p><p><small>Nov <var data-var='date'>26</var>, <var data-var='time'>13:10</var> EST</small><br><strong>Monitoring</strong> - We've implemented a fix so that signal checks are now being processed. It'll take a while for us to work through the essays previously submitted, but eventually every essay should get a score and feedback. We'll continue to monitor their progress throughout the afternoon.</p><p><small>Nov <var data-var='date'>26</var>, <var data-var='time'>11:33</var> EST</small><br><strong>Investigating</strong> - We're aware that signal checks and scores for some prompts are failing. We believe we've identified a cause of the issue but are continuing to investigate to get a complete picture.</p>tag:status.revisionassistant.com,2005:Incident/32040832019-11-17T07:57:05-05:002019-11-17T07:57:05-05:00Database Updates<p><small>Nov <var data-var='date'>17</var>, <var data-var='time'>07:57</var> EST</small><br><strong>Completed</strong> - Scheduled maintenance has completed as planned, Revision Assistant is back up and ready to go.</p><p><small>Nov <var data-var='date'>17</var>, <var data-var='time'>07:00</var> EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'>11</var>, <var data-var='time'>11:32</var> EST</small><br><strong>Scheduled</strong> - We'll be applying routine database updates to the Revision Assistant system at this time, and we'll post updates here once we're done.</p>tag:status.revisionassistant.com,2005:Incident/22037382019-02-12T08:45:24-05:002019-02-12T11:36:04-05:00Unplanned Downtime<p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>08:45</var> EST</small><br><strong>Resolved</strong> - This morning from approximately 8:00 am to 8:40 am (EST), Revision Assistant experienced downtime while our teams worked to fix a security issue. During that time, students working in Revision Assistant received error messages or were unexpectedly logged out of the system. The security issue has been resolved and Revision Assistant is working properly again. We apologize to any students and institutions who were affected by this unplanned maintenance. We take user experience very seriously and will work harder to make sure this does not happen again in the future.</p>tag:status.revisionassistant.com,2005:Incident/20976762018-12-13T18:13:44-05:002018-12-13T18:13:44-05:00Signal Checks are Failing<p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>18:13</var> EST</small><br><strong>Resolved</strong> - Essays that did not receive signal checks during this afternoon's outage have been proccessed. To view Signal Checks, users should refresh their browser webpage, or log out and back in. We appreciate your patience.</p><p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>17:28</var> EST</small><br><strong>Update</strong> - Most assignments have had signal check functionality restored, but we are still receiving reports of failed signal checks from a few users, and working to identify the source of that problem. Users who received failed signal checks during the core outage period will have their drafts evaluated. This process is ongoing. Users may need to refresh the page in their browser, or log out and back in.</p><p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>15:32</var> EST</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>15:32</var> EST</small><br><strong>Identified</strong> - We are experiencing an issue where assignments are unable to receive Signal Checks. We have identified the cause, and are implementing a fix.</p>tag:status.revisionassistant.com,2005:Incident/18627292018-08-23T01:00:04-04:002018-08-23T01:00:04-04:00Database Upgrades<p><small>Aug <var data-var='date'>23</var>, <var data-var='time'>01:00</var> EDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Aug <var data-var='date'>22</var>, <var data-var='time'>21:00</var> EDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Aug <var data-var='date'>22</var>, <var data-var='time'>21:00</var> EDT</small><br><strong>Scheduled</strong> - We're going to upgrade the databases behind Revision Assistant before everyone gets back to school. (Even though some of you are already writing away!) The upgrades mean the site will be down while we're doing the work and then making sure that everything is back to normal. But once we're back we'll look and feel and act just the same.</p>tag:status.revisionassistant.com,2005:Incident/15642422018-01-14T12:01:36-05:002018-01-14T12:01:36-05:00System upgrades<p><small>Jan <var data-var='date'>14</var>, <var data-var='time'>12:01</var> EST</small><br><strong>Completed</strong> - We've completed our work so you can get back to writing! We're going to be keeping a close eye on things, but as always if you find any issues please contact our support crew via https://help.revisionassistant.com/
<br />
<br />Thanks for your patience!</p><p><small>Jan <var data-var='date'>14</var>, <var data-var='time'>08:09</var> EST</small><br><strong>In progress</strong> - Updates are currently ongoing and Revision Assistant is in maintenance mode. We'll post an update once they're complete.</p><p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>15:24</var> EST</small><br><strong>Scheduled</strong> - We're taking Revision Assistant offline to refresh many of our internal systems with needed and useful updates. Once we're done and Revision Assistant is back online everything should work just like it did before.</p>tag:status.revisionassistant.com,2005:Incident/13152492017-08-03T23:53:40-04:002017-08-03T23:53:40-04:00Database upgrades<p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>23:53</var> EDT</small><br><strong>Completed</strong> - We're done! Revision Assistant is back up and running and in tip-top shape, ready for the new school year.</p><p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>19:00</var> EDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Aug <var data-var='date'> 2</var>, <var data-var='time'>17:04</var> EDT</small><br><strong>Scheduled</strong> - We've been doing lots of work on Revision Assistant to prepare for the new school year, and updating our infrastructure is no exception. We'll take Revision Assistant offline to make sure upgrades happen smoothly. And once everything looks tip-top we'll post an all-clear message here and you can get back to your regularly scheduled writing.</p>tag:status.revisionassistant.com,2005:Incident/11420202017-02-28T18:41:44-05:002017-02-28T18:41:44-05:00Problem with Amazon Web Services Preventing Revision Assistant from Loading<p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>18:41</var> EST</small><br><strong>Resolved</strong> - Amazon has resolved the issues they were having with S3 and Revision Assistant is back in fully working order.</p><p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>14:55</var> EST</small><br><strong>Monitoring</strong> - Current status: the Revision Assistant website does load, just verrrrrry sloooooowly. If you stick it out and let the site load (which might be a minute or two) you should be able to login and work without substantial issues. Some resources (like rubrics and assignment PDFs) will fail to load, but you can write and get signal checks.
<br />
<br />For their part, Amazon has said they've identified the root cause and are continuing to work on stabilizing the system.</p><p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>13:25</var> EST</small><br><strong>Investigating</strong> - Well, shoot. A few minutes ago, our team discovered that a problem with Amazon Web Services (AWS S3) is preventing our users from being about to log in to Revision Assistant. We use AWS S3 as a bulk repository for a variety of the documents and files that keep Revision Assistant running smoothly. We did figure out, however, that users who were already logged in when AWS S3 threw its tantrum would still be able to perform most tasks in Revision Assistant except for viewing our Rubrics and Assignment PDFs. Unfortunately, we do not have a quick fix for this sudden Amazon hissy fit and many other websites that use this service are also experiencing the same problems. When we figure out a solution you will be the first to know. (You can also follow their progress at https://status.aws.amazon.com/)
<br />
<br />Until then, Amazon Web Services is in time-out.</p>tag:status.revisionassistant.com,2005:Incident/10338512016-12-19T16:43:21-05:002016-12-19T16:43:21-05:00Autosave data not stored<p><small>Dec <var data-var='date'>19</var>, <var data-var='time'>16:43</var> EST</small><br><strong>Resolved</strong> - We've determined that student writing wasn't being autosaved from early Sunday morning through Monday about 3:30 PM EST. Students who submitted signal checks had their work saved, but students who didn't lost work. Initial investigations show that this was due to an automated database upgrade early Sunday morning, but the autosave service didn't adapt correctly. Alerts were raised but not caught and responded to by the engineering team. We'll be looking into fixing the root cause as well as improving our alerting.
<br />
<br />We take data loss like this very seriously and recognize that it has a big impact on students. Fixes to the root cause will make sure that we store student work if at all possible so we can at least recover it later.</p>tag:status.revisionassistant.com,2005:Incident/9819772016-11-07T11:09:42-05:002016-11-07T11:09:42-05:00Main Revision Assistant site not responding<p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>11:09</var> EST</small><br><strong>Resolved</strong> - For a little while this morning (10:52 AM EST - 11:05 AM EST) users trying to access https://www.revisionassistant.com/ received a cryptic error message staging 'Access Denied'. The problem has been identified and fixed.</p>tag:status.revisionassistant.com,2005:Incident/7484052016-08-26T09:06:13-04:002016-08-26T09:06:13-04:00Revision Assistant service instability<p><small>Aug <var data-var='date'>26</var>, <var data-var='time'>09:06</var> EDT</small><br><strong>Resolved</strong> - Additional monitoring has shown that the server components have remained stable. We are continuing to investigate root causes and will apply changes as needed.</p><p><small>Aug <var data-var='date'>25</var>, <var data-var='time'>12:03</var> EDT</small><br><strong>Monitoring</strong> - Starting at approximately 10:04 AM (EST) today a number of services comprising Revision Assistant failed to respond to requests. This prevented logins until approximately 10:28 AM (EST), and other services (such as listing assignments or autosaving student data) had smaller windows of downtime that we're still identifying. The system is currently stable but engineers are monitoring for additional problems as well as looking into root causes.</p>tag:status.revisionassistant.com,2005:Incident/3821922015-12-20T10:54:16-05:002015-12-20T10:54:16-05:00Planned maintenance<p><small>Dec <var data-var='date'>20</var>, <var data-var='time'>10:54</var> EST</small><br><strong>Resolved</strong> - RevisionAssistant was offline on December 20 from 8:00 AM to 10:30 AM (EST) for planned maintenance. The system is back online and functioning normally.</p>