AWS/3rd Party Incident (Limited impact)

post by Michael_I on Nov 26, 2020

There is an AWS incident that is impacting some of our 3rd party integrations. This is affecting our skip processing and resulting in some customer’s schedules failing to rebuild and apply the necessary weather skips.

We know what the issue is and are working with AWS and our 3rd party vendors to make sure they are aware of the issue and tracking their progress as they resolve it.

We will update this thread once it is resolved.

post by Michael_I on Nov 26, 2020

This issue has been resolved and our systems for functioning as expected.

Have a wonderful Thanksgiving evening everyone.

post by Tintn00 on Nov 29, 2020

I’m not sure if these things are related or not. Seems like the problem in this thread started happening the same week. Unfortunately it’s still been slow.

post by franz on Nov 29, 2020

Thanks for the feedback, I will have the engineering team look into this performance impact.

post by franz on Nov 30, 2020

@Tintn00 Are you still experiencing latency with the app?

post by jhhaller on Nov 30, 2020

You might want to consider a multi-region strategy for your AWS account. Storing your backups in another region would allow recovering from backup in a region unaffected by outages in your main region.

post by franz on Dec 1, 2020

@jhhaller Thanks for the advice. In this particular case since we are not in the AWS region impacted our systems were not affected. Our downstream weather service providers were impacted which ultimately impacted us.