Temporary disruption Plus service
Incident Report for WeTransfer
Postmortem

On November 27, starting at midnight (UTC) nearly all Plus transfers from web (both EU and US regions) and all transfers from the Mac app failed to complete the upload. We released a fix for the Plus transfers by 12:15 (UTC) and a solution for the Mac App by 15:00 (UTC).

A technical summary of what happened follows. The format of the AWS S3 notifications we use to process file uploads was changed at 00:00 UTC on November 27, and the version number included in the event schema was increased. Our systems then assumed the notifications could no longer be accepted due to being in a newer format than the system can process. This led to uploads being done via POST requests (used for Plus transfers from web and uploads from the Mac App) to process for a very long time and then to fail since the POST upload processing relies on receiving an S3 notification for each uploaded chunk. Upload processing failing and retrying has led to us overloading the S3 bucket we use to store the incoming data, which made uploads via PUT requests (which do not use S3 notifications) less reliable as well.

Once we figured out why uploads were not processing, which was also witnessed by one of our internal system tests, it was quick to repair by letting the system accept the newer notification format as well.

Posted 10 days ago. Dec 07, 2018 - 11:49 UTC

Resolved
This incident has been resolved.
Posted 19 days ago. Nov 27, 2018 - 16:26 UTC
Monitoring
From midnight (UTC) nearly all Plus transfers from web (both EU and US region) and all transfers from the Mac app failed to complete the upload. We now released a fix for the Plus transfers from web, it requires a page refresh if you had the page already open. We’re investigating further to find a solution for the Mac App as well.
Posted 20 days ago. Nov 27, 2018 - 12:19 UTC
Identified
We've identified an issue on our platform where users who are logged into their Plus account are experiencing issues while transferring their files. While we work on a fix, we advise everyone to use the free service as a temporary work-around.
Posted 20 days ago. Nov 27, 2018 - 09:02 UTC