Jottacloud and the reliability of cloud storage

Jun 09, 2021

I use Jottacloud for backing up my personal data to a remote location.

While I've yet to write in detail about my backup setup, I've basically implemented the 3-2-1 backup strategy where I keep my off-site copy at Jottacloud.

Instead of their official client I use the excellent rclone program to periodically mirror my backup data - encrypted, of course.

Missing files

This setup has worked fine for years, but a couple of months ago I noticed strange errors from rclone when trying to download some files from Jottacloud:

2021-04-25 21:23:33 ERROR : my/file: Failed to copy: \
  failed to open source object: error 404: \
  no.jotta.backup.errors.PhysicalFileNotFoundException: \
  File unavailable directly on dfs bricks; \
  dfs=dfs-046, relativePathOnDfs=/404/<user-id>/<some-id>/<checksum> (Not Found)

I didn't think much of it at first as cloud storage availability is typically on the low end, especially for consumer-level services. However, when I retried the same download a few days later I got a different message:

2021-04-27 16:33:10 ERROR : my/file: Failed to copy: \
  failed to open source object: error 404: \
  MissingChecksumException{username='<user-id>', checksum='<checksum>'} (Not Found)

Still failing. Did Jottacloud actually lose my files? But how is rclone finding these files in the first place, if they're missing? Testing with rclone ls and the Jottacloud web UI confirmed that Jottacloud thinks the files exist, but any attempt to read them failed.

This is the worst kind of failure for a storage system. The file is clearly corrupt, or non-existent, depending on how you interpret the errors above. And the filesystem is lying about the file's existence. I figured it was time to contact support.

Investigation

I notified Jottacloud about this issue in late April. After a few emails back-and-forth over the next weeks I was told that their developers were digging into the issue.

A month after I initially reported the issue I finally received a conclusion from support. Jottacloud had found 50 files that were corrupted and permanently lost, and attached a CSV file listing the affected files.

I was told the unreadable files were stored on old disks with a legacy setup, which are in the process of being retired. As an outsider I don't know much about the Jottacloud architecture, but they could be referring to the system described in this blog post from 2010.

In any case they apologized profusely for losing my data and promised to improve their service and routines in light of these issues.

Trust, but verify

Since I had a local copy of the corrupted files, I could just re-upload them. But how do I know that my other files are intact? Do I trust the CSV file I received from support to be complete?

Luckily, rclone has a check command that can compare local and remote files and report any differences.

Running rclone check for all my files stored on Jottacloud found another 30 files corrupted. So that's 80 corrupted files in total.

Takeaways:

Keep local copies of your files, preferably two, on different mediums.
Cheap cloud storage is a great match for backups, but you cannot trust it blindly.
Verify your backups periodically.