Disk space on a cluster that queues documents filled due to a large bulk submission. This led to a random, small subset of customers receiving 500 errors upon paper submission for a short amount of time.
We quickly removed stale files, and have followup actions to programmatically manage the storage on these servers so that this issue will not recur.
The failed database hardware was identified and replaced. The service has returned back to normal.