Be it a flood, electrical failure, ransomware attack or other broad geographic events; we don't know what it is really like to have to restore IT infrastructure unless we have had to do it ourselves.
We look at other people's backup and recovery issues and hope we are smarter or clever enough to keep it from happening to us.
Recovery from a downtime event includes inconvenience, extra work, embarrassment and yes, real pain.
A ransomware attack is a good example.
Unitrends—an American company specialised in backup and business continuity solutions—recently shared with us a real cyber-attack incident happened with one of their customers to describe the required steps they took to recover functionality following a CryptoLocker attack against a US city.
Also, how it cost city's Governance team days of production and hundreds of man-hours to recover.
The Challenge
Issaquah is a small city of 30,434 people in Washington, United States. According to Forbes, they are the 2nd fastest growing suburb in the state of Washington.
John T, IT Manager leads a team of five employees who execute all IT initiatives co-developed with the city's IT Governance team. John's team manages all technology, from phones, networks, servers, desktops, applications and cloud services.
The city has only two IT staff dedicated to infrastructure.
"We are spread so thin that logs are not monitored consistently," reports John. "We are slowly recovering from a decade of underinvestment in IT and have a large backlog of software, hardware and network upgrades."Part of that underinvestment is that they continued to rely on a tape drive that was ten years old using Backup Exec.
They continued to stumble along until they were hit with a CryptoLocker ransomware attack.
The Infection
Here below find the complete story shared by John with us:In the final analysis, we believe the ransomware attack originated from a "drive-by" where a single city employee visited and opened a .pdf file that had been compromised on a grant coordination site run by a non-profit. This is not an uncommon risk—a small company or organisation website that doesn't have IT funding to keep up with the security risks in today's lightspeed world.
Most entries in the User's Log file were harmless, though the way this virus worked, it could have been downloaded at any time but still needed to be executed by the user. It could have been sitting on the hard drive for weeks (looking like a .pdf) before being executed, though we would need to interview the user to see if she remembers anything like this. This ransomware appeared to disable our anti-virus systems, and is known to remove all traces once finished.
This virus ran only in PC memory and did not turn up on any other devices in our system. It only attacked Microsoft Office, image, .pdf, and text files in folders on the user's PC and file shares to which the user had to write access. It stopped encrypting files once the PC was restarted in safe mode. The lack of propagation could have been a result of either the virus being designed to reside solely in memory to prevent triggering alarms or because our anti-virus software intercepted it at other devices as it attempted to propagate.
The physical server that hosted the file also hosted five critical virtual application servers. After careful analysis, it was determined these were not compromised. We immediately moved these virtual machines onto a different host. This was done prior to kicking off the server restore to reduce processor and NIC load on the file server host.
When we began the file server restore process it quickly became apparent it would take a long time… four days as it turned out. A quick analysis revealed we had no other options to restore the file server. The backup.exe device did work and never failed or stopped during the restore process. It seems the scale of the restore was too big for the device capacity and it had to chunk the workout, making the process very long.
Fortunately for us, the attack had happened on a Thursday, so only Thursday and Friday office productivity was lost. Even so, our users were very negatively impacted and quite upset (as were we). This led to funding being released to move to a modern backup appliance.
The Real Cost to Recover from a Ransomware Attack
John said senior executives agreed to fund an upgrade to the backup system, and after a vendor selection process, his team chose what it felt was the best combination of features and capacity with reasonable costs.
If the same Ransomware attack occurred today with data backed up on the Unitrends Recovery Series 933S appliance the results would have been much different.
First, the attack would have been discovered very quickly as all Unitrends appliances include predictive analytic software and machine learning that will automatically recognise the effects of ransomware on backup files.
An email would then automatically be sent to administrators warning of the attack and identifying the affected files. Then the disaster recovery plan they had in place would be executed.
Secondly, deleting, reinstalling affected files and restarting affected servers would take minutes, not hours and probably not four days.
Critical applications could have been spun up instantly on the backup appliance using the last good backups made before the infection. This would greatly limit the negative impact on employees and office productivity.
The Results
There have been several backup and recovery incidents since the Unitrends Appliance was installed, reported John.
"We have used our backup appliance to recover files that were accidentally deleted by end users. We had also used it to recover virtual machines when we had a host system failure. The downtime in the latter case was limited to staff response time as the mission-critical backup VM was up in less than five minutes!"
"We also plan on moving to the cloud very soon since the Unitrends appliance comes with integrated cloud software. The biggest benefits we expect to see from the cloud are low-cost off-site storage, the ability to recover applications in the cloud if needed as a DraaS feature, and access from anywhere in case of a natural disaster type emergency."
"We now have peace of mind knowing that we can recover quickly when needed. We also have increased shared team knowledge on backup and DR with the easy-to-use user interface."
