The story you are about to read is a dramatic telling of a tale of data loss; the names and companies have been changed to protect the identity of the data loss victims.
This is the story of Rick. Rick is a Sr. Database Administrator for a well-known consumer products company. Rick is responsible for support, maintenance and performance of the company’s Cassandra databases.
That Thursday was date night and Rick had dinner reservations and tickets to see Aziz Ansari’s stand up show with his wife. He couldn’t wait.
Data loss was the last thing on Rick’s mind this particular day. The company had deployed its Cassandra cluster with a replication factor of three. The probability of all three nodes dying at the same time was almost non-existent. Everyone felt they were well protected against data loss. His focus was performance tuning these days.
For the most part, the day had gone like any other. Rick had supported a routine compliance audit, worked with his director on capacity planning for the upcoming budget cycle, and had created some scripts to support the engineering team’s request for some production data. As the day was nearing an end, Rick noticed that there was an old test data keyspace in one of the Cassandra clusters slowing things down. He quickly decided that he should drop it to improve performance. It was already 5:15, and Rick really needed to get out of the office by 5:30 if he was going to make it home in time to pick up his wife for the show. This wouldn’t take very long, he could do this and then head out. Easy-peasy.
Just as Rick pulled up the CQL command line interface to enter the DROP KEYSPACE statement, Sara from the Product Management team stopped by to tell him how she had just gotten caught up on this season of Game of Thrones, and did he remember that scene with Arya Stark, and blah blah blah. By the time Rick was able to interject and let her know he had to get going and would catch her tomorrow, it was already 5:25.
Oh no!, and he needed to go. Quickly he typed in the name of the keyspace he wanted to drop and voila, there it goes, now he could go. Wait…in his distracted state he accidentally typed in the WRONG KEYSPACE!! He had just dropped two terabytes of product catalog information.
They had replication though, surely he could recover it, right? WRONG! Almost instantly from when he had entered the command, Cassandra had updated the metadata and deleted the storage directory associated with that keyspace, also deleting the keyspace across all the replicas.
It was only a few seconds before he realized what he had done. No….No…NOOOOOOOO!!!! He cried out. Slamming his fists on the table. There it was staring him in the face. He was officially a victim of DATA LOSS, and he had done it to himself.
Moral of the story: Data replicas cannot protect you from yourself. Anyone can become a victim of Data Loss.