Live System

Like playing with a ball in a moving car – Why you should avoid live database changes

You would be amazed how often we are called out to try and roll back a system that has been taken down by someone deleting or changing key information on a live system. There are some things in your IT where you can just go ahead and make a change. Switching out a printer? Go for it. Want to change the desktop? Help yourself. Feel like making a live change to your key database information or the software that talks to the tables? No, back away from the keyboard, please.

Many businesses need to access a working, active database. Online retailers are an obvious example because they need to show customers stock in real-time. Many businesses rely on access to up to date information. Share prices, ticket sales, room availability, are all excellent examples.

What we are talking about in this article though is live ‘key’ data. The systems we mentioned above are accessing and updating a dynamically changing data set via software designed to do so. While obviously important, the data is not key in the sense that the vital information such as the product details, the buyers’ personal information, the cost of goods, and so on, are not necessarily being accessed. When they are, it is access to a singular set of data such as the bank details of an individual customer. If it goes wrong, it’s annoying, but if you take down the key data for the whole system, for example, the bank details for every one of your customers in the UK, you have a disaster on your hands.

Don’t push the BRB when you are working on a live system

There is a BRB moment for updating any system. BRB stands for Big Red Button. It’s the button you press to make a final change and permanently adopt the update. You see it in action when you get the ‘we will be doing a server update’ style warning from your bank or other providers. They are installing an update that will affect critical systems. Usually, you will even be given a start and an end time. They can so accurately tell you when this will happen because they have already installed the update on a test system, and they know it works. The risk to the business’s actual operation is minor, and the disruption to service is minimal. When they push the actual BRB, all they are doing is watching the change happen and making sure it all goes well.

Smaller businesses need to adopt the same process. When you are making an important change, which you will certainly need to do at some point, you must be sure that the vital data, the nuts and bolts of your daily trade, can be restored. So, regularly you should make easily retrievable backups of your key data and processes such as:

  • Customer data
  • Accounting systems
  • Accounting data
  • Personnel records
  • Any other information that is vital to the running of your business
  • The software that you use to run your operation
  • Any software that attaches to the above and can access it such as CRM portals, apps and intranet functions

 

What constitutes a regular basis will be decided by how your business works. For some companies, it could be a daily run, for others, hourly or even more regular. You need to decide on whether cloud or a more traditional physical back up is better for you. Most importantly, though, you need to make sure that the backup is appropriate and will work with the minimum delay if you need it.

What happens when you work on live data?

Who knows, you may get lucky and nothing will ever go wrong. When it does, again you may be lucky enough to be able to pick up where you left off. On the other hand, you could have a major disaster that will cost you downtime, money and potentially, if you lose personal information, put you in breach of GDPR.

Think of it like repeatedly throwing a ball in the air in a speeding car with the windows open. Most of the time you will catch the ball fine, the one time you don’t though, it probably vanishes through the window and into the trees at the side of the road. You may be able to stop and find it, but if you can’t, the ball is simply gone forever.

Imagine if the lost ball is the field in your database that contains the location of stock in a warehouse. If that field gets deleted with no backup, then the only recourse is to physically recheck the stock and re-enter it all. Even if you did have a backup from the previous day’s data, then you would probably still need to recover all the stock locations for the day’s trade.

What if that was data with a unique reference number (URN) attached or it was encrypted in some way. Without the URN, you would not be able to match the information without starting again. If it was encrypted, it is probably locked out forever.

Without a working backup and proper controls in place, you could find yourself with real problems. For example, as the Home Office recently found out, you could be in a position where you lose thousands of criminal records with no real idea if you can retrieve them.

Call us if you need help with data security. Part of our holistic approach at Datasense is to work with you across all of your IT, security, telecoms, and compliance. Even if something does go wrong, it is a matter of recovery, not a disaster.

Share on facebook
Facebook
Share on twitter
Twitter
Share on linkedin
LinkedIn
Share on email
Email