Database Administration

Electronic Health Record Software Provider Database Support

It is easy to maintain data on the cloud

Today with cloud based database systems like Amazon Web Services (AWS) Relational Database Service (RDS), and Microsoft RDS, it is easy to maintain backups of important data.  

We have helped clients migrate their databases from local to RDS services.  We have also helped clients with local data, migrate that data to offsite backups. In this post, I want to briefly discuss the importance of backups, and verification of backups.

Not to many years ago we we where contacted to help a major provider of electronic health record software.  This software stored medical records locally in hospitals and medical offices around the United States.  Several of their clients, including some very large hospitals where experiencing database corruption.  This was resulting in lost data, and loss of access to data.

Expert IT Staff is sometimes not enough

Most of these clients had professional IT Staff or at least service providers maintaining their local systems.  However, because they are busy on a day to day basis managing the complexities of their jobs and the many demands on their time, these providers are often not able to see things which are more obvious to an external party.

In this case the proper maintenance of the databases which supported these organizations had been overlooked.  This resulted in potential loss of data and an inability to use the system, which effectively stopped the operation of these businesses.

How big is the risk of poor data management?

In this case the risk of having the poorly managed data was too large to bear.  Had we not been able to restore the data, the costs to the health care providers would have been very large, not only damaging their reputations, but potentially causing the failure of their business entirely.  In fact, though this was a major provider of electronic health care software, the company itself eventually ended up closing its doors entirely.

How could they have mitigated the risk?

There is a lot that can be done to mitigate the risk of data failure. The plan also should be in line with the cost or associated risk.  In this instance in most cases we were able to restore the clients data.  We also proposed plans to prevent this problem from occurring in the future.

Backups

Obviously creating backups is a good idea.  But there is more to it than this.  Most consultants have heard horror stories of when the backups where not working properly. Here are a few simple things to consider.

Snapshots

Keep incremental snapshots, maybe daily for a week, weekly for a month and monthly for a year depending on your needs.  Why are snapshots valuable?  Because if someone did something accidentally or intentionally which no one noticed for a day or two, that "something" would have been backed up and replaced previous backups. Without snapshots it would have been hard to go back to the previous state.  Sometimes proper logging will also allow going back to a certain point in time on a database.

Off site backups

Even huge databases can be backed up remotely using incremental backups, log shipping or wall logs can be forwarded to a remote server giving near instant backups, there are lots of options.

Testing of backups

Periodic testing of backups should be done.  This can even be automated with sanity checks on the backed up database.

Summary

In the case of this electronic health record software provider, they technically aren't responsible for the data of their clients since the data was on site and managed by their clients.  However, if their clients have bad experiences with the software, or those clients have to hire in experts like iBCSCorp to help recover data, then their customers would not be happy with the product.  The cost to repair damage is usually much higher than the cost of mitigating that damage in advance with proper practices.  In this example the clients data was very important, and loss of that data catastrophic, even life threatening. 

The proper practices for securing and maintaining data are beyond the scope of this article.  It seems like weekly if not daily we hear about data breaches and lost data.  This could largely be avoided by having an third party access the situation.  We recommend that you periodically have a third party evaluate your data storage, backup, retention and access policies.