Deleting personal information from databases is harder than it sounds

Since the period for opting out of My Health Record began on July 16, experts in health, privacy and IT have raised concerns about the security and privacy protections of the system, and the legislation governing its operation.

Now federal health minister Greg Hunt has announced two key changes to the system.

First, the legislation will be amended to explicitly require a court order for any documents to be released to a law enforcement agency. Second, the system will be modified to allow the permanent deletion of records:

In addition, the Government will also amend Labor’s 2012 legislation to ensure if someone wishes to cancel their record they will be able to do so permanently, with their record deleted from the system.

But while this sounds like a simple change, permanently and completely deleting information from IT systems is anything but straightforward.

Read more: My Health Record: the case for opting out

Systems designed for retention, not deletion

The My Health Record database is designed for the long-term retention of important information. Most IT systems designed for this purpose are underpinned by the assumption that the risk of losing information – through a hardware fault, programming mistake, or operator error – should be extremely low.

The exact details of how My Health Record data is protected from data loss are not public. But there are several common measures that systems like it incorporate to greatly reduce the risks.

At a most basic level, “deletion” of a record stored in a database is often implemented simply by marking a record as deleted. That’s akin to deleting something on paper by drawing a thin line through it.

The software can be programmed to ignore any such deleted records, but the underlying record is still present in the database – and can be retrieved by an administrator with unfettered permissions to access the database directly.

This approach means that if an operator error or software bug results in an incorrect deletion, repairing the damage is straightforward.

Read more: My Health Record: the case for opting in

Furthermore, even if data is actually deleted from the active database, it can still be present in backup “snapshots” that contain the complete database contents at some particular moment in time.

Some of these backups will be retained – untouched and unaltered – for extended periods, and will only be accessible to a small group of IT administrators.

Zombie records

Permanent and absolute deletion of a record in such a system will therefore be a challenge.

If a user requests deletion, removing their record from the active database will be relatively straightforward (although even this has some complications), but removing them from the backups is not.

If the backups are left unaltered, we might wonder in what circumstances the information in those backups would be made accessible.

If, by contrast, the archival backups are actively and irrevocably modified to permit deletion, those archival backups are at high risk of other modifications that remove or modify wanted data. This would defeat the purpose of having trusted archival backups.

Backups and the GDPR’s ‘right to be forgotten’

The problem of deleting personal information and archival backups has been raised in the context of the European Union’s General Data Protection Regulation (GDPR). This new EU-wide law greatly strengthens privacy protections surrounding use of personal information in member states.

The “right to erasure” or “right to be forgotten” – Article 17 of the GDPR – states that organisations storing the personal information of EU citizens “shall have the obligation to erase personal data without undue delay” in certain circumstances.

How this obligation will be met in the context of standard data backup practices is an interesting question, to say the least. While the legal aspects of this question are beyond my expertise, from a technical perspective, there is no easy general-purpose solution for the prompt deletion of individual records from archived data.

In an essay posted to their corporate website, data backup company Acronis proposes that companies should be transparent about what will happen to the backups of customers who request that records be deleted:

[while] primary instances of their data in production systems will be erased with all due speed … their personal data may reside in backup archives that must be retained for a longer period of time – either because it is impractical to isolate individual personal data within the archive, or because the controller is required to retain data longer for contractual, legal or compliance reasons.

Who might access those backups?

Data stored on archival backups, competently administered, will not be available to health professionals. Nor will they be available to run-of-the-mill hackers who might steal a practitioner’s credentials to gain illicit access to My Health Record.

But it’s not at all clear whether law enforcement bodies, or anyone else, could potentially access a deleted record if they are granted access to archival backups by the system operator.

Under amended legislation, such access would undoubtedly require a court order. Nevertheless, were it to be permitted, access to a deleted record under these circumstances would be contrary to the general expectation that when a record is deleted, it is promptly, completely and irrevocably deleted, with no prospect of retrieval.

Read more: Opting out of My Health Records? Here's what you get with the status quo

Time required to work through the details

In my view, more information on the deletion process, and any legislative provisions surrounding deleted records, needs to be made public. This will allow individuals to make an informed choice on whether they are comfortable with the amended security and privacy provisions.

Getting this right will take time and extensive expert and public consultation. It is very difficult to imagine how this could take place within the opt-out period, even taking into account the one-month extension just announced by the minister.

Given that, it would be prudent to pause the roll-out of My Health Record for a considerably longer period. This would permit the government to properly address the issues of record deletion, as well as the numerous other privacy and security concerns raised about the system.