Investing in big data could improve research quality in Australia

The Productivity Commission is running an inquiry into whether the government can provide better access to the data it collects for universities and other groups.

Commercial and public sector innovation alike can benefit from a better policy framework for data collection and publication.

If access to datasets and the expertise to design, implement and extract information matter to business and government, this will also inform new research.

Australia requires serious investment in this domain.

We need only look to the achievements of nations such as Sweden and Canada as international leaders in providing multi-generational longitudinal datasets.

Big data is not sufficient; researchers require rich data, able to inform complex logical deductions from large population samples.

The difference between “big” and “rich” data is more than semantic. Not all data can be mined reliably. Educating those who collect data about quality, storage and integrity issues is important if the nation is to benefit from the opportunities.

Academics too have an interest in data that can be interrogated, and consequently in important issues about privacy and fair use of public information. There are serious questions already to debate about integrity of government data.

Why big data is a big deal

Data is a big deal and will be the main focus for many tech companies in 2016.

Thanks to large storage capability through cloud computing, companies have amassed vast amounts of information on consumers, on peak electricity demand, on water flows, on every imaginable subject. The race is now to find ways to mine those bits for commercial advantage.

This year the hottest career prospects in Silicon Valley are for quants – for mathematical majors, engineers, those with the expertise to extract trends and package new opportunities. The international delegations each return home determined to lift enrolments in the quantitative sciences, sensing the shift in priorities for leading-edge companies.

The logic is simple. LinkedIn has millions of curriculum vitae from around the world online. It can track career trends, emerging professions and areas of shortage, covering both the superficial and the profound in job markets. The better the platform’s global coverage, the more opportunity to extract and sell data.

Likewise, companies supporting massive online open courses know more than anyone about student habits, interests and capabilities across the globe. In the inevitable move from free uncredited courses to full curriculum delivery online, they are uniquely placed to shape the emerging market.

Not all applications are commercial. In the public realm, the availability of data can improve markedly the quality of service delivery. A health system with integrated electronic medical records can spot early trends in infection, institute preventative programs and move health professionals to cover gaps.

So quants will be equally in demand by governments. Their ability to make a difference will be influenced, in turn, by the reliability and accessibility of data held by public agencies.

Job opportunities

The expertise required for big data will need many new graduates from data and analytics-based curricula. It will also require collaborative research on a large scale, and so create a new field for universities to work closely with business and government agencies.

To be effective, such collaborative research rests on common standards, shared access and interaction among policy advisers seeking to understand a problem and academic researchers keen to apply and improve data sets.

We can see already the power of big data in policy questions. The Melbourne Institute administers the Household, Income and Labour Dynamics in Australia (HILDA) survey, one of the peak annual instruments for assessing the economic and subjective wellbeing of Australians.

The data collected provide insights on community views on everything from our desire for children to how green spaces, physical activity and urban density in our major cities can affect personal wellbeing.

Last year Professor Philip Clarke used data collected by the Department of Veterans Affairs to look at the disability claims of Vietnam veterans. His research provides one of the first clear pictures of when, how and why veterans seek support. Such analysis can enable new or improved pathways to better public services.

Productivity Commission inquiries do not always attract strong public engagement, but this is a review worth attention.

Improving data quality and accessibility will provide an important platform for Australian business, policy innovation and academic research. It ensures the world experienced in Silicon Valley is not so remote from our shores, not so advanced we can never participate in what already makes up the vast troves of big data.