Dark Data Management: The Next Frontier in Information Governance

Compliance managers are swimming in a sea of data. Across the enterprise, various types of data, structured and unstructured, current and ancient, sensitive and trivial, pile up in disparate systems like tiny needles in an ever-growing haystack.

Getting a handle on all this data, especially “dark data,” that is, data that is hard to identify and manage is an increasingly important priority for every business – everything from legal contracts, customer proposals, sensitive client data, financial models, and confidential business plans. Without the tools to gain visibility into their data repositories, a simple regulatory or legal request can cost a firm weeks of lost time and productivity.

Data management is a huge challenge for financial services firms and other highly regulated entities because they have so many knowledge workers creating content that they need to identify, manage and dispose of once it is no longer needed.

For several years, the conversation in information technology circles centered on ‘big data’ and how to collect and manage the ever-increasingly volumes of unstructured data across the enterprise. In the last year, the conversation has shifted from being about how to better manage data to how to get the most value from – and insight into -enterprise information. Once data has been collected, the next step is to identify and manage the ‘dark data’ such as files, documents, emails and instant messages, lurking behind every corporate firewall within file shares, Sharepoint sites, and in cloud-based collaboration tools like Box.com, Dropbox, and Salesforce. Today, technology solutions, such as Bloomberg Vault’s File Analytics solution, are helping firms leverage the cloud to manage and analyze data efficiently or more intelligently than ever before.

This was the focus of a presentation that I recently gave at the Gartner ITxpo in Orlando called “Controlling dark data in the new age of compliance”. I discussed how technology is helping organizations safeguard sensitive information and quickly respond to regulatory or legal requests by locating files, documents, and other types of unstructured data that lie behind corporate firewalls and file systems and could potentially pose a regulatory or reputational risk to regulated entities. Beyond that, my presentation also touched on how secure and flexible hybrid-cloud solutions can help firms analyze immense amounts of data efficiently and securely.

In addition to speaking with clients and analysts about managing dark data at the recent Gartner conference, my team also interviewed two experts about how and why identifying and managing dark data is an essential part of an information governance process; Larry Tabb from the Tabb Group and Neil Roberts, the Chief Technologist at Polygon Global Partners. The short video below explains the business and technology challenges that managing dark data presents, as well as the opportunity firms have to capitalize on the data residing across the enterprise.

I believe we’re at a turning point in the evolution of technologies for big data management. We now have technology tools that enable regulated firms to transform the unstructured data across their enterprise into a valuable asset that enables them to make informed business decisions. As we begin 2015, I think this will be the year that firms harness the value and power of cloud-based data analytics.

— Harald Collet, Global Head of Bloomberg Vault

Bloomberg Vault File Analytics