21st Century Investment Themes

Big data: an 'industrial revolution' in data

At a glance:

  • Big data is the term given to large and typically unstructured data sets that are difficult to work with using standard database software.

  • 80% of the world’s data is unstructured. Unstructured data is growing at 15 times the rate of structured data.

  • This presents a huge technical challenge for companies trying to analyse their proprietary data.

  • Examples of big data include: search engine queries and results, social network data including tweets, weather data, astronomical data, military surveillance, economic and market data, medical records, physics experiments (Large Hadron Collider) photographic archives, broadcast media, video (CCTV and YouTube), and transaction data.

  • If big data can be effectively captured and analysed, it has the potential to enhance the productivity and competitiveness of a companies in a wide range of sectors.

  • Big data has tremendous potential to add value to both the public and private sector. Tech firms that specialise in information systems are building huge data centres to help capture this market.

  • Retailers and manufacturers are using big data solutions to improve supply chain management.

  • Big data can help company management make better informed decisions and improve corporate governance by creating more efficient risk management systems.

  • There is strong return potential for companies providing big data solutions – these include current technology giants as well as novel start ups and smaller companies. Fundamental research is critical in identifying the key beneficiaries.

A staggering 2.9 quintillion bytes of data are created everyday.*

It is an amount that is difficult to comprehend. To put it into perspective, it is the equivalent of 5.8 billion average-sized PC hard drives. It is even more impressive when you consider that 90% of the world’s data was only created in the last two years1, and 80% of this data is unstructured2, which makes it difficult for traditional database technology to interpret.

This phenomenon has given rise to a new term – big data. Typically complex and unstructured, big data presents a major technological challenge for companies who are at pains to capture, analyse and exploit it. Experts believe we are entering an ‘industrial revolution of data’ with significant rewards available for companies and investors.

Introducing Big Data

Burgeoning amounts of data are being churned out everyday to the extent that current database technologies are rapidly becoming obsolete. Digitalisation has accelerated the growth in data across every organisation, sector and economy. Companies, government agencies, market researchers and meteorologists are waging a technological battle to comprehend big data and extract some of its value. The ability to store, aggregate and combine data has led to numerous technology firms rushing to build data warehouses to meet rapidly-growing demand.

Everyday, posts are made on Facebook, tweets sent on twitter and videos uploaded to YouTube. But social networking is just one of the many drivers behind big data. Networked sensors are collecting huge amounts of data from mobile phones, gas and electric meters, aircraft engines, drilling platforms and atmospheric equipment. Satellites are collecting weather and geographical data and military intelligence. There is ‘exhaust data’ created as by-products from normal everyday activities. And there is transaction data, collected for example by supermarket tills.

As media formats become more advanced, more and more data is being created. For example, one second of high definition video requires more than 2,000 times as many bytes as is required to store a single page of text.3 The creation of all this data is driving rapid technology progress in data storage. In 1980, the world’s first gigabyte-capacity hard drive, the IBM 3380, was the size of a refrigerator, weighed 550 pounds (about 250 kg), and had a price tag of $40,000.4 These days, you can pick up a gigabyte of storage on a postage-sized memory card for only a few dollars.5 And if you have certain email accounts, online storage is actually free (up to a certain level).


“Big data is a key theme within my portfolio. For example, Cisco Systems is a key beneficiary of the need to build out and improve network infrastructure due to the exponential growth in data volumes”

Dmitry Solomahkin, Portfolio Manager, Fidelity Global Technology Fund

Big Data is not just about size

According to IDC, the world’s total digital output was about 180 exabytes in 2006 and grew to about 1,800 exabytes in 2011 - a ten-fold increase in five years (one exabyte = 1,000,000,000 gigabytes). Total data volume is expected to reach 35,000 exabytes in 2020 representing a 20-fold increase in the next ten years. It is not just storage capacity that is constraining big data, however. There are also challenges in the way it is being used and interpreted. To understand big data, we can look at three key characteristics:

Volume, Variety, Velocity

Volume: the exponential growth in data volumes is a key issue driving improvements in communications networks, lead by companies such as Cisco Systems as well as faster broadband speeds, lead by companies such as Virgin Media. However, it is a mistake to believe volume is the only big data characteristic that deserves attention. Arguably, it is the least important in terms of usefulness to companies. Variety and velocity is where the most value add is likely to be found.

Variety describes the number of different data types. Interpreting and analysing different data types in conjunction with each other can create tremendous value. For instance, the social media website, Facebook, stores a vast amount of information about its account holders. It stores different types of information: sex, age, location, martial status, favourite book or films, and even which brands users “like”. Companies can find out who ‘likes’ their brand, the age group they fall under and whether they are more likely to be male or female. Accordingly, they can selectively target this segment in tailored marketing campaigns.

Velocity refers to the shelf life of data because there is little point in storing data that is out of date. For instance, in the US, healthcare providers discard up to 90% of the data that they generate. Much of it is collected from real time video feeds during surgery. Likewise, shops often delete CCTV footage from security cameras after a certain time period has elapsed.

Big Data is set to grow strongly

Big data is expected to drive returns for the next generation of multi-billion dollar software companies. It is the fastest growing market for information systems globally. Analysts believe that the total addressable market is worth $100 billion.6

Companies like General Electric, IBM, Oracle, Microsoft, SAP and Symantec have invested heavily in global data centres designed to interpret big data. These companies are prepared to pay a premium to hire the best talent. Oracle, Microsoft, IBM, and SAP have collectively spent more than $15 billion acquiring software companies that specialise in business intelligence tools7. There are roles for software developers, statisticians, quant analysts, econometricians and data mining analysts. Big data has its own supply chain, allowing companies to benefit whether they specialise in hardware, infrastructure, software or services.

The technological prowess these companies already have is impressive and they already have the capability to create benefits at client firms who seek their services. The data recorded by sensors located on aircraft engines, for example, could be studied to find ways to improve fuel efficiency for an airliner. Data analysis can also benefit the oil and gas industry by helping to find ways to extend the lives of oil and gas fields and the recovery of natural resources. Looking forward, these large tech firms are in a good position to develop lucrative consultancy businesses based on their ability to analyse big data.

Market research firms are also trying to make inroads. The internet has dramatically increased the number of methods of reaching and marketing to an audience because every page viewed and every click made is recorded. Many marketing companies are leveraging off the huge amounts of data available from search and social media leaders such as Google and Facebook.

The sheer amount of data out there can pose a huge problem for companies. The danger is that companies do not rise to the challenge, miss out on opportunities and lose their competitive edge. Large online businesses, such as Amazon.com, already use the data they collect from their customers to suggest products that they might also wish to purchase. However, the setup costs and necessary infrastructure is problematic for smaller companies. An alternative solution is to outsource the data analysis and targeted marketing to external firms that specialise in this field.

Understanding the Technology

NoSQL is widely used by the likes of Google and Amazon. It differs from relational databases that use tables, schemas or rows to store data, and then interpret it using SQL; a query script language. Rather than taking data, defining relationships and storing it in a database for analysis later, NoSQL can analyse data at source. This gives it a real-time edge on existing technologies and makes it more capable of handling large amounts of unstructured data. For example, Google will scan submitted web pages and documents for a key word rather than querying a centralised relational database.

Hadoop is probably the most well-known big data technology. It is an open source software framework that is free and widely available, having been born out of Google and used extensively by Yahoo.

Massive Parallel Processing (MPP) is what has made Hadoop so popular. It uses multiple computer processors running in parallel to analyse data. In the past, large supercomputers were used for this type of analysis. Hadoop’s use of MPP has meant that small businesses can use their network of office PCs to carry out complex data analysis relatively cheaply.


Small companies offer great potential

Although some of the companies mentioned, such as SAP and Oracle, are already considered software giants, many smaller companies and start-ups could enter the mix. The growth potential for big data is immense and there is room for new entrants with innovative solutions. Popular open-source database tools such as Hadoop are readily accessible to start-ups. We have also seen examples of big data start-ups maturing into global firms.

Indeed, the threat of start-ups poses a challenge for large software companies trying to make inroads into this field – as a result, acquisitions made by large technology companies have picked up. A good example is Autonomy, a UK-based software company founded in 1996 in Cambridge. It was bought by Hewlett-Packard in 2011 for $11 billion.8 Following the deal, HP announced a new big data analytics platform - ‘HP Next Generation Information Platform’.

Other small technology firms have seen their share price outperform even though they may not be acquisition targets yet. A good example is Microgen – a small cap UK software company specialising in software that allows companies to run processes in real-time, which would have previously been done in overnight batches. Microgen software was recently tested at 7 billion transactions per hour and is used by many leading businesses including banks, transport and logistics operators and digital media companies. For a leading digital media company Microgen built a royalty management system accounting for millions of songs, apps and other downloads every day.

Some companies manage data for a living

There are also those companies where data analysis is already at the heart of their business model – take Experian for instance. It helps organisations manage credit risk and prevent fraud. On the consumer side it helps individuals manage their credit profile and prevent identity theft. Experian uses its data and analytical tools to sell products to businesses across a wide range of markets, from retailers to financial services, in over 80 countries. It is now expanding in emerging markets such as Brazil; in 2006 it had no exposure to Brazil, yet now the company generates over $1 billion in revenue in that market.


Case study: SAP

SAP is a multinational software corporation based in Germany. It specialises in developing enterprise software that helps manage business operations and customer relations.

The company is well known for its data warehousing solutions. It recently developed a new product called SAP HANA (High Speed Analytic Appliance). One of the frustrations many company managers have is that their traditional hard drive-based systems do not process data quickly enough. Often, insights into business operations are needed as they happen rather than after events have occurred. SAP HANA uses in-memory technology, data compression and parallel data processing to achieve fast, real-time results. It means companies can make quick decisions when they need to be made, giving their business an edge against their competitors.

Looking across the broader business, SAP generates about 19% of new licence revenue from business intelligence and analytics. €1.5 billion is already in the pipeline for their new SAP HANA product – this is impressive for a product that has been on sale for less than a year. SAP’s mobile division also has €800 billion in the pipeline7. This aims to deliver mission critical information to mobile phone users anytime, anywhere and on any mobile device. Both products should be a key area of structural growth for the company.

Big Data can give retailers a competitive edge

Big data can help retailers automate, optimise and fine tune inventory levels based on current demand. Large retailers such as Wal-Mart have already adopted big data techniques to help optimise their supply chain. Wal-Mart’s data warehouse is massive and includes some 2.5 petabytes (a petabyte is one quadrillion bytes) of information, largely from till receipts – it is the equivalent of roughly half of all letters delivered by the US postal services.9 What has given Wal-Mart an edge on its competitors is the way it pioneered the expansion of its electronic data interchange systems to connect its supply chain electronically. It gave its suppliers a ‘retail link’ to view when restocking was required rather than wait for an order from Wal-Mart. These ‘vendormanaged inventory’ techniques are nothing new and have been around since the 1980s, but Wal-Mart has deployed it on a massive scale.

Overall, data mining has been widely embraced by supermarkets. It gives them the ability to monitor customer preferences, analyse buying behaviour and set prices accordingly. Most notably, it has allowed supermarkets to differentiate themselves, offer the right concessions to attract customers and compete on a targeted, price-comparison basis. A recent innovation has been to leverage the data collected by loyalty card programs. UK supermarket chains such as Tesco and Sainsbury’s regularly use the past purchase histories on loyalty cards to tailor promotions to individual shoppers in the form of vouchers.

Other retailers, such as those in the fashion industry, are also making inroads into big data. Inditex for instance, which owns retail-chain Zara, collects data from its till receipts to identify demand for certain garments. It controls most of it supply chain and employs a just-in-time (JIT) production strategy, which prevents unwanted build-up of inventory. It also means that if it identifies an emerging trend, it takes just weeks to develop a product and get it onto the shop floor, while six months is considered the industry average.

Manufacturers use big data to boost efficiency and cut costs

Since the dawn of the computer age, manufacturers have been an early and intensive user of data to help drive quality and efficiency during the production process. However, as data continues to grow exponentially and global competition intensifies, they are under pressure to continually improve performance.

Big data can significantly accelerate the rate of product development. It allows designers and manufacturers to share data quickly and cheaply, and create simulations that test different designs. Both the aerospace and car industry use big data for this purpose.

Car manufacturers in particular have invested heavily in trying to optimise costs across their supply chain using proprietary systems that monitor the price and quality of each part sourced. Toyota, Fiat and Nissan claim to have cut development time by 30-50%. While the payoff is certainly large, huge amounts of money are required to invest in such systems. Not every manufacturer has the budget to build such systems in-house.

Therefore, many manufacturing companies have an incentive to outsource big data analysis to technology firms that specialise in this field. In an environment that is plagued by difficult economic conditions and increasing competition from emerging markets, manufacturers in the developed world are under pressure to cut costs and improve efficiency. Outsourcing big data their big data challenges offers a cost-effective solution.

Big data and the cloud

One of the dilemmas companies face is that they cannot afford the physical infrastructure to interpret big data. Many data warehouse companies now offer cloud-based solutions as part of their product offering. They are promoted to clients as both affordable and accessible.

Essentially, companies rent space on powerful servers that they access online. These servers offer sophisticated software that has been specially designed to handle big data. The benefit to clients is that they can get quick results, often in real-time and it is highly accessible (some companies, like SAP, offer access via mobile phones).

Big Data is not entirely new

NASA’s effective analysis of big data gave it an edge during the space race, especially when they built the Saturn V rockets that took men to the moon.

During the cold war both the US and the Soviet Union crunched vast amounts of data from military surveillance during intelligence gathering operations.

For decades, daily weather forecasts have been predicted by huge supercomputers.

Bookmakers and casinos have continually developed and improved their statistical models to stay one step ahead of punters.

Sports teams (in baseball, cycling and increasingly in football) employ analysts and statisticians in the hope of gaining a competitive edge, something that was popularised in the book (and film) Moneyball by Michael Lewis.

“Data is widely available, what is scarce is the ability to extract wisdom from them.”

Val Harian, Google Chief Economist

Big Data can help enhance business management

Big data not only offers immense commercial value for companies, but it can also help improve the business management process. As mentioned, retailers and manufacturers use big data to optimise their supply chain and inventory levels. Big data can also be used to help maximise cash flows and minimise the length of the company’s cash conversion cycle - this is the time span between spending cash during the production process and collecting it from customers.

Big data can also help improve elements of corporate governance by providing certain risk controls. A common problem is that management decisions can be highly ‘ad hoc’ and at times, ill-informed. Companies such as SAP have developed business tools that keep the senior management well informed with real-time data (see case study above).

Making big data available across an entire business has considerable benefits. It can encourage underperforming divisions, for example, to improve without direct management intervention. A common application is by ranking sales targets by division, or even individual employee. Demand is growing for software that can deliver this variety of statistics and performance indicators.

Big data is also being used extensively in the recruitment industry and in HR departments at large companies. Job candidates are screened and CVs searched automatically by specialist software.

The public sector

The public sector generates and stores huge amounts of data. Indeed, big data has been used for some time by public institutions for a wide range of applications form military intelligence to running space programmes. In Europe, many countries have national healthcare systems, and these often tend to be large, complex (if not cumbersome), and expensive. Big data can be used as a management tool to reduce waiting list times and improve the way patient files are retrieved and updated.

In the foreseeable future, government contracts for big data solutions across the public sector will become commonplace. Indeed, big data consultants that specialise on the public sector are likely. This is an area that is very much in it infancy at present but one that offers tremendous return potential.

“Big data is a powerful investment theme and I am invested in a number of potential beneficiaries in my funds. SAP is a core holding; the company should deliver double-digit earnings growth in the coming years as it rolls out innovative new products, such as SAP HANA, and expands its footprint into new markets.”

“There are also interesting opportunities further down the market capitalisation scale. I own Microgen, which is a small UK software company with an innovative product, which allows companies to run processes in real-time that used to be processed in overnight batches. It already supplies companies in the banking, wealth management, energy and entertainment sectors.”

Colin Stone, Portfolio Manager, Fidelity European Opportunities


If big data can be effectively supported, analysed and exploited, it has the potential to enhance the productivity and competitiveness of companies, industries and, ultimately, entire economies. Retailers and manufactures are already using big data to improve supply chain management and speed up the development of new products. Big data is also being harnessed by the senior management of companies to monitor performance and allow effective real-time decision making. The benefits of big data are not exclusive to the private sector – it has the potential to improve public services as well, particularly healthcare.

The return potential for investors lies in identifying the beneficiaries of what seems set to become a multi-billion dollar industry. Stock selection is critical, as the winners will come from today’s tech giants as well as fast-growth smaller companies. Acquisition activity is likely to be a supportive feature of the industry. In big data, there is a big opportunity.


1. IBM, ‘Bringing big data to the enterprise’, [web page] (2012) <http://www-01.ibm.com/software/data/bigdata/>; average size hard drives defined as 500GB in size.
2. P. Zikopoulousa and C. Eaton, Understanding Big Data: Analytics for Enterprise Class Hadoop and Streaming Data, (edn: McGraw-Hill, 2011)
3. McKinsey Global Institute, Big data: The next frontier for innovation, competition, and productivity (June 2011)
4. The Computer History Museum, Time line of computer history, [webpage] (2012) <http://www.computerhistory.org/timeline>
5. Amazon.com, Search: ‘1GB micro SD cards’, [webpage] (2012) <http://www.amazon.com/SanDisk-MicroSD-SDSDQ-1024-A10M-Retail-Package/dp/B000EEZCEG>, accessed 07-Aug-2012
6. BofA Merrill Lynch, Big Data II – New Themes, New Opportunities, [white paper] (20-Mar-2012)
7. FILLimited, 31.07.2012
8. Financial Times, Autonomy shares soar on $11bn HP deal, 19-Aug-2011
9. McKinsey Global Institute, Loc. Cit.

Important information

The value of an investment can go down as well as up so you may get back less than you invested. Overseas investments are subject to currency fluctuations and emerging markets may be more volatile than established markets. Reference to specific securities should not be construed as a recommendation to buy or sell these securities, but is included for the purposes of illustration only. Investors should also note that the views expressed may no longer be current and may have already been acted upon by Fidelity. This information does not constitute investment advice and should not be used as the basis of any investment decision, nor should it be treated as a recommendation for any investment.