gartner research on big data

More From Forbes

Gartner's big data definition consists of three parts, not to be confused with three "v"s.

• Share to Facebook
• Share to Twitter
• Share to Linkedin

By Svetlana Sicular

Gartner , Inc.

Volume, velocity and variety characteristics of information assets are not three parts of Gartner’s definition of big data , it is part one, and oftentimes, misunderstood. Most people only retain about one-third of what they read — that explains the truncation.

However, to get to the essence of the definition, an effort to comprehend and retain more than what is limited to a single tweet is well-advised even in our fast-paced time. Especially given that Gartner’s big data definition is not much longer than a tweet:

"Big data" is high-volume, -velocity and -variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision making.

The definition consists of 23 words, 181 characters with quotation marks. The latter is a hint that Gartner believes “big data” will be the new normal in the very foreseeable future. I also like that this definition reflects relativity of big data. I use it in many dialogs with my clients not just to set a common ground, but to point out where big data challenges and opportunities are. This is how I usually explain it.

Part One: 3Vs

Gartner analyst Doug Laney came up with famous three Vs back in 2001. In 2011, Gartner has identified twelve dimensions of data management — all of which interact with each other and confound each other. We have four dimensions of Management & Control and four dimensions of Qualification. The three V’s are the driving dimensions of big data Quantification (there is a fourth too).

The most interesting of 3Vs is variety : companies are digging out amazing insights from text, locations or log files. Elevator logs help to predict vacated real estate, shoplifters tweet about stolen goods right next to the store, emails contain communication patterns of successful projects. Most of this data already belongs to organizations, but it is sitting there unused — that’s why Gartner calls it dark data. Similar to dark matter in physics, dark data cannot be seen directly, yet it is the bulk of the organizational universe.

Velocity is the most misunderstood data characteristic: it is frequently equated to real-time analytics. Yet, velocity is also about the rate of changes, about linking data sets that are coming with different speeds and about bursts of activities, rather than habitual steady tempo. It is important to realize that events in data arise out of the available data and that available data forms its own “social network”. This means that some data serves as a “canary”, other data influences and yet more data results in decisions. When the temporal relationship between two or more data sets changes (more data suddenly becomes less data), then everything else changes, even the definition of a “data event”.

Volume is about the number of big data mentions in the press and social media.

Part Two: Cost-Effective, Innovative Forms of Information Processing

This picture illustrates a typical situation when all problems are labeled as big data problems.

To sort out what can indeed be solved by the new technologies — and this is not one technology — apply part two of our big data definition. Think about technology capabilities to store and process unstructured data; to link data of various types, origins and rates of change; and to perform comprehensive analysis, which became possible for many, rather than for selected few. Don’t expect inexpensive solutions, but expect cost-effective and appropriate answers to your problems.

One of my clients even asked about “big processing of small data.” That counts.

Part Three: Enhanced Insight and Decision Making

Part Three is the ultimate goal. Business value is in the insights, which were not available before. Acting upon the insights is imperative. Missing part three is the most laborious and painful path to the bottom of the Trough of Disillusionment in the Gartner Hype Cycle, especially when parts one and two are present. Other paths to the Trough are also thorny, but necessary on the way to the Slope of Enlightenment.

I tell my clients that their main goals for now are to learn how to identify and formulate big data problems, and to grow their own skills and experience with big data technologies, while these technologies are evolving and maturing. Good solutions are possible although not easy. Just this week I have had several briefings with the companies that deliver unique innovations.  These innovations combined represent the power of big data.  May its force be with you.

Svetlana Sicular is a research director at Gartner, where she focuses on big data, data governance, and enterprise information management strategy. Additional analysis is available in the Gartner Special Report, “Big Data, Bigger Opportunities: Investing in Information and Analytics” at http://www.gartner.com/technology/research/big-data .

• Editorial Standards
• Reprints & Permissions

• Gartner client? Log in for personalized search results.

Insights / Information Technology / Article

12 data and analytics trends to keep on your radar.

5 April 2022

Contributor: Laurence Goasduff

Adaptive artificial intelligence (AI) systems, data sharing and data fabrics are among the trends that data and analytics leaders need to build on to drive new growth, resilience and innovation.

• These data and analytics (D&A) trends will allow you to anticipate change and manage uncertainty.
• Investing in those trends that are most relevant to your organisation can help you meet your CEO’s priority of returning to and accelerating growth.
• Proactively monitor, experiment with or then decide to aggressively invest in key trends based on their urgency and alignment to your strategic business priorities.

Russia’s invasion of Ukraine added a geopolitical crisis to the enduring global pandemic, and managing consequent and persistent uncertainty and volatility will be a key focus for data and analytics leaders this year.

“Now is the time to anticipate, adapt and scale the value of your D&A strategy by monitoring, experimenting with or aggressively investing in key D&A technology trends based on their urgency and alignment to business priorities”, says Rita Sallam , Distinguished VP Analyst at Gartner.

Download now: The IT Road Map for Data and Analytics

This year’s top trends in data and analytics relate to three imperatives:

• Activate diversity and dynamism. Use adaptive AI systems to drive growth and innovation while coping with fluctuations in global markets.
• Augment people and decisions to deliver enriched, context-driven analytics created from modular components by the business.
• Institutionalise trust to achieve value from D&A at scale. Manage AI risk and enact connected governance across distributed systems, edge environments and emerging ecosystems.

12 data and analytics (D&A) trends on the radar in 2022

We have identified the data and analytics trends that represent business, market and technology dynamics that you cannot afford to ignore. These trends also help prioritise investments to drive new growth, efficiency, resilience and innovation.

No. 1: Adaptive AI systems

As decisions become more connected, contextual and continuous, it is increasingly important to re-engineer decision-making . You can do so by using adaptive AI systems, which can offer faster and flexible decisions by adapting more quickly to changes.

However, to build and manage adaptive AI systems, adopt AI engineering practices. AI engineering orchestrates and optimises applications to adapt to, resist or absorb disruptions, facilitating the management of adaptive systems.

Download eBook: 5 Key Actions for IT Leaders to Make Better Decisions

No. 2: Data-centric AI

Many organisations attempt to tackle AI without considering AI-specific data management issues. “Without the right data, building AI is risky and possibly dangerous”, says Sallam. As such, it is critical to formalise data-centric AI and AI-centric data. They address data bias, diversity and labelling in a more systematic way as part of your data management strategy—including, for example, using data fabric in automated data integration and active metadata management.

No. 3: Metadata-driven data fabric

The data fabric listens, learns and acts on the metadata. It flags and recommends actions for people and systems. Ultimately, it improves trust in, and use of, data in the organisation and can reduce by 70% various data management tasks, including design, deployment and operations.

As an example, the city of Turku in Finland found that gaps in its data held back its innovation. By integrating fragmented data assets, it was able to reuse data, reduce time to market by two-thirds and create a monetisable data fabric.

Find out more: Your Ultimate Guide to Data and Analytics

No. 4: Always share data

While data and analytics leaders often acknowledge that data sharing is a key digital transformation capability, they lack the know-how to share data at scale and with trust.

To succeed in promoting data sharing and increasing access to the right data aligned to the business case, collaborate across business and industry lines. This will accelerate buy-in for increased budget authority and investment in data sharing. In addition, consider adopting data fabric design to enable a single architecture for data sharing across heterogeneous internal and external data sources.

No. 5: Context-enriched analysis

Context-enriched analysis builds on graph technologies. The information on the user’s context and needs is held in a graph that enables deeper analysis using the relationships between data points as much as the data points themselves. It helps identify and create further context based on similarities, constraints, paths and communities.

Capturing, storing and using contextual data demands capabilities and skills in building data pipelines, X analytics techniques and AI cloud services that can process different data types. By 2025, context-driven analytics and AI models will replace 60% of existing models built on traditional data.

No. 6: Business-composed D&A

Gartner champions a modular approach to data and analytics, or “ composable D&A ”. Business-composed data and analytics builds on this trend, but the focus is on the people side, shifting from IT to business.

Business-composed D&A enables the business users or business technologists to collaboratively craft business-driven data and analytics capabilities.

Find out more: Everything You Need to Know About Artificial Intelligence

No. 7: Decision-centric D&A

The discipline of decision intelligence, which is careful consideration of how decisions should be made, is causing organisations to rethink their investments in D&A capabilities. Use decision intelligence disciplines to design the best decision, and then deliver the required inputs.

Gartner estimates that by 2023, more than 33% of large organisations will have analysts practising decision intelligence, including decision modelling.

No. 8: Skills and literacy shortfall

D&A leaders need talent on their team to drive measurable outcomes. However, virtual workplaces and the heightened competition for talent have increased the lack of data literacy —the ability to read, write and communicate data in context—within the workforce.

By 2025, Gartner estimates that the majority of CDOs will fail to foster the necessary data literacy within the workforce to achieve their stated strategic data-driven business goals.

As the cost of investing in data literacy and employee upskilling is constantly rising, start inserting “claw-back” or “payback” clauses into contracts with new hires to recover costs in the event that an employee departs your organisation.

No. 9: Connected governance

Organisations need effective governance at all levels that not only addresses their existing operational challenges, but is also flexible, scalable and highly responsive to changing market dynamics and strategic organisational challenges.

However, the pandemic has further highlighted the urgent need for strong cross-functional collaboration and readiness to change organisational structures to achieve business model agility.

Use connected governance to establish a virtual D&A governance layer across business functions and geographies to achieve desired cross-enterprise business outcomes.

Watch Gartner experts discuss: How to Build a Comprehensive Data & Analytics Governance Framework

No. 10: AI risk management

If organisations spend time and resources on supporting AI Trust, Risk and Security Management (TRiSM), they will see improved AI outcomes in terms of adoption, achieved business goals and both internal and external user acceptance.

Gartner predicts that by 2026, organisations that develop trustworthy, purpose-driven AI will see over 75% of AI innovations succeed, compared to 40% among those that do not.

Increased focus on AI TRiSM will lead to controlled and stable implementation and operationalisation of AI models. In addition, Gartner expects far fewer AI failures, including incomplete AI projects, and a reduction in unintended or negative outcomes.

No. 11: Vendor and region ecosystems

Regional data security laws are making many global organisations build regional D&A ecosystems to comply with local regulation. This trend will accelerate in the new multipolar world.

You will need to consider migrating and duplicating some or all parts of your D&A stack within specific regions, and by design or by default, manage a multicloud and multivendor strategy. 

To build a cohesive cloud data ecosystem , consider several actions. Evaluate the extensibility and broader ecosystem offerings of your vendor’s solutions and consider aligning with them. Also, reevaluate the policies favouring a best-of-breed or best-fit strategy for end-to-end D&A capabilities in the cloud by weighing the benefits of a single vendor ecosystem in terms of cost, agility and speed.

No. 12: Expansion to the edge

More D&A activities are executed in distributed devices, servers or gateways located outside data centres and public cloud infrastructure. They increasingly reside in edge computing environments , closer to where the data and decisions of interest are created and executed.

Gartner analysts estimate that by 2025, more than 50% of enterprise-critical data will be created and processed outside the data centre or cloud.

Extend D&A governance capabilities to edge environments and provide visibility through active metadata. In addition, provide support for data persistence in edge environments by including edge-resident IT-oriented technologies (relational and nonrelational database management systems), as well as small-footprint embedded databases for the storage and processing of data closer to the device edge.

Recommended resources for Gartner clients*:

Top Trends in Data and Analytics, 2022

*Note that some documents may not be available to all Gartner clients.

Receive Exclusive Content

5 data and analytics actions for your data-driven enterprise, key success factors in any data and analytics strategy, maximize your data & ai to become a decision-centric enterprise, subscribe to the latest insight.

By clicking the "Continue" button, you are agreeing to the Gartner Terms of Use and Privacy Policy.

• Audit and Risk
• Customer Service and Support
• Technology/Service Providers
• Human Resources
• Information Technology Professional
• Investment Professional
• Legal and Compliance
• Marketing and Communications
• Marketing at a Technology/Service Provider
• Procurement
• Research and Development
• Supply Chain

Please provide the consent below

I have read, understood and accepted Gartner Separate Consent Letter , whereby I agree (1) to provide Gartner with my personal information, and understand that information will be transferred outside of mainland China and processed by Gartner group companies and other legitimate processing parties and (2) to be contacted by Gartner group companies via internet, mobile/telephone and email, for the purposes of sales, marketing and research.

By clicking the "Subscribe" button, you are agreeing to the Gartner Terms of Use and Privacy Policy.

Explore deep-dive content to help you stay informed and up to date

Turn your it strategy into reality with a business-aligned execution blueprint, what if your most human-centric leader is a machine, top practices to enable digital engineering for government organizations, is ai the key to greater software developer productivity, the gartner cybersecurity outlook for midsize enterprises, protect california state & local government it spending and mission delivery, drive stronger performance on your mission-critical priorities..

Microsoft Power BI Blog

• Announcements

Microsoft named a Leader in the 2023 Gartner® Magic Quadrant™ for Analytics and BI Platforms

We are thrilled to announce that for the sixteenth consecutive year, Microsoft has been positioned as a Leader in the  2023 Gartner Magic Quadrant for Analytics and Business Intelligence Platforms . 1 Microsoft has also been positioned furthest to the right for Completeness of Vision, and highest in the Ability to Execute in the Magic Quadrant for the fifth consecutive year.

With Microsoft Power BI, we strive to bring our customers and users the richest and most intuitive business intelligence (BI) experience on the market. We are helped tremendously in this effort by the millions of Power BI community members whose feedback has helped us create and refine so much of what makes Power BI great. Thank you all for your ideas and constant support.

The last year has been a truly exciting period of innovation with hundreds of features and enhancements which help us more effectively empower every individual, team, and organization who wants to get more value from their data.

Empowering every individual

For our engaged and growing community of report builders, we have released a set of features designed to help them do more, faster. First, on the data side, we announced datamarts in Power BI which is a new Power BI Premium self-service capability that enables users to perform relational database analytics and uncover actionable insights from their data. It seamlessly puts the power of SQL databases in the hands of our report builders, while ensuring these datamarts can be centrally governed and managed by admins. We also announced data model editing in the Power BI service which enables users on the web to navigate to a dataset and edit it using a similar model view interface as found inside Power BI Desktop—allowing users to work and collaborate simultaneously on the same data model. On the tooling side, we released a new Optimize ribbon in Power BI Desktop, which includes options to pause and refresh visual queries and optimization presets. Report builders also now have access to contextual on-object controls for Power BI visuals, enabling users to select elements in charts and edit the format or properties of those elements to create stunning, pixel-perfect reports. In October, we released a feature that combined the power of the Microsoft Azure OpenAI Service to create quick measures suggestions for Data Analysis Expressions (DAX), enabling report builders to create DAX measures using natural language instead of using templates or writing DAX from scratch.

Empowering every team

Insights are only impactful when they are in the hands of those who can use them to inform actions and decisions. That’s why we have worked to improve how users share their reports and visuals with their team members. With new integrations across Microsoft PowerPoint, Outlook, OneDrive, SharePoint, Office Hub, and Teams, organizations can turn their Microsoft 365 apps into hubs for uncovering insights and fostering a data-driven culture. Users can now include interactive Power BI reports or even single visuals directly in PowerPoint and include reports as descriptive cards inside of Outlook emails. We also announced the ability to save Power BI files to OneDrive and SharePoint and edit from there directly to help make it easier to collaboratively build reports. To bring Power BI and Microsoft Teams together even further, we’ve released a score of new features including the ability to embed organizational apps in Teams, the ability to share Power BI reports and visuals directly in chats with preview cards, the new Power BI in Teams mobile experience, and so much more. Outside of Microsoft 365, we announced a much-improved cross-tenant sharing experience for datasets, reports, and everything else to help make collaboration easier and more fruitful across organizations.

To help teams align around the same targets, we also announced the general availability of metrics in Power BI —a new experience to help teams keep track of their most important business metrics and objectives in a single pane view. To make this experience even more useful, we released an array of new features including hierarchies; linked metrics; the ability to move, copy, embed, and follow scorecards; the ability to share links to scorecards and individual metrics; a mobile experience for metrics in Power BI; and new Microsoft Viva Goals and metrics in Power BI integrations.

Empowering every organization

For organizations trying to manage the complexity of a data-driven organization, we’ve released several features to help governance and administration. We announced an extension of the Microsoft Purview data loss prevention (DLP) policies for Power BI to support detection of uploading sensitive information such as social security numbers and credit card numbers. We then released a couple enhancements for these DLP policies that included CPU metering for DLP policy evaluation and overriding policy tips for reporting false positives. We also released greater governance controls over My workspaces that allow admins to gain access to the contents of any user’s My workspace, designate a capacity for all existing and new My workspaces, and prevent users from moving My workspaces to a different capacity. For organizations that have Power BI reports with a live connection to Azure Analysis Services (AAS), we published an automated migration tool that simplifies and accelerates this migration experience.

For organizations that use Power BI Embedded to provide dashboards and analytics in their own applications, we’ve released a library of new features including a quick create software development kit (SDK) . This SDK allows organizations to embed an experience in their own apps that auto-generates basic Power BI reports for users based on data they are currently viewing. We also introduced service principle profiles which gives the ability for app owners to manage a much larger number of customers on one service principal. Recently, we announced an exciting new form of Power BI embedded analytics that enables organizations to embed an interactive data exploration and report creation experience in their applications. By using this new set of Power BI client APIs, third-party apps can pass in data source information, and Power BI will automatically generate a dataset and report for users to explore, modify, and if they choose, save to a workspace in their Power BI tenant.

Find out why Microsoft was named a Leader

Check out how companies like Walmart , T-Mobile , and Bayer have used many of these features to turn their data into immediate impact across their teams. And these features only scratch the surface of hundreds of features and enhancements we’ve released over the past year. Follow the monthly features summaries on our Power BI blog and YouTube channel to stay up to date with the latest feature releases.

Get your copy of the  2023 Gartner Magic Quadrant for Analytics and Business Intelligence Platforms ,  and find out why we were named a Leader for the sixteenth year in a row.

1 Gartner;   Magic Quadrant for Analytics and Business Intelligence Platforms; Kurt Schlegel, Julian Sun, David Pidsley, Anirudh Ganeshan, Fay Fei, Aura Popa, Radu Miclaus, Edgar Macari, Kevin Quinn, Christopher Long; 5 April 2023.

Recognized in Magic Quadrant for Analytics and Business Intelligence Platforms, from 2013 – 2022 and Magic Quadrant for Business Intelligence Platforms, from 2008 – 2012.

Gartner is a registered trademark and service mark and Magic Quadrant is a registered trademark of Gartner, Inc. and/or its affiliates in the U.S. and internationally and are used herein with permission. All rights reserved

*This graphic was published by Gartner, Inc. as part of a larger research document and should be evaluated in the context of the entire document. The Gartner document is available upon request from Microsoft.

Gartner does not endorse any vendor, product or service depicted in its research publications and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

Organizations are making rapid strides in deploying decision-making systems based on Artificial Intelligence (AI), and this is underpinned by the ability to tap vast amounts of data across distributed data landscapes. However, poor data quality can be a powerful barrier to maximizing value from AI.

Regulations around the use of AI have emphasized high data quality as a key imperative for the success of AI systems. Reputational damage and missed revenue opportunities are just a few of the consequences of using bad quality data in business decision-making. Advancements in generative AI have further fueled the need for strong data quality management practices to deliver trust in AI outcomes.

A holistic approach to data quality with IBM 

We are excited to share that Gartner recently named IBM a Leader in the 2024 Gartner® Magic Quadrant™ for Augmented Data Quality Solutions for the 17 th time  in a row.

Access the full report here .

We believe this is a testament to IBM’s holistic approach to data quality management to deliver trust in data sources, pipelines and data outcomes.

IBM helps organizations scale AI through the IBM watsonx ™ platform. With IBM data fabric, clients can build the right data infrastructure for AI using data integration and data governance capabilities to acquire, prepare and organize data. IBM helps accelerate data quality initiatives by embedding generative AI to automate and simplify critical data quality tasks.

Adopt the right architecture for data quality

Embracing the right data architecture strategy is critical for effective data quality management. With the right architecture, organizations can design data quality initiatives that deliver not just on accuracy of data but also accessibility, timeliness and relevance.

To deliver accurate data as data volumes and complexity multiply, enterprises require the ability to automate data profiling, conduct data quality analysis and enforce data quality rules. With a data fabric architecture, organizations can use active metadata to gather insights into data across their data landscape. They can act on those insights to deliver the right quality data to the right data consumers in a compliant manner. Advanced data quality SLA rules can strengthen trust in data through efficient monitoring and management of data quality.

Accessibility

As enterprises increasingly adopt data marketplaces to share data products, including data sets and machine learning (ML) models, there is a growing demand for managing data quality to meet service level agreements (SLAs) between data producers and consumers including business analysts, data scientists and business users. A data fabric simplifies the orchestration of data required to build high-quality data products, which can then be published on a marketplace for large-scale data sharing.

End-to-end monitoring of data health across data pipelines, with ML-based anomaly detection, can reduce the time needed to detect and resolve pipeline issues. This requires data observability capabilities to continuously detect and resolve data quality incidents in real-time. It also provides visibility into pipeline quality issues. Also, a data fabric simplifies the delivery of end-to-end data lineage , so organizations can gain visibility into the entire journey of their data, from source systems to end use.

A modern data architecture, such as data fabric, can help deliver the right data for each business use case by providing a shared semantic knowledge layer that helps to enable a consistent understanding of data across the organization. It also helps to enable automation to act on these insights. Data fabric enables in-depth analysis of data relationships and provides automated entity resolution capabilities to improve data quality on a larger scale.

IBM’s approach to data quality

IBM data fabric enables a holistic approach to data quality management with integrated data quality and governance capabilities. With IBM Knowledge Catalog (rebranded from IBM Watson® Knowledge Catalog), Match 360 on IBM Cloud Pak® for Data, Data Quality for AI library or API and IBM® QualityStage® via IBM DataStage , organizations can gain a composable data quality solution within a unified platform that facilitates automated data quality, along with data governance , data lineage and data protection.

The recent acquisition of Manta has further strengthened IBM’s data quality credentials with the ability to deliver greater transparency into data flows and determine whether the right data was used for AI and other decision-making systems. When combined with the data observability capabilities delivered by IBM® Databand ®, IBM offers a holistic data quality approach to help accelerate data and AI outcomes.

With IBM data fabric services and watsonx , enterprises gain access to high-quality, trusted data, whether they build or tune generative AI models or traditional ML models. This is underpinned by a semantic layer powered by generative AI to help organizations discover, understand, cleanse and augment data.

IBM offers a wide range of data quality management capabilities, including data profiling, data cleansing, data monitoring, data matching and metadata enrichment powered by AI/ML.  The unified data quality experience within IBM Knowledge Catalog is designed to accelerate the identification and remediation of quality issues.

IBM continues to introduce new product innovations that simplify the curation of high-quality data for self-service consumption by data consumers. Through AI-powered data quality rules , support for SLA rules to monitor the quality of critical data elements, and intelligent matching algorithm s to deliver a single, trusted view of organizational master data entities, IBM continues to deliver powerful data quality management capabilities to data teams.

If you’re ready to try IBM data governance capabilities, access the free trial . Read the report to learn why IBM is a Leader in the 2024 Gartner Magic Quadrant for Augmented Data Quality Solutions.

Gartner, Magic Quadrant for Augmented Data Quality Solutions, Melody Chien, Jason Medd, 6 March 2024 The report was previously named as Magic Quadrant for Data Quality Solutions

Gartner is a registered trademark and service mark and Magic Quadrant is a registered trademark of Gartner, Inc. and/or its affiliates in the U.S. and internationally and are used herein with permission. All rights reserved.

Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as sta`tements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

More from AI for the Enterprise

Using generative ai to accelerate product innovation.

3 min read - Generative artificial intelligence (GenAI) can be a powerful tool for driving product innovation, if used in the right ways. We’ll discuss select high-impact product use cases that demonstrate the potential of AI to revolutionize the way we develop, market and deliver products to customers. Stacking strong data management, predictive analytics and GenAI is foundational to taking your product organization to the next level.   1. Addressing customer inquiries with an AI-driven chatbot  ChatGPT distinguished itself as the first publicly accessible GenAI-powered…

Building for operational resilience in the age of AI and hybrid cloud

4 min read - Each year we see the challenges that enterprises face become more complex as they strive to keep up with the latest technologies, such as generative AI, and increasing customer expectations.   For highly regulated industries, these challenges take on an entirely new level of expectation as they navigate evolving regulatory landscape and manage requirements for privacy, resiliency, cybersecurity, data sovereignty and more. Organizations in the financial services, healthcare and other regulated sectors must place an even greater focus on managing risk—not…

Maximizing business outcomes and scaling AI adoption with a Hybrid by design approach 

3 min read - For established businesses, the debate is settled: a hybrid cloud approach is the right strategic choice.   However, while embracing hybrid cloud might be intrinsic, clients continually seek to derive business value and higher return on investment (ROI) from their investments. According to a study conducted by HFS Research in partnership with IBM Consulting®, only 25% of surveyed enterprises have reported solid ROI on business outcomes from their cloud transformation efforts.   The lack of ROI progress can be attributed to several…

IBM Newsletters

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Eur J Public Health

Benefits and challenges of Big Data in healthcare: an overview of the European initiatives

Roberta pastorino.

1 Sezione di Igiene, Istituto di Sanità Pubblica, Università Cattolica del Sacro Cuor, e, Rome, Italy

4 Department of Biology, University of Patras, Patras, Greece

Corrado De Vito

2 Department of Public Health and Infectious Diseases, Sapienza University of Rome, Rome, Italy

Giuseppe Migliara

Katrin glocker.

3 Division of Medical Informatics for Translational Oncology, German Cancer Research Center, Heidelberg, Germany

Ilona Binenbaum

Walter ricciardi.

5 Department of Woman and Child Health and Public Health—Public Health Area, Fondazione Policlinico Universitario A. Gemelli IRCCS, Rome, Italy

Stefania Boccia

Healthcare systems around the world are facing incredible challenges due to the ageing population and the related disability, and the increasing use of technologies and citizen’s expectations. Improving health outcomes while containing costs acts as a stumbling block. In this context, Big Data can help healthcare providers meet these goals in unprecedented ways. The potential of Big Data in healthcare relies on the ability to detect patterns and to turn high volumes of data into actionable knowledge for precision medicine and decision makers. In several contexts, the use of Big Data in healthcare is already offering solutions for the improvement of patient care and the generation of value in healthcare organizations. This approach requires, however, that all the relevant stakeholders collaborate and adapt the design and performance of their systems. They must build the technological infrastructure to house and converge the massive volume of healthcare data, and to invest in the human capital to guide citizens into this new frontier of human health and well-being. The present work reports an overview of best practice initiatives in Europe related to Big Data analytics in public health and oncology sectors, aimed to generate new knowledge, improve clinical care and streamline public health surveillance.

Introduction

Data have become an omnipresent concept in our daily lives with the routine collection, storage, processing and analysis of immense amount of data. This characteristic is cross-sectorial, ranging from the domain of machine learning and engineering, to economics and medicine.

Over the last decades, there has been growing enthusiasm of the potential usefulness of these massive quantities of data, called Big Data, in transforming personal care, clinical care and public health. 1

An overview of Big Data definitions

Despite the term Big Data having become ubiquitous, there is no universal definition until now on the use of this term. According to McKinsey the term Big Data refers to datasets whose size is beyond the ability of typical database software tools to capture, store, manage, and analyse . 2 Gartner proposed the popular definition of Big Data with the ‘3V’: Big Data is volume, high-velocity and high-variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision making . 3 According to other definitions, instead, Big Data is also characterized by a fourth dimension: Veracity, concerning the quality, authenticity, ‘trustworthiness’ of data. 4

Furthermore, there is an emergent discussion that ‘Big’ is no longer the defining parameter, but rather how ‘smart’ the data are, focusing on the insights that the volume of data can reasonably provide. 5 This aspect is fundamental in the health sector. The potential of Big Data in improving health is enormous. However, its potential value is unlocked only when leveraged to drive decision making and, to enable such evidence-based decision making, it is necessary to have efficient processes to analyse and turn high volumes of data into meaningful insights.

A specific definition of what Big Data means for health research was proposed by the Health Directorate of the Directorate-General for Research and Innovation of the European Commission: Big Data in health encompasses high volume, high diversity biological, clinical, environmental, and lifestyle information collected from single individuals to large cohorts, in relation to their health and wellness status, at one or several time points . 6

Big Data analytics for health systems

The complexity of Big Data analysis arises from combining different types of information, which are electronically captured. The last years have seen an explosion of new platforms, tools and methodologies in storing, and structuring such data, followed by a growth of publications on Big Data and Health ( figure 1 ). To date, we can collect data from electronic healthcare records, social media, patient summaries, genomic and pharmaceutical data, clinical trials, telemedicine, mobile apps, sensors and information on well-being, behaviour and socio-economic indicators.

Number of publications on ‘Big Data and health’ reported by year (from 2010 to 2018). The publications are identified through a search of MEDLINE with the following terms for the literature search: (‘Big Data’) AND (‘Health’)

Healthcare professionals can, therefore, benefit from an incredibly large amount of data. Recent reports suggest that US healthcare system alone stored around a total of 150 exabytes of data in 2011 with the perspective to reach the yottabyte. 7

Starting with the collection of individual data elements and moving to the fusion of heterogeneous data coming from different sources, can reveal entirely new approaches to improve health by providing insights into the causes and outcomes of disease, better drug targets for precision medicine, and enhanced disease prediction and prevention.

In light of this, opportunities and potential are enormous for the benefit of patients and, in general, of the healthcare system.

The potential benefits of Big Data for healthcare in the European Union

Big Data is a sensitive issue for European Union (EU) institutions: the availability of health-related Big Data can have a positive impact on medical and healthcare functions. EU is faced with several changes that may affect the sustainability of its healthcare system. By 2025 life expectancy is expected to further increase, and this may result in more people living longer, but not necessarily in a healthy and active condition. This will put further pressure on Europe’s healthcare costs and economic productivity.

In this context, the data sharing approach can improve outcomes for patients and evidence-based healthcare decision making as reported during the workshop on ‘Digitalisation and Big Data: implications for the health sector’, held on 19 June 2018 at the European Parliament. 8

The use of Big Data in healthcare, in fact, can contribute at different levels as reported by the Study on Big Data in Public Health, Telemedicine and Healthcare of the European Commission: 9 (i) increasing earlier diagnosis and the effectiveness and quality of treatments by the discovery of early signals and disease intervention, reduced probability of adverse reactions, etc. (sector 1); (ii) widening possibilities for prevention of diseases by identification of risk factors for disease (sector 2); (iii) improvement of pharmacovigilance and patient safety through the ability to make more informed medical decisions based on directly delivered information to the patients (sector 3); (iv) prediction of outcomes (sector 4).

In the next paragraphs, examples of EU initiatives in the four macro sectors are listed.

Big Data have the potential to yield new insights into risk factors that lead to disease. There is the possibility to engage with the individual patient more closely and import data from mobile health applications or connected devices. These data have the potential to be analysed and used in real-time to prompt changes in behaviours that can reduce health risks, reduce harmful environmental exposures or optimize health outcomes.

Finally, Big Data can help identify and promptly intervene on high-risk and high-cost patients. 10 Effective ways of managing these data can therefore facilitate precision medicine by enabling detection of heterogeneity in patient responses to treatments and tailoring of healthcare to the specific needs of individuals. 11 All these aspects should eventually lead to a reduction in inefficiency and improvement in cost containment for the healthcare system.

Examples of Big Data analytics for new knowledge generation, improved clinical care and streamlined public health surveillance are already available. Below we report a selection of best practices in Europe in the public health and oncology fields.

Big Data in public health

Efforts to improve the availability and accessibility of data in the EU appear to be driven mainly by socio-economic purposes. However, great importance is placed on the need of using data and new information and communication technology (ICT) in public health to improve quality of prevention and care.

In the next years, European health systems must respond more efficiently to the exponential increase of chronic patients identifying the most efficient interventions and releasing the full potential of ICT. The e-health platforms that many European governments are trying to implement can be effective in improving the management of chronic patients in the community setting by interfacing between different health professionals and specialists and with the patients. Furthermore, a large part of EU citizens uses the internet looking for information on health and access to health services.

Moreover, Big Data and predictive analytics can contribute to precision public health by improving public health surveillance and assessment, therefore, in a public health perspective, the gathering of a very large amount of data, constitute an inestimable resource to be used in epidemiological research, analysis of the health needs of the population, evaluation of population-based intervention and informed policy making. 9

Many projects across the EU are exploring the potential of available Big Data in a wide range of fields. A systematic review published in 2016 from the European Commission identified at that time 10 priority projects on Big Data implemented in Europe that fall in the four macro sectors described above and are aimed to support the sustainability of health systems by addressing the improvement of the quality and effectiveness of treatment, fighting chronic disease and supporting healthy lifestyles. 9 Some of these projects focussed on gathering a very wide range of data types, from GP records, hospitalizations, drug prescription and laboratory and radiology analyses in order to create comprehensive national data warehouses. Among them, the ‘Decision Support for Health Policy and Planning: Methods, Models and Technologies based on Existing Health Care Data’ (DEXHELPP), the eHealth project in Estonia, the ARNO observatory in Italy and the Hospital Episode Statistics in the United Kingdom. The DEXHELPP project (mainly regarding sectors 1 and 4) used routinely collected health data sources to analyse the performance of the health system, to forecast future changes and to simulate the application of policy and interventions. The Estonian eHealth project (mainly regarding sectors 1, 2 and 3) was more oriented toward the improvement of the quality and efficiency of health services, aiming to digitalize all the information and prescription of each patient. Furthermore, in order to facilitate data collection, they provide an environment called X-Road to which all healthcare providers can link while using their own ICT solutions. The ARNO project (mainly regarding sector 4), was committed to epidemiological research, giving the possibility of deep stratification of the general population. The Hospital Episode Statistics (mainly regarding sector 4) was in charge of the Secondary Uses Service that publishes reports and analyses to support the National Health Service in the delivery of healthcare services.

Beside these projects characterized by a comprehensive approach, other initiatives focused on specific conditions (e.g. chronic conditions, rare diseases and psychiatric disorders) are also available in the EU. 9 Among them, the PASSI surveillance project (mainly regarding sectors 1 and 2) in Italy provides large amount of information on the life style of almost the 90% of the population, enabling to individuate specific targets to implement and assess public health actions. The Shared Care Platform (mainly regarding sectors 1 and 3) in Denmark is focused on chronic patients, aiming to harmonize the course of treatment among health and social care providers. The Spanish Rare Diseases Registries Research Network (SpainRDR) (mainly regarding sector 1) focuses on the development of clinical research on rare diseases, providing the harmonization and unification into one comprehensive platform of pre-existing databases and registries of rare diseases. CEPHOS-LINK (mainly regarding sectors 1, 2 and 4), is a platform dedicated to mental health that involves six EU countries. It is committed to collect data on psychiatric hospital admissions and re-admissions, with the aim of finding determinants of re-admissions and to harmonize the psychiatric care pathways across the EU.

In addition to the projects reported above, the EU’s framework programmes for research and innovation funded a large number of initiatives on Big Data in public health. In table 1 , we list 11 projects funded from the EU between 2012 and 2018 with a contribution over €499.999 that are captured from the Cordis website (source: cordis.europa.eu).

EU supported initiatives concerning activities that involve the use of Big Data in public health in Europe from 2012 to 2018, in chronological order (EU contribution from: 499.999€)

Source : CORDIS, https://cordis.europa.eu/en , retrieved on 05.07.2019.

Query : contenttype=‘project’ AND exploitationDomain/code=‘health’ AND (‘public’ AND ‘health’ AND ‘data’ AND ‘“big’ AND ‘data”’) AND/project/ecMaxContribution>=499999.

Notes : Four projects (iManagerCancer, MedBionformatics, Mocha, Iasis) that involve the use of Big Data in oncology ( table 2 ) result also from the query above.

Big Data in oncology

Cancer is one of the major health problems affecting our society, a situation that is set to deteriorate globally as the population grows and ages. According to the State of Health in the EU reports, cancer is recognized as one of the major contributors to premature deaths in the EU. It also has an impact on the economy in terms of lower labour market participation and productivity. Advances in Big Data analytics are given cancer researchers powerful new ways to extract value from diverse sources of data.

These diverse sources include a huge amount of data for one patient. As cancer is a molecularly highly complex disease with an enormous intra- and intertumoral heterogeneity among different cancer types and even patients, the collection of various different types of omics data can provide a unique molecular profile for each patient and significantly aid oncologists in their effort for personalized therapy approaches. 12

The approach of combining these sources of data is implemented in Comprehensive Cancer Centres (CCCs). 13 One of 13 CCCs in Germany is the National Center of Tumor Diseases, where the Molecularly Aided Stratification for Tumor Eradication Research (MASTER) trial is conducted (mainly regarding sector 1, 2 and 3). In a recent review article, 14 this trial was illustrated as an example of a highly successful programme addressing the molecular profiling in cancer patients. Within the MASTER trial data relevant to diagnostic information of young patients with advanced-stage cancer diseases is collected by performance of whole exome or whole genome sequencing and RNA sequencing, analysed and discussed.

Another example for a success story given in the review is the INdividualized therapy FOr Relapsed Malignancies in children (INFORM) (mainly regarding sector 1, 2 and 3) registry which aims to address relapses of high-risk tumours in paediatric patients. Data from whole-exome, low-coverage whole-genome, RNA sequencing and microarray-based DNA methylation profiling are utilized to identify patient-specific therapeutic targets. The INFORM registry started as a national effort in Germany and has been extended with the participation of eight European countries, as well as Australia.

Next to the described projects there are many other initiatives which focus on the value of Big Data in oncology, the EU alone funds more than 90 projects working on this topic (projects with a funding over €499.999 are listed in table 2 ). A potential umbrella for bringing together national efforts such as those mentioned above at the European level is Cancer Core Europe. 15

EU supported initiatives concerning activities that involve the use of Big Data in oncology in Europe, in chronological order (EU contribution from: 499.999€)

Query : contenttype=‘project’ AND exploitationDomain/code=‘health’ AND ((‘tumor’ OR ‘tumour’ OR ‘cancer’ OR ‘oncology’) AND (‘“big data”’)) AND/project/ecMaxContribution>=499999.

Collaborations are of extremely high importance especially in the case of paediatric or other rare types of cancer, where the data collected for one patient is indeed enormous, however the number of patients a single centre can have access to is too low to obtain statistical power high enough to reach meaningful results. One of the main challenges of these collaborations is the access to the data as well as the opportunity to analyse the huge amount of data in an efficient way. Physicians, researchers and informatics experts can only benefit from collected data and expert knowledge when they get easy and intuitive access to own data or data of partners. For example, at the German Cancer Research Center, tools are developed to grant ways to access and analyse own data together with data from partners.

Additionally, international exemplary approaches of sharing data among partners or the public are done by The Cancer Genome Atlas (TCGA) and the International Cancer Genome Consortium (ICGC) which provide researchers with access to thousands of sequenced patients with different types of cancer. The availability of these data jointly with data by other partners has enabled large meta-analyses and machine learning algorithms, integrating different types of cancer that led to the identification of novel cancer driver genes that belong to specific pathways and can be possible therapy targets. Furthermore, there are a number of public databases that provide access to catalogues of mutations found to be involved in cancer, such as the Catalogue of Somatic Mutations in Cancer (COSMIC).

All these multiple sources of information combined and the establishment and support of CCCs across Europe offer the potential to increase the number of patients that can be offered molecular profiling and individualized treatment based on Big Data analysis.

Ethical and legal issues for the effective use of Big Data in healthcare

The use of Big Data in healthcare poses new ethical and legal challenges because of the personal nature of the information enclosed.

Ethical and legal challenges include the risk to compromise privacy, personal autonomy, as well as effects on public demand for transparency, trust and fairness while using Big Data. 16

Data heterogeneity, data protection, analytical flows in analysing data and the lack of appropriate infrastructures for data storage emerged as critical technical and infrastructural issues that might endanger a Big-Data-driven healthcare.

Recently, Skovgaard et al. 17 explored attitudes among people living in the EU toward the reuse of health data. The review indicates that the use of health data for purposes other than treatment enjoys support among people, as long as the data are expected to further the common good. In this context, the recent call reported in Science from a number of eminent scientists worldwide, for the unrestricted use of public genomic data, finds a fertile ground from the public. 18 Concerns evolve around the commercialization of data, data security and the use of data against the interests of the people providing the data.

The recent EU Data Protection Regulation (GDPR) tries to balance patients’ privacy while ensuring patient’s data can be shared for healthcare and research purposes.

On 23 January 2017 the Consultative Committee of the Council of Europe’s data protection convention adopted ‘Guidelines on the protection of individuals with regard to the processing of personal data in a world of Big Data’, 19 the first document on privacy and Big Data which provides suggested measures for preventing any possible negative effects of the use of Big Data on human rights and freedoms.

Therefore, any government that uses Big Data in the health sector needs to establish affirmative policies to protect the health data of individuals, in terms of confidentiality, privacy and security, while ensuring that advancements in science can take advantage from the open use of data for the community well-being.

Conclusions

Big Data is beginning to revolutionize healthcare in Europe as it offers paths and solutions to improve health of individual persons as well as to improve the performance and outcomes of healthcare systems.

The implementation of precision medicine remains contingent on significant data acquisition and timely analysis to determine the most appropriate basis on which to tailor health optimization for individual prevention, diagnosis and disease treatment. Achieving effective and proportionate governance of health-related data will be essential for the future healthcare systems, and it requires that stakeholders collaborate and adapt the design and performance of their systems to reach the maximum innovative potential of information and innovation technology on health in the EU.

In this context, EU Member States should agree on international technical standards, taking also into account openness that is considered as the basic paradigm for digital transformation. Additionally, new approaches must be found for translating the vast amount of data into meaningful information that healthcare professionals can use. Further efforts must be made to make information for doctors and health professionals more accessible and understandable.

To achieve this, existing training and education programmes for healthcare professionals should integrate the issues of data handling in the curricula to ensure the development of the necessary skills and competencies. This is one of the objectives of the ‘European network staff eXchange for integrAting precision health in the health Care sysTems’ consortium (ExACT) 20 project that aims to integrate precision health in European health systems by training a new generation of healthcare professionals across and outside of the EU.

In conclusion, we are living in fast-moving times, not least in terms of healthcare innovation. Whilst there are pressing needs for more personalized and sustainable health services, science and technology are offering a host of potentially invaluable new tools to deliver them. A cooperation at the EU level is needed to facilitate investments both in new technology and in the human capital, in order to guide citizens into this new frontier of human health and well-being where data are becoming a significant corporate asset, a vital economic input and the foundation of new business models.

Conflicts of interest : None declared.

Big Data—Technologies and Potential

• First Online: 01 January 2014

Cite this chapter

• Hansjörg Fromm 3 &
• Stephan Bloehdorn 4  

Part of the book series: VDI-Buch ((VDI-BUCH))

8901 Accesses

3 Citations

Recently, the term Big Data has gained tremendous popularity in business and academic discussions and is now prominently used in scientific publications (Jacobs, Communications of the ACM—A Blind person’s interaction with technology, 2009), business literature (Mayer-Schönberger and Cukier, Big Data. A revolution that will transform how we live, work, and think, 2013; McAfee and Brynjolfsson, Harvard Business Review 90, 2012), whitepapers and analyst reports (Brown et al., Big Data. The next frontier for innovation, competition, and productivity, 2011b ; Economist Intelligence Unit 2012; Schroeck et al., Analytics: The real-world use of Big Data, 2012 ), as well as in popular magazines (Cukier 2010 ). While all these references somewhat associate the term with a new paradigm for data processing and analytics, the perception of what exactly it refers to are very diverse. The gap in the understanding of the phenomenon of Big Data is highlighted by the results of a recent study – in which respondents were asked to choose descriptions of the term Big Data – resulting in diverse characterizations such as, e.g., “ A greater scope of information ”, “ New kinds of data and analysis ” or “ Real-time information ” (Schroeck et al. 2012 ).

• Wind Turbine
• Stream Processing
• Economist Intelligence Unit
• Complex Event Processing
• Stream Processing System

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Durable hardcover edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Abadi DJ et al (2009) A comparison of approaches to large scale data analysis. In: Çetintemel U, Zdonik S (eds) Proceedings of the 35th SIGMOD International Conference on Management of Data. June 29–July 2, 2009, Providence, RI, USA. ACM Press, New York, pp 165–178

Google Scholar  

Abadi DJ et al (2010) MapReduce and parallel DBMSs. Friends or Foes? Communications of the ACM. Amir Pnueli: Ahead of His Time 53(1):64–71

Babu S, Widom J (2001) Continuous queries over data streams. ACM SIGMOD 30(3):109–120

Article   Google Scholar  

Ballard C et al (2010) IBM InfoSphere Streams. Harnessing data in motio, 1st edn. International Technical Support Organisation, New York

Barton D, Court D (2012) Making advanced analytics work for you. Harvard Bus Rev 90(10):79–83

Biem A et al (2010) Real-time traffic information management using stream computing. Bull Tech Committee Data Eng 33(2):64–68

Bollier D (2010) The promise and peril of Big Data. Aspen Institute, Communications and Society Program, Washington, DC

Brown E et al (2010) Building Watson. An overview of the Deep QA Project. AI Magazine 31(3):59–79

Brown B, Chui M, Manyika J (2011a) Are you ready for the era of ‘Big Data’? McKinsey & Company (ed). McKinsey Global Institute, o. O

Brown B et al (2011b) Big Data. The next frontier for innovation, competition, and productivity. McKinsey & Company (ed). McKinsey Global Institute, o. O

Brynjolfsson E, Hitt L, Kim H (2011) Strength in Numbers. How does data-driven decision-making affect firm performance? In: Beath C, Myers MD, Wei K (eds) Proceedings of the International Conference on Information Systems, Shanghai

Cate F et al (2012) The challenge of ‘Big Data’ for data protection. Int Data Privy Law 2(2):47–49

Stonebraker M, Cetintemel U, Zdonik S (2005) The 8 requirements of real-time stream processing. ACM SIGMOD Record 34(4):42–47

Chansler R et al (2010) The hadoop distributed file system. In: Factor M, He X, Khatib MG (eds) 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST). 6–7 May 2010, Lake Tahoe, Nevada, USA. IEEE, Piscataway, NJ, S 1–10

Cukier K (2010) Data, data, everywhere. A special report on managing information. The economist 2010, Feb 27th–March 5th, vol 394, Nr. 8671

Davenport TH, Harris JG (2007) Competing on analytics. The new science of winning. Harvard Business School Press, Boston

Davenport TH, Patil D (2012) Data scientist: the sexiest job of the 21st century. Harvard Bus Rev 90(10):70–76

Davis K, Patterson D (2012) Ethics of Big Data. O’Reilly Media, Sebastopol

Dean J, Ghemawat S (2008) MapReduce. Simplified data processing on large clusters. Communications of the ACM—50th anniversary issue: 1958–2008 51(1):107–113

Economist Intelligence Unit (2012) The deciding factor. Big Data & decision making (ed) Capgemini

Fowler M, Sadalage PJ (2012) NoSQL distilled. A brief guide to the emerging world of polygot persistence. Addison-Wesley, Boston

Fromm H (2010) Analytic solutions for fraud management. Presentation at the “Journeed’Etudes: Techniques des assurances et assurances techniques”. Published in: Les Cahiers de la Mathématique Appliquée No. 6. Accessed 22 Nov, Brussels

Gantz J, Reinsel D (2012) The digital universe in 2020. Big Data, bigger digital shadows, and biggest growth in the far East. IDC iView

Gartner (2013) IT Glossary. http://www.gartner.com/it-glossary/big-data/ . Accessed 5 Aug 2013

Ghoting A et al (2011) SystemML. Declarative Machine Learning on MapReduce. In: Proceedings of the 2011 IEEE 27th International Conference on Data Engineering (ed) IEEE Computer Society pp 131–142

Gray J, Szalay A (2006) 2020 Computing. Science in an exponential world. Nature 440(7083):413–441

Guizzo E (2011) IBM’s Watson Jeopardy computer shuts down humans in final game. IEEE Spectrum o. Jg

Halevi G et al (2012) Research trends. Special issue on Big Data. Research Trends 26(6):o. S

Hilary M (2013) What is a data scientist? http://www.forbes.com/sites/danwoods/2012/03/08/hilary-mason-what-is-a-data-scientist/ . Accessed 7 Aug 2013

Jacobs A (2009) The pathologies of Big Data. Commun ACM. A Blind Person’s Interact Technol 52(8):36–42

Kiron D et al (2012) Sustainability nears a tipping point. MIT Sloan Manag Rev 53(2):69–74

Laney D (2001) 3D data management: controlling data volume, velocity, and variety. META Group, o. O. (ed). Accessed 7 August 2001

LaValle S, Lesser E, Shockley R (2011) Big Data, analytics and the path from insights to value. MITSloan Manag Rev 52(2):21–31

Luckham D (2002) The power of events. An introduction to complex event processing in distributed enterprise systems. Wesley, Boston

Luckham DC (2012) Event processing for business. organizing the real-time enterprise. Wiley, Hoboken

Mayer-Schönberger V, Cukier K, (2013) Big Data. A revolution that will transform how we live, work, and think. Houghton Mifflin Harcourt, Boston

McAfee A, Brynjolfsson E (2012) Big Data. The management revolution. Harvard Bus Rev 90(10):61–67

Miller CC (2013) Data science. The numbers of our lives. The New York Times o. Jg. S o. S

Polonetsky J, Tene O (2012) Privacy in the age of Big Data. A time for big decisions. Stanford Law Rev 64:63

Polonetsky J, Tene O (2013) Big Data for all. Privacy and user control in the age of analytics. Northwestern J Technol Intellect Property 11(5):240–272

Schroeck M et al (2012) Analytics: the real-world use of Big Data. How innovative enterprises extract value from uncertain data. IBM Global Business Services (ed). IBM Institute for Business Value, o. O

Sicular S (2013) Gartner’s Big Data definition consists of three parts, not to be confused with three “V”s. http://www.forbes.com/sites/gartnergroup/2013/03/27/gartners-big-data-definition-consists-of-three-parts-not-to-be-confused-with-three-vs/ . Accessed 6 Aug 2013

Stonebraker M (1986) The case for shared nothing. IEEE Database Eng Bull 9(1):4–9

White T (2009) Hadoop. The definitive guide. O’Reilly, Sebastopol

Zikopoulos P, Eaton C (2012) Understanding Big Data: analytics for enterprise class hadoop and streaming data. Osborne, New York

Download references

Acknowledgments

We are grateful to Steven O. Kimbrough (Wharton School, University of Pennsylvania, USA) and Stefan Mueck (IBM Germany) for excellent comments on draft versions of this paper.

Author information

Authors and affiliations.

Karlsruhe Institute of Technology, Kollegiengebäude am Ehrenhof (Geb. 11.40), Englerstraße 11, 76131, Karlsruhe, Deutschland

Prof. Dr. Hansjörg Fromm

IBM Deutschland, Nahmitzer Damm 12, 12277, Berlin, Deutschland

Dr. Stephan Bloehdorn

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Hansjörg Fromm .

Editor information

Editors and affiliations.

Werkzeugmaschinenlabor (WZL), RWTH Aachen, Aachen, Germany

Günther Schuh

Forschungsinstitut für Rationalisierung, RWTH Aachen, Aachen, Germany

Volker Stich

Rights and permissions

Reprints and permissions

Copyright information

© 2014 Springer-Verlag Berlin Heidelberg

About this chapter

Fromm, H., Bloehdorn, S. (2014). Big Data—Technologies and Potential. In: Schuh, G., Stich, V. (eds) Enterprise -Integration. VDI-Buch. Springer Vieweg, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-41891-4_9

Download citation

DOI : https://doi.org/10.1007/978-3-642-41891-4_9

Published : 29 March 2014

Publisher Name : Springer Vieweg, Berlin, Heidelberg

Print ISBN : 978-3-642-41890-7

Online ISBN : 978-3-642-41891-4

eBook Packages : Computer Science and Engineering (German Language)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

• Get 7 Days Free

Treasure Data Named a Leader in Inaugural Gartner® Magic Quadrant™ for Customer Data Platforms

Treasure Data Evaluated for its AI-Powered CDP Among 17 Vendors

Treasure Data today announced that it has been named a Leader in the inaugural 2024 Gartner® Magic Quadrant™ for Customer Data Platforms. The report evaluated 17 different CDP vendors on 15 criteria and placed Treasure Data in the Leaders quadrant for its Completeness of Vision and Ability to Execute.

“As a true CDP pioneer, we’re proud to build on our heritage and reinvent the CDP with AI, real-time engagement, and journey orchestration all in one platform,” said Kaz Ohta, CEO and co-founder, Treasure Data. “Our singular focus is to provide the world’s largest companies a best-of-breed CDP that unifies data from anywhere and drives ROI by delivering connected customer experiences throughout the world. As we empower organizations to expand the CDP beyond just marketing, we look forward to extending our position as a market leader.”

Treasure Data believes its recognition from Gartner as a Leader affirms its success in reinventing the CDP to better help all teams that rely on customer data enrich experiences across the entire customer lifecycle, at every moment of interaction. Treasure Data CDP helps the world’s largest companies increase revenue by improving customer acquisition and growth, drive operational efficiencies that lower costs, and reduce risk using customer data across multiple teams and geographies.

"We are proud to congratulate Treasure Data on being recognized as a Leader by Gartner,” said Luiz Gama, senior global martech manager, consumer data and CRM, at Anheuser-Busch InBev, whose global Treasure Data CDP deployment spans more than 90 million unique consumer records, 42 countries and more than 2,000 unified data sources. “As a valued partner of AB InBev, Treasure Data has helped us unlock valuable insights and deliver exceptional experiences for our consumers across the globe. This recognition is a testament to their commitment to innovation and excellence in the field of CDPs, and we are proud to have them as a key business partner.”

As evidenced by large-scale CDP deployments such as AB InBev, Treasure Data is built on a foundation of trust and scale that supports the requirements of the world’s largest companies. Treasure Data’s proprietary big data platform processes customer data at infinite scale and massive speeds, while its enterprise-grade security, privacy and data governance inspires the trust needed to drive CDP adoption across the entire company, worldwide.

As Treasure Data reinvents the CDP to bring data and decisioning together and deliver the business outcomes its customers need most, its latest innovations include:

Accelerating time to insight and driving efficiencies with AI and machine learning

AI and machine learning are infused across Treasure Data CDP, from profile stitching to auto-segmentation and targeting. Treasure Data plans to release generative AI-assisted data analysis and exploration capabilities into its platform this year.

Enabling all customer-centric teams to deliver connected customer experiences

Treasure Data empowers teams across the company, including marketing, IT, service, sales, and data science, to use its CDP to drive connected customer experiences. This allows customer data to be used across every interaction, from sales to service.

Powering more intelligent journeys with real-time interactions and insights

Treasure Data unveiled Real-Time 2.0 CDP, one of the only enterprise solutions that combines both batch and real-time data processing at the profile level, which is accessible in Treasure Data’s unified journey management interface.

Download a complimentary copy of the Gartner Magic Quadrant for Customer Data Platforms here .

Learn more about the reinvention of the CDP at CDP World 2024 powered by Treasure Data. Join hundreds of data and marketing leaders at the premier customer data conference in Las Vegas on October 15–16, 2024.

Gartner Disclaimer

Gartner, Magic Quadrant for Customer Data Platforms, Lizzy Foo Kune, Rachel Smith, Suzanne White, Benjamin Bloom, Adriel Tey, David Walters, 14 February 2024.

Gartner does not endorse any vendor, product, or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner's research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally, and MAGIC QUADRANT is a registered trademark of Gartner, Inc. and/or its affiliates and are used herein with permission. All rights reserved.

About Treasure Data

Treasure Data empowers the world’s largest and most innovative companies to drive connected customer experiences that increase revenue and reduce costs. Built on a big data foundation of trust and scale, Treasure Data is a customer data platform (CDP) pioneer and continues to reinvent the CDP by putting AI and real-time experiences at the center of the customer journey. Our CDP gives customer-centric teams across Fortune 500 and Global 2000 companies – marketing, sales, service, and more – the power to turn customer data into their greatest treasure. Visit www.treasuredata.com to learn more.

Marci Bontadelli [email protected]

View source version on businesswire.com: https://www.businesswire.com/news/home/20240221863857/en/

Market Updates

Markets brief: why q1 was a triumph for the optimists, q2 stock market outlook: contrarian plays look increasingly attractive, what’s happening in the markets this week, what the next bitcoin halving means for etf investors, 5 undervalued stocks to buy after they’ve been dumped, morningstar market insights: central banks, inflation, and patient investing, where to invest in bonds now, reddit stock surges in ipo, stock picks, the 10 best companies to invest in now, trump media is not a meme stock. it’s a cryptocurrency, what investors need to know about ge’s spinoff, investors can find more than gold in the mining industry, an ultracheap stock to buy in a rallying sector, celsius stock has soared 206%. is it a buy or a sell, 10 undervalued wide-moat stocks, investment opportunities in us renewable energy, sponsor center.

Analyst Reports , Announcements , Azure API Management , Azure Analysis Services , Azure Data Factory , Azure Functions , Azure Integration Services , Azure IoT , Azure Logic Apps , Azure OpenAI Service , Event Grid , Integration , Service Bus

Microsoft named a Leader in 2024 Gartner® Magic Quadrant™ for Integration Platform as a Service 

By Naga Surendran Director of Product Marketing, Azure

Posted on March 12, 2024 4 min read

• Tag: Gartner® Magic Quadrant™

We’re thrilled to announce that Microsoft has once again been named a Leader in the 2024 Gartner® Magic Quadrant™ for Integration Platform as a Service*. It’s the sixth time in a row that we’ve been recognized.

Businesses have seen a sharp rise in the number of applications they manage. Seamlessly integrating applications across on-premises and multi-cloud environments is not only vital for streamlined operations, but it also stands as a strategic imperative to outperform competitors in the market. Microsoft’s enterprise integration offering, Azure Integration Services , is purpose-built to address the unique requirements and challenges faced by today’s businesses. 

Learn why Microsoft was named a Leader

2024 Gartner® Magic Quadrant™ for Integration Platform as a Service

Tailored integration to meet your unique business needs 

In a world where every enterprise is unique, we understand the importance of providing the freedom to choose and adopt only the services that align with your requirements. Our enterprise integration offering, Azure Integration Services—which is comprised of Azure API Management, Azure Logic Apps, Azure Service Bus, Azure Event Grid, Azure Functions, and Azure Data Factory—is designed with this flexibility in mind. Azure Integration Services allows you to choose and adopt only the services you need. Whether you’re looking to integrate on-premises systems, cloud-based applications, or a hybrid of both, Azure Integration Services provides the building blocks necessary for your integration success. 

Anil Shah, Chief Executive Officer of CloudFronts, finds great value in the adaptability provided by Azure Integration Services .

“ Azure Integration Services gives organizations the flexibility to tailor their integration solutions to their needs, while keeping costs under control .”   Anil Shah, Chief Executive Officer, CloudFronts

Don’t pay for what you don’t need 

We understand the common challenges of budget constraints and scalability issues, and we’ve designed Azure Integration Services to tackle these head-on. Our flexible pay-as-you-go pricing model ensures that you only pay for what you use, eliminating the burden of hefty upfront licensing fees. As your business expands, Azure Integration Services seamlessly scales to meet your evolving needs. This guarantees that your integration solution remains not only efficient but also cost-effective. 

For Ajay Dhingra, Founder and President of Khoj Information Technology, the flexible pricing model of a Platform-as-a-Service (PaaS) like Azure Integration Services was a big selling point.

“ We evaluated top integration products, but they required a big initial expenditure. With a PaaS solution, US LBM could start small and grow over time as its needs evolved, and leadership liked the flexibility of that approach .”  Ajay Dhingra, Founder and President, Khoj Information Technology

Worldwide support and reliability 

In today’s global business landscape, the ability to seamlessly operate across regions is crucial. That’s where Azure Integration Services steps in. It’s a hybrid, multi-cloud, globally managed offering accessible in over 50 Microsoft Azure regions. This means you can effortlessly connect your applications and systems across various geographical locations, all while upholding the performance and reliability standards that users demand. 

KPMG Netherlands, a Dutch professional services firm providing advisory and audit services, cites the valuable assistance they’ve gotten from Microsoft tech support during a recent platform integration project . Carel Nederveen, the Enterprise Technology Architect at KPMG Netherlands, commended the Microsoft team’s reliability.

“ We didn’t want to spend excessive time on research. If we couldn’t resolve an issue within a few hours, we’d seek assistance from the Microsoft team, a highly valuable and supportive resource .”  Carel Nederveen, Enterprise Technology Architect, KPMG Netherlands

Robust partner ecosystem with over 1,400 managed partners worldwide 

Strong partnerships underpin thriving ecosystems. Teaming up with partners globally, we aim to drive and scale your integration efforts successfully. Notable successes include SPAR NL’s event-driven Azure platform and BÜCHI’s real-time data exchange, both achieved through strategic partnerships. With a shared commitment to innovation and customer-centricity, Microsoft and our certified partners are consistently leading the way in delivering groundbreaking integration solutions. 

Unlock the full potential of Azure    

Azure Integration Services goes beyond being a standalone solution—it seamlessly integrates with the broader Azure ecosystem. This means that you’re not only getting a powerful integration platform, but you’re also unlocking the full potential of Azure. Seamlessly integrating with other Azure offerings—such as Azure OpenAI Service, Azure analytics, and Azure IoT—empowers you to build holistic solutions that quickly drive innovation and efficiency across the entire organization. 

Ready to explore further?

• Download a complimentary copy of the 2024 Gartner® Magic Quadrant™ for Integration Platform as a Service report to learn why Microsoft is named a Leader.
• Learn more about Azure Integration Services .   

*Gartner is a registered trademark and service mark and Magic Quadrant is a registered trademark of Gartner, Inc. and/or its affiliates in the U.S. and internationally and are used herein with permission. All rights reserved. 

This graphic was published by Gartner, Inc. as part of a larger research document and should be evaluated in the context of the entire document. The Gartner document is available upon request. 

Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

Gartner, Magic Quadrant for Integration Platform as a Service, by K, Keith Guttridge, Andrew Comes, Shrey Pasricha , Max van den Berk , and Andrew Humphreys , 19, February, 2024.

Let us know what you think of Azure and what you would like to see in the future.

Provide feedback

Build your cloud computing and Azure skills with free courses by Microsoft Learn.

Explore Azure learning

Related posts

AI + Machine Learning , Azure AI , Azure AI Services , Azure AI Studio , Azure Cosmos DB , Azure Kubernetes Service (AKS) , Azure OpenAI Service , Customer stories , Integration , Thought leadership

Microsoft Azure AI celebrates Women’s History Month through our customers   chevron_right

AI + Machine Learning , Announcements , Azure AI , Azure AI Content Safety , Azure AI Studio , Azure OpenAI Service , Security

Announcing new tools in Azure AI to help you build more secure and trustworthy generative AI applications   chevron_right

AI + Machine Learning , Announcements , Azure AI , Azure AI Speech , Azure OpenAI Service , Speech to text

Accelerate your productivity with the Whisper model in Azure AI now generally available   chevron_right

AI + Machine Learning , Announcements , Azure AI , Azure OpenAI Service , Events , Health Bot

Azure AI Health Bot helps create copilot experiences with healthcare safeguards   chevron_right

• Cybersecurity

Gartner reveals top eight cybersecurity predictions for years ahead

Impact of generative AI on closing skills gap, and efforts against misinformation rising to account for more than 50% of marketing and cyber budgets, among headline forecasts.

By Tech Monitor Staff

• Gartner%20reveals%20top%20eight%20cybersecurity%20predictions%20for%20years%20ahead %20-%20https://techmonitor.ai/technology/cybersecurity/cybersecurity-predictions-2024-gartner" target="_blank" class="c-share__link icon-email" title="Share in an email">

Gartner has unveiled its top cybersecurity predictions for 2024 and beyond. They were presented by director analyst Deepti Gopal at the Gartner Security & Risk Management Summit in Sydney. Gopal cautioned that cyber leaders must begin integrating these trends into their security strategies and assumptions as soon as possible.

An overarching theme was the extent to which advances in AI were transforming the space, both in regard to how one secures an enterprise, and the threats enterprises now face.

“As we start moving beyond what’s possible with GenAI, solid opportunities are emerging to help solve a number of perennial issues plaguing cybersecurity, particularly the skills shortage and unsecure human behaviour,” Gopal observed.

“The scope of the top predictions this year is clearly not on technology, as the human element continues to gain far more attention. Any CISO looking to build an effective and sustainable cybersecurity program must make this a priority.”

1. GenAI will close the skills gap by 2028

Gartner forecasts that the requirement of specialised training should be removed from 50% of entry level cybersecurity roles within the next four years.

This development will be welcome news to cyber leaders, who have found it increasingly challenging to recruit requisite skillsets within the sector. Last year, ISC2 found that the global gap had reached four million people, with 62% of surveyed cybersecurity teams defining themselves as being understaffed.  

The growing use of GenAI should allow leaders to recruit on aptitude, rather than training or experience, and dedicate more budget and focus on filling critical cyber roles.  

Content from our partners

How businesses can safeguard themselves on the cyber frontline

How hackers’ tactics are evolving in an increasingly complex landscape

Green for go: Transforming trade in the UK

2. a 40% drop in employee-driven cybersecurity incidents by 2026.

GenAI’s capacity to deliver “hyper-personalised” content and materials should empower enterprises to offer training that considers the specific characteristics of individual employees. This means far richer security behaviour and culture programs (SBCPs), which, Gartner argues, will lead to a far more engaged, cyber-savvy workforce.   

“Organisations that haven’t yet embraced GenAI capabilities should evaluate their current external security awareness partner to understand how it is leveraging GenAI as part of its solution roadmap,” said Gopal.

3. 75% of organisations will exclude legacy, cyber-physical infrastructure from zero trust strategies by 2026

If enterprises are to provide users and endpoints only with the access needed to do their jobs, while retaining the ability to continuously monitor behaviour and the flexibility to adapt to evolving threat vectors, difficult decisions will need to be made, Gartner believes.

The normalisation of remote and hybrid working environments, prompted at speed by the pandemic and largely retained in its wake, has seen a proliferation of end-points, many of which are unmanaged devices. The ongoing shift to zero trust policies will necessitate the exclusion of such devices from any such strategies moving forward.

4. The CISO will require greater protection form personal legal exposure

Gartner believes that two-thirds of global 100 organisations will have extended directors and officers (D&O) insurance to cybersecurity leaders by 2027. This will be driven in large part by new laws and regulations – Gartner specifically cites the SEC’s new cybersecurity disclosure and reporting rules – exposing those leading the security function like never before.

Whether the answer lies in incorporating the CISO into one’s existing D&O insurance package, or exploring other solutions and providers, Gartner cautions that this is an issue enterprises need to get out ahead of quickly.

5. Spend on battling misinformation to exceed $500 billion by 2028

The rise of GenAI cuts both ways. Yes, it has the potential to help leaders secure their enterprises, but it also opens up a slew of new avenues and strategies for potential attack. Gartner points to “the combination of AI, analytics, behavioral science, social media, Internet of Things and other technologies enabling bad actors to create and spread highly effective, mass-customised malinformation”.

Battling this trend will, it forecasts, come to account for over 50% of marketing and cybersecurity budgets within the next four years, requiring the refining of responsibilities, technologies, techniques, and greater use of “chaos engineering” to build resilience and counter external threats.

6. We will see an overhaul of how identity and access management is executed   

Gartner believes that, by2026, 40% of identity and access management (IAM) leaders will taken on primary responsibility for detecting and responding to IAM-related breaches.

IAM leaders are typically not involved in security resourcing and budgeting discussions, and often face issues conveying security and business value in order to secure requisite investment, Gartner argues. However, as the significance of IAM continues to grow, so too will their scope of influence and visibility across the enterprise.

Gartner recommends CISOs reform traditional IT and security silos, providing those holding the purse strings with more clarity and insight into the role IAM plays, integrating the function into wider security initiatives.

7. Data loss prevention and insider risk management will be incorporated with IAM

One visible impact of the previous prediction will be the combining of various disciplines to better identify and combat suspicious behaviour. Indeed, Gartner forecasts 70% of organisations having integrated data loss prevention and insider risk management disciplines with IAM by 2027.

“This introduces a more comprehensive set of capabilities for security teams to create a single policy for dual use in data security and insider risk mitigation,” the prediction reads. “Gartner recommends organisations identify data risk and identity risk, and use them in tandem as the primary directive for strategic data security.”

8. Application security to be increasingly consumed directly by non-cyber experts and owned by application owners

The sheer number of applications that technologists and delivery teams now create means the scale of exposures and potential attack points are hitting numbers that dedicated application security teams simply can’t tackle alone.

With this in mind, Gartner is predicting that, by 2027, 30% of cybersecurity functions will redesign application security so that it is no longer the preserve of the security function.

Gopal observed: “To bridge the gap, cybersecurity functions must build minimum effective expertise in these teams, using a combination of technology and training to generate only as much competence as is required to make cyber risk informed decisions autonomously,”

Tech Monitor Staff

• Architect Founder

Thales Blog

Data security trends: 2024 report analysis, march 25, 2024.

Amid ongoing economic uncertainty and a progressively complex threat landscape, businesses are trying to navigate increasingly stringent regulatory requirements while bolstering their security posture.

The 2024 Thales Global Data Threat Report , conducted by S&P Global Market Intelligence, which surveyed almost 3,000 respondents from 18 countries and 37 industries, revealed how decision-makers navigate new threats while trying to overcome old challenges. The report explores their experiences, hurdles, approaches, and achievements and offers insights into the security implications of new technologies and the organizational adaptations necessary for future success.

Compliance and Residency Are Key

The study revealed that although risk is volatile and cyber regulations constantly change, nearly half (43%) of businesses did not pass a compliance audit in the past year. Among those failing audits, 31% suffered a breach in the same period, compared to a mere 3% among compliant businesses. This highlights a significant link between compliance adherence and data security.

Challenges also persist in managing operational complexity, leading to data-related issues. A substantial number of organizations struggle to identify and classify their at-risk systems, applications, and data, with only a third (33%) achieving full classification. Alarmingly, 16% admitted to hardly classifying any of their data.

The rampancy of multi-cloud usage across services, along with evolving global data privacy regulations, has underscored the importance of data sovereignty for businesses. According to the report, 28% of respondents consider mandatory external key management as the primary method to achieve sovereignty.

A Matter of Trust

The Report also revealed that most customers (89%) are willing to share their data with organizations, but this willingness comes with certain non-negotiable conditions. Nearly nine out of ten (87%) expect some level of privacy rights from the companies they engage with online. In addition to these high consumer privacy expectations, respondents highlighted that many customers access their organization's internal systems or assets. They indicated that up to 16% of those accessing corporate cloud, network, and device resources could be customers.

Similarly, external vendor and contractor access accounted for an average of 15% and 12% of users, respectively. Given the combination of heightened consumer privacy expectations and extensive external user access, Customer Identity and Access Management ( CIAM ) emerged as one of the primary emerging security concerns.

However, while improvements in CIAM, such as passkeys and password deprecation, enhance user experience, they also introduce new challenges like deepfake attacks from generative AI, and simplifying this complexity is crucial to reducing opportunities for adversaries and improving usability and engagement.

Emerging Tech: Threats and Opportunities

The report also delved into the emerging technologies that security practitioners are eyeing. More than half (57%) cited Artificial Intelligence (AI) as a major worry, with IoT hot on its heels with 55%. Next came Post Quantum Cryptography with 45%.

Having said that, these technologies also promise a host of benefits. Some 22% of respondents said they were planning to integrate generative artificial intelligence (GenAI) into their security solutions and services over the next year, and another third (33%) plan to experiment with the technology.

Ubiquitous Connectivity, Pervasive Threats

In the era of ubiquitous connectivity, IoT and 5G bring about pervasive threats too. While operational technology (OT) deployments have been criticized for their lax security focus, this year's survey reveals that 75% of IT security teams prioritize OT as a defense against IoT threats.

OT devices like power meters and "smart" sensors in various distributed physical plants are often designed for minimal oversight and reduced operational costs, exacerbating security risks. This means proactive security measures are essential. Despite the increasing connectivity options, traditional methods like physical or network isolation ("air gapping") are less favored for securing IoT/OT environments.

Reflecting zero-trust principles, respondents show reluctance to rely solely on carrier security, with only 33% expressing concern about carrier network security in the context of 5G. However, IoT and OT devices face persistent security challenges.

Establishing Centrally Defined Principles

As enterprises expand, so too will their use and integration of these technologies. This is why establishing centrally defined security principles can improve the likelihood of successful delegation and implementation, mainly when rooted in the fundamental concepts of guidance and agreement.

Like how the rule of law thrives in societies where individuals and institutions understand their rights and obligations, enterprise data security risks can be mitigated by empowering and entrusting other stakeholders to adhere to these principles voluntarily.

Download the full Thales 2024 Thales Data Threat Report now.

Related Articles

Qr code scams: what you need to know about this phishing tactic, keeping customer data safe: ai's privacy paradox, ransomware attacks: the constant and evolving cybersecurity threat, making waves: empowering women in cybersecurity, what the nfl can teach us about building a winning data security strategy.

• Share full article

Advertisement

Supported by

What the Data Says About Pandemic School Closures, Four Years Later

The more time students spent in remote instruction, the further they fell behind. And, experts say, extended closures did little to stop the spread of Covid.

By Sarah Mervosh ,  Claire Cain Miller and Francesca Paris

Four years ago this month, schools nationwide began to shut down, igniting one of the most polarizing and partisan debates of the pandemic.

Some schools, often in Republican-led states and rural areas, reopened by fall 2020. Others, typically in large cities and states led by Democrats, would not fully reopen for another year.

A variety of data — about children’s academic outcomes and about the spread of Covid-19 — has accumulated in the time since. Today, there is broad acknowledgment among many public health and education experts that extended school closures did not significantly stop the spread of Covid, while the academic harms for children have been large and long-lasting.

While poverty and other factors also played a role, remote learning was a key driver of academic declines during the pandemic, research shows — a finding that held true across income levels.

Source: Fahle, Kane, Patterson, Reardon, Staiger and Stuart, “ School District and Community Factors Associated With Learning Loss During the COVID-19 Pandemic .” Score changes are measured from 2019 to 2022. In-person means a district offered traditional in-person learning, even if not all students were in-person.

“There’s fairly good consensus that, in general, as a society, we probably kept kids out of school longer than we should have,” said Dr. Sean O’Leary, a pediatric infectious disease specialist who helped write guidance for the American Academy of Pediatrics, which recommended in June 2020 that schools reopen with safety measures in place.

There were no easy decisions at the time. Officials had to weigh the risks of an emerging virus against the academic and mental health consequences of closing schools. And even schools that reopened quickly, by the fall of 2020, have seen lasting effects.

But as experts plan for the next public health emergency, whatever it may be, a growing body of research shows that pandemic school closures came at a steep cost to students.

The longer schools were closed, the more students fell behind.

At the state level, more time spent in remote or hybrid instruction in the 2020-21 school year was associated with larger drops in test scores, according to a New York Times analysis of school closure data and results from the National Assessment of Educational Progress , an authoritative exam administered to a national sample of fourth- and eighth-grade students.

At the school district level, that finding also holds, according to an analysis of test scores from third through eighth grade in thousands of U.S. districts, led by researchers at Stanford and Harvard. In districts where students spent most of the 2020-21 school year learning remotely, they fell more than half a grade behind in math on average, while in districts that spent most of the year in person they lost just over a third of a grade.

( A separate study of nearly 10,000 schools found similar results.)

Such losses can be hard to overcome, without significant interventions. The most recent test scores, from spring 2023, show that students, overall, are not caught up from their pandemic losses , with larger gaps remaining among students that lost the most ground to begin with. Students in districts that were remote or hybrid the longest — at least 90 percent of the 2020-21 school year — still had almost double the ground to make up compared with students in districts that allowed students back for most of the year.

Some time in person was better than no time.

As districts shifted toward in-person learning as the year went on, students that were offered a hybrid schedule (a few hours or days a week in person, with the rest online) did better, on average, than those in places where school was fully remote, but worse than those in places that had school fully in person.

Students in hybrid or remote learning, 2020-21

80% of students

Some schools return online, as Covid-19 cases surge. Vaccinations start for high-priority groups.

Teachers are eligible for the Covid vaccine in more than half of states.

Most districts end the year in-person or hybrid.

Source: Burbio audit of more than 1,200 school districts representing 47 percent of U.S. K-12 enrollment. Note: Learning mode was defined based on the most in-person option available to students.

Income and family background also made a big difference.

A second factor associated with academic declines during the pandemic was a community’s poverty level. Comparing districts with similar remote learning policies, poorer districts had steeper losses.

But in-person learning still mattered: Looking at districts with similar poverty levels, remote learning was associated with greater declines.

A community’s poverty rate and the length of school closures had a “roughly equal” effect on student outcomes, said Sean F. Reardon, a professor of poverty and inequality in education at Stanford, who led a district-level analysis with Thomas J. Kane, an economist at Harvard.

Score changes are measured from 2019 to 2022. Poorest and richest are the top and bottom 20% of districts by percent of students on free/reduced lunch. Mostly in-person and mostly remote are districts that offered traditional in-person learning for more than 90 percent or less than 10 percent of the 2020-21 year.

But the combination — poverty and remote learning — was particularly harmful. For each week spent remote, students in poor districts experienced steeper losses in math than peers in richer districts.

That is notable, because poor districts were also more likely to stay remote for longer .

Some of the country’s largest poor districts are in Democratic-leaning cities that took a more cautious approach to the virus. Poor areas, and Black and Hispanic communities , also suffered higher Covid death rates, making many families and teachers in those districts hesitant to return.

“We wanted to survive,” said Sarah Carpenter, the executive director of Memphis Lift, a parent advocacy group in Memphis, where schools were closed until spring 2021 .

“But I also think, man, looking back, I wish our kids could have gone back to school much quicker,” she added, citing the academic effects.

Other things were also associated with worse student outcomes, including increased anxiety and depression among adults in children’s lives, and the overall restriction of social activity in a community, according to the Stanford and Harvard research .

Even short closures had long-term consequences for children.

While being in school was on average better for academic outcomes, it wasn’t a guarantee. Some districts that opened early, like those in Cherokee County, Ga., a suburb of Atlanta, and Hanover County, Va., lost significant learning and remain behind.

At the same time, many schools are seeing more anxiety and behavioral outbursts among students. And chronic absenteeism from school has surged across demographic groups .

These are signs, experts say, that even short-term closures, and the pandemic more broadly, had lasting effects on the culture of education.

“There was almost, in the Covid era, a sense of, ‘We give up, we’re just trying to keep body and soul together,’ and I think that was corrosive to the higher expectations of schools,” said Margaret Spellings, an education secretary under President George W. Bush who is now chief executive of the Bipartisan Policy Center.

Closing schools did not appear to significantly slow Covid’s spread.

Perhaps the biggest question that hung over school reopenings: Was it safe?

That was largely unknown in the spring of 2020, when schools first shut down. But several experts said that had changed by the fall of 2020, when there were initial signs that children were less likely to become seriously ill, and growing evidence from Europe and parts of the United States that opening schools, with safety measures, did not lead to significantly more transmission.

“Infectious disease leaders have generally agreed that school closures were not an important strategy in stemming the spread of Covid,” said Dr. Jeanne Noble, who directed the Covid response at the U.C.S.F. Parnassus emergency department.

Politically, though, there remains some disagreement about when, exactly, it was safe to reopen school.

Republican governors who pushed to open schools sooner have claimed credit for their approach, while Democrats and teachers’ unions have emphasized their commitment to safety and their investment in helping students recover.

“I do believe it was the right decision,” said Jerry T. Jordan, president of the Philadelphia Federation of Teachers, which resisted returning to school in person over concerns about the availability of vaccines and poor ventilation in school buildings. Philadelphia schools waited to partially reopen until the spring of 2021 , a decision Mr. Jordan believes saved lives.

“It doesn’t matter what is going on in the building and how much people are learning if people are getting the virus and running the potential of dying,” he said.

Pandemic school closures offer lessons for the future.

Though the next health crisis may have different particulars, with different risk calculations, the consequences of closing schools are now well established, experts say.

In the future, infectious disease experts said, they hoped decisions would be guided more by epidemiological data as it emerged, taking into account the trade-offs.

“Could we have used data to better guide our decision making? Yes,” said Dr. Uzma N. Hasan, division chief of pediatric infectious diseases at RWJBarnabas Health in Livingston, N.J. “Fear should not guide our decision making.”

Source: Fahle, Kane, Patterson, Reardon, Staiger and Stuart, “ School District and Community Factors Associated With Learning Loss During the Covid-19 Pandemic. ”

The study used estimates of learning loss from the Stanford Education Data Archive . For closure lengths, the study averaged district-level estimates of time spent in remote and hybrid learning compiled by the Covid-19 School Data Hub (C.S.D.H.) and American Enterprise Institute (A.E.I.) . The A.E.I. data defines remote status by whether there was an in-person or hybrid option, even if some students chose to remain virtual. In the C.S.D.H. data set, districts are defined as remote if “all or most” students were virtual.

An earlier version of this article misstated a job description of Dr. Jeanne Noble. She directed the Covid response at the U.C.S.F. Parnassus emergency department. She did not direct the Covid response for the University of California, San Francisco health system.

How we handle corrections

Sarah Mervosh covers education for The Times, focusing on K-12 schools. More about Sarah Mervosh

Claire Cain Miller writes about gender, families and the future of work for The Upshot. She joined The Times in 2008 and was part of a team that won a Pulitzer Prize in 2018 for public service for reporting on workplace sexual harassment issues. More about Claire Cain Miller

Francesca Paris is a Times reporter working with data and graphics for The Upshot. More about Francesca Paris