TL;DR:
- 76% of businesses face data-related challenges when adopting AI, with inconsistency of data sources, uncertain timeliness or quality, and data spread across separate silos being the top concerns.
- Lack of AI-related skills, data lineage, and insufficient infrastructure for real-time data processing are major obstacles in scaling AI and machine learning initiatives.
- Data streaming platforms (DSPs) have helped 51% of IT leaders tackle these challenges, significantly fueling AI progress.
The Rise of AI and the Data Conundrum
Artificial intelligence (AI) and agile general intelligence (AGI) are transforming the way businesses operate across Asia. However, a recent study reveals that data-related challenges are hindering AI adoption in many organisations. In the “2024 Data Streaming Report” by Confluent, 76% of the 4,110 IT leaders surveyed cited five or more challenges related to data management. So, what are these obstacles, and how can businesses overcome them?
Data Management Challenges: The Top Culprits
Inconsistency of data sources, uncertain timeliness or quality, and data spread across separate silos are the most common challenges faced by businesses. Let’s take a closer look at these issues:
- Inconsistency of data sources (66%): Businesses often rely on various data sources, which can lead to inconsistencies and inaccuracies.
- Uncertain timeliness or quality (65%): Ensuring data is up-to-date and of high quality is crucial for AI applications, but many businesses struggle with this aspect.
- Data spread across separate silos (64%): When data is stored in different locations or systems, it can be difficult to access and integrate it for AI initiatives.
Additional data management challenges include fragmented ownership of data, unwillingness of owners to share, and government-related disjoints.
Scaling AI and Machine Learning: Skills, Infrastructure, and More
As businesses ramp up AI and machine learning (AI/ML) adoption, they encounter additional challenges. The report found that 70% of respondents face three or more obstacles when scaling AI/ML initiatives. Some of the most significant hurdles include:
- Insufficient skills and expertise (65%): A lack of AI-related skills makes it difficult for businesses to manage AI products and workflows effectively.
- Data lineage and fragmentation (64%): Understanding the origin and history of data is crucial for AI applications, but many businesses struggle with data lineage and fragmentation.
- Insufficient infrastructure for real-time data processing (63%): Processing data in real-time is essential for many AI applications, but businesses often lack the necessary infrastructure.
Data streaming platforms (DSPs) have emerged as a promising solution for addressing these challenges. According to the study, 51% of IT leaders reported that DSPs have helped their organisations become more agile and tackle data-related obstacles. Here’s how DSPs are making a difference:
- Breaking down data silos (93%): DSPs enable businesses to integrate data from various sources, making it more accessible and useful for AI applications.
- Improving data access and discovery (88% and 86%): DSPs help businesses access existing data and discover new data sources, which can enhance AI initiatives.
- Addressing governance-related disjoints (84%): DSPs can help businesses manage data governance issues, ensuring data is used safely and responsibly.
Streamlining Data Management
To overcome data-related challenges in AI adoption, consider implementing a data streaming platform. By integrating data from various sources, improving data access and discovery, and addressing governance issues, DSPs can help your organisation become more agile and fuel AI progress.
How has your business tackled data-related challenges when adopting AI? Have you considered using data streaming platforms to improve data management and fuel AI progress? Share your experiences and thoughts below, and don’t forget to subscribe for updates on AI and AGI developments in Asia.
You may also like:
