BigData LDN (London) September 2024 - Our Top Ten Takeaways

1. Most users are still expecting personalisation, but only starting to get it from Big Data and AI strategies

The majority of users want and expect personalization, but a lot of big data and AI projects are still in the pilot phase. However amazing case studies are starting to emerge regarding the impact of using these strategies for product specific pilots or for certain use cases in full service delivery.  Accor hotels, Easyjet, Legal and General, Klarna and Just Eat shone in terms of application and results through their case studies at the event.  Watch this space for what comes next from Zoopla, Aviva and Vodafone Supertobi chatbot.

 

2. Systems integration for data is dead – long live the separate datastore!

Almost all of the architectures achieving traction and success are based on separate data stores, datalakes or CDPs rather than total systems integration.  Time to think seriously about dumping your current ‘frankenstack’ of systems, attempted data integrations and random data storage.  Say goodbye to the monster, build you data lake house and enjoy a different kind of boat ride.

 

3. Data use is still growing and quality data is the foundation

Although many things in the data world can be seen as a fad or a hype cycle. The fundamental need for good quality data is not. Whether you intend for analysis or gen AI training, reliable and clean data is needed. If you don’t have that you need to start now, otherwise you will be behind within just a few years, playing catchup. The phrase we heard the most at Big Data LDN, as we have heard for years, is garbage in, garbage out. Bad data leads to bad analysis and bad models which has a negative business impact with inaccurate assessment and even data hallucinations.

4. GenAI is becoming more accessible and integrated

2023 was the year of exploratory generative AI, with many people starting to integrate tools such as Chat GPT and CoPilot into their workflow. But 2024 has been the year of integrating them in products and offerings. Look at Microsoft integrating CoPilot into their office package or Google letting you search with Gemini and integrating SGE – search generative experience.  With easier integration tools from Microsoft such as Databricks, or a more user-friendly option, Fabric, there are now ways you can integrate your companies data into Gen AI models with just a few clicks of a button provided you already have the data stored in a accessible location. This means you can have your own AI which can view your business data and make more informed decisions/assumptions to help you answer those ad-hoc questions faster. Your data team's role will evolve to train and prompt these models to answer in a way that your business needs and to give additional insight.

 

5. Align your infrastructure ready for growth and change, and get the foundations right

Gen AI is the current talked about topic, but who knows what it will be in 2-3 years time. If you build your infrastructure all around Gen AI modelling then that will be troublesome when the next big thing comes. It’s important to start building a strong foundation which can be built upon in years to come. This involves getting all your data into an accessible place, whether that being a large Data Lakehouse or a organised data mesh. Without unlimited budget, this will need to be built on a scalable cloud platform. Google BigQuery has been the first to separate their storage modules from their compute modules allowing for increased scalability and protection on investment when better methodologies come out. AI is chimney on your dream house, but if not built on a solid data foundation it is doomed to fail and fall. You may not be ready to build the house now, but you should start on the foundations, so when you are ready you have a strong platform to build on. If you wait, just like now, the foundation wont be there when you need it. A foundation is not sexy, its not something you can show off, but with this you can begin putting the bricks down; a strong analytics reporting dashboard, a robust data security protocol, integrate API data streams. But it all starts with a strong foundation of high quality data, and you must start now.

6. Proprietary data is the key to getting ahead in the data arms race.

In a market where everyone has access to the same cloud systems and the same machine learning models out of the box, the best way to get ahead is quality proprietary data. Data about your company and your business can help refine these OOTB algorithms for your business case. If you have more and higher quality data than your competitors then you will have the advantage. You could always create your own models from scratch, but you will struggle to compete with any sort of gen AI modal without spending millions.

7. Data Pinwheels and integrating feedback into your systems for AI chatbots.

Data pinwheels is the idea of reinforcement learning, allowing AI to learn from its conversations that it has and consistently improve. Klarna used new Gen AI chatbots with this idea, saving over 40 million dollars from their old system. These chatbots and faster and cheaper than human agents and only have a slightly less customer satisfaction. To get more feedback around your system, be sure to integrate feedback into your users workflow. If you can add in a feedback option when and where the user completes a task, companies can see an up to 80% interaction rate.

 

8. Data modelling is not a 1 man job.

Data should be seen as a product and not as a secondary thought at the end of the project. Their should be data ownership throughout the data supply chain; from the raw data, initial movement, processing and reporting. Having people across the business with the data outcomes in mind will help produce the output wanted as well be able to identify issues and resolve problems much faster. Teams of people skilled in data and with business knowledge is required for this level of scalability.

9. Address business problems - not the need for AI

Customers come asking for an ‘AI’ when this may not be the best solution. We should use AI as a potential answer but not a target solution. A better question would be “We need a solution for problem x”. More often than not there are quicker solutions to get the useful information that people need. The existence of this info may already exist, so we should try to not repeat effort. But for this we need a data team with knowledge and the ability to take a step back and think of better solutions rather than fast ones.

10. A lot of users want to share data, but watch out because data governance is now so much more than GDPR

According to GDMA - 75% of users not overly concerned about privacy with 31% just open to sharing, 46% ready to share to get value while 23% don't want any data share out of principle.  So that explains your cookie consent percentages! While there is great openness to data sharing many professionals have only really thought about GDPR, UK GDPR and other equivalent legislation around the world.  In reality there are 4 grounds you can apply to use data, and consent is only one.  There are current 70 laws governing digital in the EU that have data usage considerations, and in the UK the Kings Speech announced another one on its way from the new government.  Anonymous data sits outside most of this but if you bind it to other information it becomes instantly subject to a lot of legislation, and consent can become easily invalidated with changed use or combinations of data.  Really picking your way through what you can and cant do is a hugely valuable journey though, as from this exercise will come the use cases that you hadn't thought of or didnt think you would ever be able to implement.

 

At Appius we have so much more information on these subjects and case studies available on request, and can put you in touch with the source of the insights and how to get access to videos of presentations etc.  Just contact us for a no obligation chat or information exchange on any or all of the points in the key takeaways. Our expert strategists and data scientists in our Experience Team would love to share thoughts in this progressive and exciting arena, and help you find the solutions and opportunities that relate to your current status and organisational needs