Evil Works

Google Trends Data, or any normalised data for that matter, can be a real pain to work with. Our founder and Chief Community, Client and Product officer Leigh got a painful reminder of that when she tried to use it to create a YouTube video about motivation. Thankfully, painful problems make good content and instead Leigh was able to turn it into a talk on working with difficult data, the transferability of data science and the number one data science skill you need to develop. So thank you to the Data Science Festival for the opportunity to share this as part of the 10 Year Big Birthday Bash! Data Science Festival have now made the live recording available to watch on their YouTube channel so click here to go have a look: https://lnkd.in/eMWs7TGJ #DataScience #MachineLearning #GoogleTrends

One of our Founders, Leigh Collier will be speaking at the Data Science Festival in May! Sign up to get your ticket now!

Super exciting project to work on and looking forward to sharing more!

In case you missed it, our first in a series of thoughts about how to turn Google Trends Data into something you can actually do Data Science with got published in Towards Data Science! We'll be following up with some more thoughts on this topic very soon...

We've been published in Towards Data Science, talking about how we can modify Google Trends data to turn it into something we can actually do data science with!

Google Trends is used in journalism, academic papers and Machine Learning projects too so I assumed it was mostly safe, if you knew what you were doing. Turns out there’s a fundamental property of the data that makes it very easy to mess up, especially for time series or machine learning. Google Trends normalises every query window independently. The maximum value is always set to 100, which means the meaning of 100 changes every time you change the date range. If you slide windows or stitch data together without accounting for this, you can end up training models on numbers that aren’t actually comparable. It gets worse when you factor in: - sampling noise - rounding to whole numbers - extreme spikes (e.g. outages) compressing everything else toward zero I tried to reconstruct a clean daily time series by chaining overlapping windows and stress-tested it on Facebook search data (including the Oct 2021 outage spike). At first it looked completely broken. Then I sanity-checked it against Google’s own weekly data and got something surprisingly close. I walk through: - why the naive approaches fail - how the normalisation actually behaves - a robust way to build a comparable daily series - and why this matters if you want to do ML with Trends data at all Full explanation (with graphs) here: https://lnkd.in/eVAvTBtn

A Christmas video on Machine Learning to get you feeling festive 🎅

Wait, people actually lie about their companies using AI? It’s happened, from products off shelves being identified by human reviewers to “AI coders” actually being human.  So how do we actually spot whether something is using AI for real? Here are a few things to keep in mind: 1. Look at whether the system is really learning or just following simple rules. For example, a toy ball that rolls to random spots might seem clever, but it isn’t learning from your cat’s behavior or getting better over time. It’s just repeating pre-programmed movements. 2. Ask for transparency. Find out what kind of AI the system is using, such as machine learning, natural language processing or deep learning, and understand why. Then ask yourself if that type of AI is truly essential for the product to work or if it is just being used as a marketing term. Most importantly, does it actually make the product better? 3. Look for evolving insights instead of static outputs. Real AI adapts and changes recommendations over time. For example, if you start watching one K-drama like *Crash Landing on You* on Netflix, the system will notice and keep adjusting by recommending more K-dramas until you finish their whole library. 4. Check for evidence. Some companies add the term “AI-driven” to describe what is really just simple automation. If there is no clear technical documentation or explanation of how AI is actually being used, that is a red flag. Evil Works is on a mission to demystify some of the mystique around data science. Come on the journey with us. Join the Evil Lair via our profile.

I didn’t accidentally reverse engineer Google trends.  I didn’t do it because it’s against the terms of service. And anyway, I’m sure the methodology is mysterious and important and not something I’d accidentally stumble across anyway. So I didn’t do that, I promise!! What I did instead was to figure out a way to make sense of the google trends data so we can start to use it to derive all sorts of interesting insights about the world.  Want to know how I did it? Take a look at the comment below.  At Evil Works we’re all about improving the Data Science eXperience (DSX), through better tooling, smarter training and a healthy dose of treachery. Follow us to get involved, or join the Evil Lair via the link on our profile.

Wait, people actually lie about their companies using AI? It’s happened, from products off shelves being identified by human reviewers to “AI coders” actually being human.  So how do we actually spot whether something is using AI for real? Here are a few things to keep in mind: 1. Look at whether the system is really learning or just following simple rules. For example, a toy ball that rolls to random spots might seem clever, but it isn’t learning from your cat’s behavior or getting better over time. It’s just repeating pre-programmed movements. 2. Ask for transparency. Find out what kind of AI the system is using, such as machine learning, natural language processing or deep learning, and understand why. Then ask yourself if that type of AI is truly essential for the product to work or if it is just being used as a marketing term. Most importantly, does it actually make the product better? 3. Look for evolving insights instead of static outputs. Real AI adapts and changes recommendations over time. For example, if you start watching one K-drama like *Crash Landing on You* on Netflix, the system will notice and keep adjusting by recommending more K-dramas until you finish their whole library. 4. Check for evidence. Some companies add the term “AI-driven” to describe what is really just simple automation. If there is no clear technical documentation or explanation of how AI is actually being used, that is a red flag. Evil Works is on a mission to demystify some of the mystique around data science. Come on the journey with us. Join the Evil Lair via our profile.