Your SaaS startup reaches its two-year anniversary, and you lock a new round of funding. Every measure of customer success is strong. Users report high levels of satisfaction. They log in a lot, they like you on Facebook and they read a lot of your emails. In a survey, 90% said theyd recommend your product to a friend. Investors are impressed. Churn is at a high but acceptable level for a young startup, but over the next six months, it fails to improve. Instead, it slowly creeps up to problematic levels and you cant understand why.
Startups get blindsided like this when they rely on feel good data science: big data analytics that mashup qualitative measurements with quantitative science. Being data-driven is the stated goal of most tech executives, but you cant be data-driven just because you wave your magical data science hands in the air. If you want to really understand what your customers think, and whether they are prime for upselling, conversion or churn, you need to strictly separate qualitative and quantitative data. Its time to discover rather than assume what metrics mean, and its time to stop dicing customers into imaginary groups.
We intuitively know that qualitative metrics are unscientific, but they look good. When you take a number like average log-ins and arbitrarily give it a weight of 20% in your customer success algorithm, youre converting it into a qualitative metric. This kills the data science and lulls you into a fantasy.
Unfortunately, that is how most data science is conducted today. All sorts of measurements logins, time spent in the product, engagement with marketing emails, etc. are given subjective weights.
Companies also rely heavily on self-reported data. Customers are often willing to give their satisfaction levels, rate different experiences and declare whether or not theyd recommend the service to a friend. Theres nothing wrong with this data, but if you mash it and weight together with data based on user actions, you spoil the quantitative data.
Stop tricking yourself.
When it comes to understanding a customers probability of upgrading, continuing to pay for your service or unsubscribing, you cannot equate what people say with what they do. Likewise, you cant impose meaning on quantitative data until you establish correlations between actions.
The whole point of big data is to find patterns and trends independent of opinions. However, drive-by data science occasionally running large-scale data science projects to uncover correlations is common and misleading because the conclusions begin to decay immediately as your customer base, onboarding process, marketing campaigns and other variables change.
An even bigger problem is the practice of pre-assigning meaning to data. For instance, you could (smartly) assume that your most active users are most likely to upgrade. And you could be wrong.
One way is to routinely take random samples of SaaS users and split them into three groups: a random control group, the most active users (those who log in most) and an algorithmically-selected group that we identified as most likely to upgrade by applying machine learning to a large number of behavioral inputs for each customer. Then observe.
Read the original post:
Enough With Feel Good Data Science
- Dan Ariely: It’s OK to cheat and steal (sometimes) - April 7th, 2010 [April 7th, 2010]
- Full Sail Behavioral Science Leadership Podcast - August 27th, 2011 [August 27th, 2011]
- Full Sail Behavioral Science Passion - August 27th, 2011 [August 27th, 2011]
- 13H Refutes CCHR: Industry of Death (Behavioral Science) - August 28th, 2011 [August 28th, 2011]
- College of Social and Behavioral Science - August 28th, 2011 [August 28th, 2011]
- GB: Behavioral Science Bill For Propaganda - August 29th, 2011 [August 29th, 2011]
- Criminal Minds - The Behavioral Science Part 2 - August 31st, 2011 [August 31st, 2011]
- Santhula Hospital - A clinical and research center for Nuero-behavioral Science - August 31st, 2011 [August 31st, 2011]
- Behavioral Science Self-Assesment Project - September 2nd, 2011 [September 2nd, 2011]
- Behavioral Science: OPTIMAN - September 3rd, 2011 [September 3rd, 2011]
- Forget the Cinderella effect: stepparents are just as likely to kill their biological children as their stepchildren - September 4th, 2011 [September 4th, 2011]
- The Behavioural Science Blog in 2010 - September 4th, 2011 [September 4th, 2011]
- introduction to behavioral science - September 4th, 2011 [September 4th, 2011]
- CSU's Behavioral Sciences Building - September 5th, 2011 [September 5th, 2011]
- Criminal Minds Season 2 - Backstage - Behavioral Science Real-life Criminal Minds (1 of 2) - September 6th, 2011 [September 6th, 2011]
- Behavioral Science: Goldielocks - September 6th, 2011 [September 6th, 2011]
- Dean Welcome Message - CalSouthern School of Behavioral Sciences - September 7th, 2011 [September 7th, 2011]
- Criminal Minds - The Behavioral Science Part 1 - September 9th, 2011 [September 9th, 2011]
- TEDx Bandung-Roby Muhamad-Socio Behavioural Science - September 14th, 2011 [September 14th, 2011]
- How Behavioral Science Applies to Marketing - Jason Anello - September 14th, 2011 [September 14th, 2011]
- Public Health Behavioral Science - September 15th, 2011 [September 15th, 2011]
- Full Sail Behavioral Science Video - September 15th, 2011 [September 15th, 2011]
- Introduction to Behavioural Science - September 15th, 2011 [September 15th, 2011]
- University of Arizona College of Social and Behavioral Sciences Commencement Speech - September 17th, 2011 [September 17th, 2011]
- Introduction to Behavioral Sciences OSPE - September 17th, 2011 [September 17th, 2011]
- Get to know NAU's College of Social and Behavioral Sciences - September 20th, 2011 [September 20th, 2011]
- 3 Minute Thesis 2010 runner-up - Will Harrison, Faculty of Social - September 24th, 2011 [September 24th, 2011]
- New Behavioral Sciences Building at Colorado State University - September 27th, 2011 [September 27th, 2011]
- Intro to Psych Statistics - September 28th, 2011 [September 28th, 2011]
- Why Do Voles Fall in Love? - September 29th, 2011 [September 29th, 2011]
- Behavioral Science: What problems do you have in your life? - September 30th, 2011 [September 30th, 2011]
- Behavioral Solutions for Climate - October 3rd, 2011 [October 3rd, 2011]
- College of Social and Behavioral Sciences (SBS) 2010 Senior Celebration - October 4th, 2011 [October 4th, 2011]
- Enhance Student Teaching Podcast - Full Sail University Behavioral Science Class - October 4th, 2011 [October 4th, 2011]
- Music and the Brain: Depression and Creativity Symposium - October 4th, 2011 [October 4th, 2011]
- Experiment Test - October 4th, 2011 [October 4th, 2011]
- Helen Palmer and The Enneagram - October 4th, 2011 [October 4th, 2011]
- Rapid Learning: Behavioral Science And Patient Treatment - Behavioral Physiolgy - October 4th, 2011 [October 4th, 2011]
- Meet Nicole in the Colorado State University Behavioral Sciences Building - October 4th, 2011 [October 4th, 2011]
- Life As A Full Sail Student Ep. 5 (Behavioral Science) - October 5th, 2011 [October 5th, 2011]
- Happy holidays from your friends in the College of Business and Behavioral Science - October 5th, 2011 [October 5th, 2011]
- Behavioral Science Jumping Hurdles Full Sail University - October 5th, 2011 [October 5th, 2011]
- Dr. Meera Narasimhan on Mental Illnesses - Part 1 - October 5th, 2011 [October 5th, 2011]
- Science needs you: Test your word power on iPhone / iPad with Science XL free app - October 5th, 2011 [October 5th, 2011]
- Definition of Culture - October 5th, 2011 [October 5th, 2011]
- OSP Presentation: Bou's Clues (University of Santo Tomas, Behavioral Science) - October 6th, 2011 [October 6th, 2011]
- Behavioral Sciences OSPE Part 7: Informational Care - October 6th, 2011 [October 6th, 2011]
- Criminal Minds Season 2: Behavioral Science - October 7th, 2011 [October 7th, 2011]
- Behavioral Science - York College CUNY - October 7th, 2011 [October 7th, 2011]
- Miswakology Behavioral Sciences - October 7th, 2011 [October 7th, 2011]
- Criminal Minds Season 2 - Backstage - Behavioral Science Real-life Criminal Minds (2 of 2) - October 11th, 2011 [October 11th, 2011]
- Behavioral Science (A Lesson In Self Control) - October 12th, 2011 [October 12th, 2011]
- BEHAVIORAL SCIENCE LECTURE 1 - Video - October 13th, 2011 [October 13th, 2011]
- TEDxGoodenoughCollege - Jan-Emmanuel De Neve - The Genetics of Happiness - Video - October 13th, 2011 [October 13th, 2011]
- Using Denial to Cope with Grief and Depression - Winifred Gallagher - Video - October 13th, 2011 [October 13th, 2011]
- How To Survive Full Sail University - Video - October 14th, 2011 [October 14th, 2011]
- Life Positive Expo 2008 Appreciate Inquiry with Dr. Wasundhara Joshi and R Sankarasubramanyan - Video - October 14th, 2011 [October 14th, 2011]
- Dr. Meera Narasimhan on Mental Illnesses - Part 3 - Video - October 14th, 2011 [October 14th, 2011]
- Learning and Memory - Video - October 16th, 2011 [October 16th, 2011]
- Behavioral Science Proposal - Video - October 17th, 2011 [October 17th, 2011]
- 13H Refutes CCHR: IOD (Chapter 7) Psychosurgery (1/2) - Video - October 23rd, 2011 [October 23rd, 2011]
- 13H Refutes CCHR: IOD (Chapter 8) Psychiatric Drugs (1/2) - Video - October 23rd, 2011 [October 23rd, 2011]
- OUTDATED : Psychology and Behavioral Sciences Collection - Video - October 24th, 2011 [October 24th, 2011]
- CSUDH College of Natural and Behavioral Sciences - October 24th, 2011 [October 24th, 2011]
- Behavioral Science - October 24th, 2011 [October 24th, 2011]
- Behavioral science project. ( just one part of it) - Video - October 24th, 2011 [October 24th, 2011]
- Ramp Modeling for Dickies (UST- Behavioral Science Presentation) - Video - October 24th, 2011 [October 24th, 2011]
- Search in Psychology and Behavioral Sciences - PSCY 2301 - Video - October 25th, 2011 [October 25th, 2011]
- Forgiveness: Jesus vs Behavioral Science - Video - October 25th, 2011 [October 25th, 2011]
- PsyD Graduate Discusses her Learning Experience at CalSouthern School of Behavioral Sciences - Video - October 25th, 2011 [October 25th, 2011]
- Behavioral Science Statistics: One Sample and Dependent t-te - Video - October 26th, 2011 [October 26th, 2011]
- Full Sail University Behavioral Science Self-Awareness Exercise - Video - October 26th, 2011 [October 26th, 2011]
- FBI Behavioral Science Unit Interview - Part 2 - Video - October 26th, 2011 [October 26th, 2011]
- Full Sail BS Lab 2 Proposal - Video - October 27th, 2011 [October 27th, 2011]
- Full Sail University-Behavioral Science- Jumping Hurdles Project - Video - October 27th, 2011 [October 27th, 2011]
- Being Human - Behavioral Science Project - Video - October 27th, 2011 [October 27th, 2011]
- Behavioral Sciences OSPE Part 3: Compliance - Video - October 27th, 2011 [October 27th, 2011]
- Behavioral Science Film Fest feat. BSA1D '09 - Video - October 28th, 2011 [October 28th, 2011]
- Dan Pink Plenary Speaker Landsdowne Residency, March 2010 - Video - November 7th, 2011 [November 7th, 2011]
- Psychology and Behavioral Sciences Collection - Video - November 8th, 2011 [November 8th, 2011]