Creating a Data Driven Culture by Avinash Kaushik

30 minute presentation– I think 10 slides. And what I want to do is cover
this very, very difficult topic of creating a data driven
culture, because all the data and technology
coolness is great. But this is very, very
hard to pull off. It’s really hard to
fundamentally change the culture in your organization,
especially organizations that are big. It actually gets much harder,
the bigger they are. In fact, I find that it’s not
unusual for me to go and give a talk and show many of the
testing experiments and for the most part lots and lots of
tests and experiments are actually being done by
small companies. Actually the bigger the company,
the dumber it is in terms of it getting
to a cutting edge. It really is. I think it’s something about
the culture, and something about the pace at which they
can move in the bureaucracy and the agility that gets
lost when you get big. So it’s sort of a interesting
little problem. This topic is also
in the book. So you can read more,
because I will run through this quickly. So you all have a book,
which is great. It’s very heavy. It took me a long
time to write. My publisher Wiley– they just
called me in September or October and said– after six months, I had started
blogging– and they said we should make
this into a book. And I said who the hell would
want to pay for a thing that’s already free? It turns out a lot. But the core reason that I
decided to write the book is that all of the proceeds that I
make from the book actually go to two charities. My wife and I decided that
everything we make from the book will go two charities, and
the first one is Doctors Without Borders. It’s a Swiss charity. It does some really amazing
work around the world. They’re a charity that’s known
to be first in every sort of troubled spot in the world, and
they won the Nobel Peace Prize in 2003. It’s just an amazing charity. We love it. And the other half of my
proceeds go to the Smile Train, which is a charity that
does cleft palate surgery in the third world. It’s a pretty terrible
condition, emotionally, not physically, emotionally, for
a child to go through. And it only costs $50 to
actually do the surgery. And so we support
the Smile Train. So thanks to Brett and Google,
you all get a book. My unbiased opinion is
this a awesome book. But it also raises money for
charity, which is something that’s great about a book. Lots of things you’ve
heard today– hopefully you get lots of ideas,
and you’re excited to go back and do things. And often people want to– because they are really easy. I’m just going to go, and
I’m just going to do it. And a bunch of them are. And lots of things that I
especially talked about this morning, with the GA
talk is how easy it is to do these things. But often, we have to have an
inspirational call there. But often, things are actually
not quite straightforward. Life is tough. And I’m a huge fan
of Guy Kawasaki. You’ll see me thanking him in
the preface of the book. And this is a quote that I
take from Guy Kawasaki. It’s true. If you want a roast duck,
you have to work for it. Nothing is easy in life. And I think this quote captures
it very well. So I’m going to go though five
things that I have come to learn during my experience
that help create a data-driven culture. Because it’s hard to get
it right, but you can. And the very first lesson that I
have learned is always focus on the outcomes, especially
at the outset. And I like to say this story
that when I first got to my former employer, they were
using Webtrends. And they were publishing
300 Webtrends reports. So I walked around. I did not know anything
about web analytics. It’s embarrassing. Just a few years ago,
I knew nothing. And so, I met everybody for two
weeks, started to figure out what this web thing
is, and this was just four years ago. It’s more embarrassing. And then at the end of my second
week, I simply turned off Webtrends. I simply switched it off. And no one in this
$2,000,000,000 company actually called me, not a single
human being called me. And it was not that Webtrends
is a bad tool. Webtrends is actually
a pretty good tool. It’s that all of this massive
data puking did not focus on outcomes. Because at the end of the
day, all of us are motivated by outcomes– either outcomes for yourself,
or outcomes for the company. It’s really, really important
that if you want to get start or end up with creating a
culture that thrives on data, you have to connect to people
at a very, very deep level. And there is nothing deeper than
focusing on figuring out how I, as the person, director
of analytics, can help you make more money by getting
your bigger bonus. People respond to you helping
them improve their salaries. No amount of visitor trends help
John improve his salary. The amount of orders I
get from jelly beans improves his salary. And my job, as the director of
data, is to figure out how I can help him sell more
jelly beans. So at the end of the year, he
will get a higher bonus. It seems kind of odd, but tons
of reports that sit out there don’t focus on outcomes. The very first thing we did is
we said we’re going to hack together a data warehouse, quick
data warehouse, and all it will report is
on one number. How much revenue did we make? And the very first time we put
this number in front of the CEO, we shocked him
and he hired a VP. And the reason he hired a VP,
because he was surprised at how much money we were
making on the web. And so he hired a
VP for the web. Because the web was that
important to the company. And remember, they had 300
Webtrends report available all of this time. And yet they had no idea
what the outcomes were. So at the start, create
a simple dumb graph that plots revenue. Each color in this stacked
bar represents a product. I’m not going to tell
you what, obviously. But it helps them understand
two simple things. How much money do we make? What are we selling? Everybody responds to that. Because it’s the bottom line. So as you focus, as you go
back, if you want to prioritize the actions you want
to take, the first filter that you should put on is am
I reporting on bottom line? Because people care about
bottom line– theirs and the company’s. Then you’ll be ready to
go back and say– after a few months of doing
this, they’re like oh! Why are we making
so much money? Where are people coming from? They’re ready for this. And then at some point they’ll
say, ah, I know Google is great or whatever
this is saying. And then they’re ready to say
what is happening on my website that is causing
me to make money? It’s a gradual transition. What happens in reality in many
companies, is it’s easy to throw a tag on the website
and you start going the reverse way. It increases complexity, and
there’s less skin in the game for people you’re
working with. Solve for outcomes, and you
will create a culture that will demand data. I’ll give you this– another great example. The first time we did a test at
Intuit, we built a platform and ran a test– actually for Quicken. John’s from Quicken. And we created a test for
Quicken and it was a simple test. You know, three different
[UNINTELLIGIBLE] images is as simple a
test as you can get. And what I said at that time
is why don’t we bet on it? So we have this idea three,
let’s everybody bet on which one is going to be the winner. It’s only a dollar each. Everybody had skin
in the game. The next day, in the earlier
[UNINTELLIGIBLE] Tom, you get data. The next day, everybody was
checking in to see if they’re going to win the pool. Because they had skin
in the game. And people would come up to
me and say oh, what’s the definition of this
conversion rate thing that you’re measuring? I’m not winning. I think this is wrong. Great, right? People got skin in the game. And everybody was interested
in knowing how multi-grade testing worked. They all wanted to know what
the winner is going to be. And by the way, they all lost.
The techie guy won, because the interesting thing is when
we said in order to win the pool, you have to predict
which recipe would win and by how much. And so they all predicted– it’s sort of easy to say which
recipe might win, but then everybody said improvement in
conversion rate will be 45%, 15%, 19%, 28%. The improvement in conversion
was 1%. Important lesson
learned, right? You do small things,
you win small. But all of this happened
because we focused on outcomes. This is another very, very
important thing, and the reason that I think a lot
of companies fail– the smartest people in each
company are probably sitting in this conference
room, sadly. But you are. You’re the smartest people
in the company, because you know data. And you want to play with data,
and you want to use data to make decisions. In some sense, you
already get it. But yet I find that especially
as you approach the HiPPO level, people have a deep
distrust for data. Here is the other facet
of this problem. You know every nuance of how
the data is collected, processed, and presented. You are the smart ass. And when you present data up,
remember the person on the other end of the table or the
phone is not as smart as you are about data. They obviously got there
because they have other attributes. They not that smart
about data. So what happens is a lot of
times they’ll say oh, you’re saying this thing is bad about
this recipe, or this [UNINTELLIGIBLE]
is not working. They think it’s your opinion. It’s very, very, very important
that if you want to create a culture where people
use data, you have to bepersonalize decisio-making. It cannot be your opinion about something, not even remotely. If people think it is your
opinion, and you’re not a very nice person, you’re screwed. There are lots of
simple ways to depersonalize decision making. Every single report you put out,
you should pass through this filter. Does this report meet
the criteria that this is not my opinion? This is data speaking. So a simple way to do it is
something that I had shown you many screenshots of
this morning. Give context. We got a million people this
month, and it is bad. Oh, it is bad, because last
month we had three. And we have 3,000,000
versus 1,000,000. So do a simple graph that says
oh, the blue is how we did last year, and the red is how
we’re doing this year. Or the red is the goal, and the
green is how we’re doing. This is not your opinion
anymore. Whether the performance is
good or bad is not your opinion anymore. Depersonalize it. Or you can compare it to other
benchmarks and indexes. Like, for example, you cannot go
to your CEO and say we suck at 1% conversion. We can tell him just
published a study that says that an average commerce
retailer– top 300– converts at 2.3%, and our
conversion is 1%. The message that you
suck is included. But it is not your opinion. Right? A lot of analysts make
this mistake– a– because they are smart,
they can do math, and they portray data as their own
opinion and it’s not. Another one is– I am a huge fan of
doing surveying. And so this is the American
Customer Satisfaction Index. You can go to the
and you can actually get satisfaction scores for the
last 10 odd years for any company you want. So if you’re running a customer
satisfaction survey, you can say to your CEO,
our score is 53, and Marriott is 79. Or Marriott is only 79. We’re 96. You get a bonus right
away obviously. Because it’s not your
opinion anymore. And that’s the same reason
that I like testing experimentation, because
it’s not your opinion. You will always lose
against the HiPPO. You have no chance of winning. Or very little. But it’s easy to say let’s put
it up there– a, b, and c, and see which one wins. In this case, that’s the
control, that’s the one I like, it’s clean and
it has a heart. Angels are missing. It’s clean, right? The interesting thing is the
woman cooing soft nothing into the guy’s ear did not work. This did not work either. It’s the woman and what appears
to be an androgynous person winning. And if that was your idea, your
HiPPO would have never accepted it. But now it’s not your opinion. It’s the customers voting. In all examples, whether you
create index against yourself, you compare against others,
or you’re in testing and experimentation, all you’re
doing is depersonalizing decision-making. You have to take the
people around the table out of the equation. It’s not our opinion. And then, it turns out people
are more willing to listen. People listen to customers. Most reasonable HiPPOs listen
to customers and then listen to the bottom line. They listened to comparisons
to others in the industry. But they don’t want
to listen to you. Not as much as you’d like. And the other one, obviously,
is if you want to actually create a data driven culture,
you have to figure out how to empower your analysts. I mentioned this morning that
there is a huge difference between doing reporting
and analysis. Reporting means everybody
gets the data and nobody takes any action. But everybody gets data. If you actually want to create
a culture where you want to have a lot of data being used
to drive decisions, you have to figure out how to empower
analysts in your team. You have to make sure that
they have 20% time for reporting, 40%, 70% time for
doing unstructured analysis. If you don’t allow people to go
explore and find insights, how are you actually going
to create a culture that thrives on data? Reports are not going to tell
you what to do by themselves. And the other important thing
is if you do have analysts, encourage a culture
of risk taking. I’m astounded that people hire
an analyst for $100,000 a year, and then make him into
a reporting monkey. They will get somebody from
Harvard and Yale, and then rather than letting this
extremely brilliant person with their Ph.D. to do the kinds
of analysis that he has learned to do, they will say
take a, multiply it by b, and then divide it by c,
and that is what you’ll do day and night. And that’s not how you
create a culture that thrives on data. You take risks, and
you do big things. And the reason is this wonderful
little graphic. It is actually amazingly true. When you touch data, you have
absolutely no idea how big your problems are. If you don’t take risks, if you
don’t spend time in doing unstructured analysis, you’re
actually not going to be able to figure out how big
your problems are. And the next one is this
philosophy that I have. It’s called the Trinity Strategy. It’s covered in the book in
some detail, but it was developed after a little while
of me thinking through all of the problems around the web,
especially the web. And at the center of the Trinity
Strategy is what I had said this morning, is the desire
to find actionable insights and metrics that
drive change, not data. And one facet of the Trinity
Strategy is doing to click-stream analysis– what you would do with Google
analytics, lots of quick-stream analysis– is understanding the
patterns of people. And that’s good. But it’s important to realize
that there are a lot of weird people in the world. And they all have ADD. And they are all using
your website. It’s astonishingly sad,
but it is true. So as you analyze trends and
patterns in your click-stream data, it is not unusual for
you to struggle to find a semblance of insight, because
people do weird crap on your site. So when you attack click-stream
analysis, the important thing to remember is
you’re inferring intent. And hence the second element
of the Trinity is really important to do outcomes
analysis. If you have a support website,
figure out how to measure problem resolution rates. If you have an e-commerce
website, figure how to measure revenue. If you have a lead generation
website, figure out how to measure conversion. But the second element
of the Trinity is to analyze outcomes. But the thing that I have
learned that creates data driven culture is this
number three, which is experience analysis. It’s doing testing,
experimentation, lab usability studies, surveys– all of these wonderful things
that help you getting into the head of all this collective
people who had ADD. It’s me trying to get into your
head as you’re clicking around on our site to
figure out what the hell are you thinking? There are many simple and
complex ways of doing it. You could simply run a survey. Make a survey and say hey,
what do you think? You could do tests. That’s a way of getting into a
customer’s head and saying what is going to work for you? But another way is
doing remote. There’s this company in San
Francisco called Ethnio– E-T-H-N-I-O. It’s
a great company. And the wonderful thing about
Ethnio is you can buy their services, and then show a DHTML
window on your site as people come, and you say would
you like to make $50? Everyone does. Who doesn’t want $50? And if you say yes, they fill
out a quick short survey– say what’s your age group? What products do you use? How long have you
used our site? Do you use our competitors’
site? Literally anything you want. As soon as you meet
the criteria– you say fill out your
information, say yes, it comes over to this live database, and
I have your phone number to call you. And I call you and I say hey,
John, you wanted $50– no, I don’t say that– you wanted to participate
in a study with us. So you press this button, and
boom I can see exactly your browser window. I teach them to think aloud for
two minutes, and say, hey, here’s how you think. Whatever you’re doing,
just tell us. And then we sit back, relax,
and let the person go. Say do whatever you’re
doing on our site. And it will blow your mind
when you see the person behaving on your site, telling
you what they’re thinking, or what they’re looking for, and
then struggling to actually solve their problem. It gives you amazing insight
into a person’s head as to what they’re thinking. And often you want to cry, very
often you want to cry, because the interesting
phenomenon about human beings is that a lot of these people
who participate in these experiments, often if
they’re not able to complete their task– I want a prize, I want to order
something, I want to give [UNINTELLIGIBLE] I want to know how much
this product costs– they tend to blame themselves
very quickly. And they say oh I’m so sorry. I’m such an idiot. I can’t do this simple thing. The reality is you are the idiot
for making this human being feel bad. You should have figured out how
to get the price in front of them really quickly. That’s doing experience
analysis. There is a lot that it takes to
understand the experience of your customer, so that you
can understand the behavior that they have on the site. And the goal of the Trinity
Strategy is to understand the experience so well using all
this methodologies so they can influence their behavior which
leads to the right outcomes. That’s the goal. As you go out and want to create
a data driven culture in your companies, remember
this is what you have to solve for. And the reason is because at the
end of the day, we’re not analyzing cookies and sessions
and Javascript tags and other BS like that. We’re actually trying
to solve for people. And it is very, very often that
we all forget that we’re analyzing people. Some of them are frightened by
you– obviously, as this woman on the left. Lot of people love you. Lot of people are screwing
with you. And your goal, as you do all of
the analysis, when you log in to Google Analytics tomorrow
morning– which all of you will, yes?– you should try and figure out
and remember that you’re solving for people. And you’re trying to analyze
people, and not cookies. Because it’s so easy in our web
world to just forget that you’re actually analyzing
people. The last one is a particular
point of view of mine is that web analytics over the last
few years has evolved tremendously. Decision-making on the web
has evolved tremendously. We started with log files,
and analog– which is one of the old
log file parcels that are still around. My personal point of view is
that web analytics as a responsibility should sit
with the business, and not the IT team. It sounds crazy, but in a lot of
companies, web analytics is still owned by IT teams. And
that’s OK, you know? But the interesting thing I have
found is that IT people have completely and utterly
different requirements from data than do business people. As the web has matured and
become an integral part of any business, it’s no longer IT that
is driving the website. It is the business team
and the market that are driving websites. So my personal point of view is
the person who should own web analytics in your company is
the person whose neck is on the line if the site fails to
deliver for the customer. Sometimes it’s somebody in
strategy and operations. Often I find a lot of
organizations meeting success by having web analytics owned
by marketing, because marketing is responsible for
creating the experience. You can figure out what is the
right outcome for your own company, but it is important to
realize that it is probably an optimum outcome to have
business own data and decision making from data. Because of this one singular
reason, you have to think, imagine, and move at the
pace of business. And the pace at which a business
moves is radically different from a pace
at which IT moves. And it’s not that one is better
than the other, but they both solving for
different things. Inside this complex graphic
that you should not read– that I will do as a part of
an hour presentation– but rather than having a lot of
different silos on pieces of the puzzle from collecting
data all the way to making decisions in your companies,
because of all of the transformations with Javascript
tags and ASB models and hosted vendors, a lot of
this complexity is going out of the equation now. So you can stuff a lot of
things to the vendor and create a responsibility in
your company that is end-to-end and owned
by the business. It’s really, really important
that if you want to create a data driven culture that you
have a data team, an analytics team, that is owned by the
business and reports to the business, because they will make
sure that they kick them enough so that they’re doing
the right thing. That’s it.


  1. Masamune SHIMAOKA said:

    What does a data driven culture mean? Is it meant that a data has the power to drive culture-stuff into the solution we need?

    September 15, 2007
  2. Masamune SHIMAOKA said:

    Yeah,i think he is useing the words "creating a culture" as the meaning of "creating the Uni-purpose group of people".Well,i am wondering if which is the "DRIVE-side"or"Driven-side".A data or a group of people???

    September 17, 2007
  3. Masamune SHIMAOKA said:

    Oh,i am becoming to understand this lecture.Thank you all.

    September 18, 2007
  4. Masamune SHIMAOKA said:

    And i should have said "beging" instead of "becoming".And i want to say "Thank you!!!" again.You guys are so gentle.

    September 19, 2007
  5. Masamune SHIMAOKA said:

    Oh NO! "BEGINNING" hahaha…..

    September 19, 2007
  6. Huangmao Quan said:

    Wonderful speech, very useful for our web administrator… but in reality how to transfer/combine tasks of IT and Market department is still a myth ;(

    September 19, 2007
  7. azagen1 said:

    very good explication good professor.

    November 5, 2009
  8. akshat mathur said:

    So what he is saying is…that our decisions should be based on Data & not our Opinions and assumptions. Adding on to that he says that it should be a Bussiness group of people who should take actions on the data…

    October 25, 2010
  9. MDK83 said:

    @peacefulakshat Doesn't this make sense? Shouldn't our marketing campaigns always be based on what our customers want? Pretty straight forward. In addition, of course business analysts should look at the data, that's why we have charts and other data reports, so that people who understand the business, the market, the industry can analyze what the data means. Anyone can read it, the question is, why it is the way it is, and how our business can change it. We're running businesses not websites!!!

    June 7, 2011
  10. Amgad abou rabia said:

    10 years everything is changing(technology, algorithms, channels, etc..) but i don't thing so about data.. Data not changed
    Thank you avinash

    December 20, 2017

Leave a Reply

Your email address will not be published. Required fields are marked *