Why scribbling is the key to truly understanding your data

Note: This piece first appeared in VentureBeat.

When you think of your workday, which description sounds more accurate? Is it a day of relentless movement from crisis to crisis, email after email, meetings without agendas and long status reports? Or is it a day of scribbling and doodling?

If I had to guess, it’s probably full of the former. In fact, the idea of scribbling at work is probably laughable, but it shouldn’t be. Because it’s the scribbling that can lead you to brilliant, unexpected places.

Scribbling is especially valuable when it comes to data analysis. Simply put, the goal of analyzing data is to find insights and communicate them in an optimal, engaging way. But you don’t reach this goal by first deciding which chart types you want and then adding the data. Saying something like “I will start with a bar chart” and adding data to it is the equivalent of trying to create a masterpiece with a paint-by-numbers set.

Instead, you should start with freeform exploration and allow yourself to shift perspectives. Look at your data from one angle, then another, then another. Let yourself explore the data and find the trends and stories in your data.

You might try out 100 different views and keep only one, but that single output will be a powerful articulation of your data. And all the other views provide small contextual validations that you gained on the journey to the final output.

What does the first step of this freeform approach look like? Well, how about this squiggle?

Damien Newman used this squiggle to describe the design process. Does data exploration follow the same path?

The start is going to be fast, messy, and unpredictable. You’re failing fast and moving on. And over time, you’ll focus on the insights you’ve found and gradually hone in on the best articulation of the data.

Unfortunately, most of us are reluctant to approach a problem or task with a squiggle-first mentality. Yes, some level of process and order is naturally important, but too much apprehension causes us to be conservative in our thinking, which creates a bottleneck for creative breakthroughs.

Lots of people observe something they feel might be important, then don’t go beyond being surprised. But the creative process, especially when it comes to understanding data, is all about exploration. It’s about sailing into unchartered waters.

Exploration has paid off in other industries, too. Consider Alexander Fleming, who discovered antibiotics. Fleming was famous for the disorganized way he worked. He loved to leave everything out in a glorious mess so he could periodically look for the unexpected. In fact, he wasn’t looking to discover antibiotics or something similar. But he was open to improvisation and one day noticed what turned out to be penicillin.

So, what does the squiggle look like with real data and how can we make squiggles a bigger part of our workdays? I got an answer to the first half of this question during Makeover Monday, a weekly exercise in reimagining data visualizations.

We were looking at data on US executions, and I captured a screenshot of every view I made (this was automated!). My analysis took 90 minutes. In that time, I built 302 different views of the data. The final dashboard is shown at the top of this post.

Here’s what the exploration looked like:

You can see all phases of the squiggle in the animation.

At the start, I use rapid exploration. During this phase, I discover the data has a similar shape to Simon Scarr’s Iraq’s Bloody Toll visualization and decide that, based on that insight, I could base my final output on the design of that infographic.

The middle phase is interspersed with periods of formatting the time chart and periods of exploring afresh. I’m not restricted into doing one thing or another. As I play with the data, I get new ideas to go try out. I’m honing in on something, but still exploring.

Finally I spend a big chunk of time finessing the main chart. This is the phase when I’m trying to find the absolute best way to represent the data.

I created tables, bubbles, area charts, pie charts, sideways charts, ugly charts, and more. I threw most of them away. What the animation shows is what happens when an analyst is in the flow with their data. This was the end result:

I am very pleased with this visualization, which I could have only reached via freeform exploration. But I’d be remiss if I didn’t mention the fact that a perfect visualization does not exist. Instead, a “perfect” visualization is simply the best representation of your data at that time.

This is an important point as it helps build the culture of trust that underscores the squiggle mentality. This means embracing new ways of thinking and realizing that a data visualization should elicit more questions. It also means being able and willing to tear down and start fresh if the visualization is no longer serving its goal.

If you simply choose the chart everyone says you should use for a particular type of data, how can you be sure you’ve found the best story, let alone the best way to show it? By allowing yourself the room to explore and iterate, you’ll have more creative breakthroughs, uncover more data insights, and share your findings in the most engaging and effective way.