Getting data

If you’re within earshot of the tourism industry, which is always the case on Saipan, you’ll occasionally get a dose of tourism data that’s based on self-reported survey data. Worldwide, it’s a common practice to get questionnaires into the hands of incoming tourists, often as arrival paperwork given to airline passengers. The questionnaires might ask about income, marital status, trip duration, age, and so on.

The practice isn’t limited to tourism, of course. In the U.S. mainland it’s fairly common for retail stores and restaurants to have terms printed on receipts that offer small discounts in exchange for dialing a phone number and taking an automated survey. This is usually a quality-control gig, which is different than scraping up demographic data, but we’re still dealing with the self-reported element here.

Back in the day, I was sometimes involved with designing studies that used questionnaires. This was generally in the tourism, travel, and heathcare industries. The main virtue of questionnaire data was that it was generally easy to get, and, once gotten, it was always easy to compile. There were, however, some drawbacks.

One problem is that self-reported information usually comes from a self-selected pool of people. For instance, going back to my receipt example, a pool of restaurant customers that’s willing to spend five minutes on a phone pushing buttons in response to automated questions, all to save $3 on their next purchase, is likely to be far different than the pool of customers that’s not willing to do this. The issue is how closely the self-selecting pool aligns with the restaurant’s overall target market.

Another problem with self-reported data is the accuracy of the answers. What people say, and what they do, can be very different things.

For example, in just about any industry, if you ask people how important quality is, they’re likely to respond that it’s of the utmost importance. Nobody likes to think of themselves as cheesy, low-quality cheapskates. But when actual market transactions are tallied, the results can tell a different story. Indeed, at the time of purchase, quality can fade into a mere abstraction, while saving money has a more substantial ring to it.

And then there’s the “halo effect” from my healthcare days. A doctor wanted to develop a questionnaire to ascertain compliance with his post-treatment instructions. The goal was to correlate this behavior with the eventual success of the treatment. The doctor eventually surmised, however, that the least compliant patients were the least reliable reporters of their own behavior. Well, imagine that! We dubbed this the halo effect, so, while that little project turned out to be a big waste of time, at least we learned something from it.

So, in actuality, it’s often more insightful to have human Q&A sessions than to blindly bask in the glow of self-polished halos. Of course, a Q&A session is a long, slow, and expensive process, so you’re not going to get many data points this way.

But it can be scaled-up via forming what’s called a “focus group.” This is a group of people who are willing to discuss their notions about a particular product, industry, or whatever. Conducting these things is usually the domain of experts, lest things become a useless and open-ended bull-session.

However, I’ve known several business owners who maintained informal focus groups, generally a selected cadre of customers who the owner trusted to provide opinions on various matters. This approach requires charm, effort, and social skills that encourage people to say what’s really on their minds. This can deliver a deeper form of feedback than you’re going to get with the static realm of questionnaires or forms.

Another way to get insights is to get out of your office and spend some time in the field. If you’re in tourism, for example, it can make a lot of sense to just sit in a few hotel lobbies, or tourist-intensive locations, and observe things with an open mind. I’ve done a lot of this. It always added texture to the dry stacks of data that were on my desk. It’s not easy. Observing is no better than the observer, especially when you’re looking for things that you don’t even know that you’re looking for yet.

Well, such are my observations and war stories. Now, if you’ll excuse me, I’m going to get a Big Mac and then have a cold one with my pals at happy hour. This means, let’s see, oh, about 45 minutes of exercise time and two hours of spiritual development, so let me just fill out this questionnaire on time management and I’ll be running along.