- A New Twist on Garbage in, Garbage Out, or if it has Graphs, it has Got to be Good
February 17, 2010
September 09, 2010
Research Questions Without Answers
By Peter Van Brunt, PRC
As someone who is a past MRA leader and has been a MR professional for more than 35 years, people occasionally come to me with questions about MRA and the MR profession. Usually the questioner is someone who I may have met at a conference, was referred to me by someone else, or found my name through a Web search.
The question(s) I am asked are for the most part easily answered, but occasionally the question might deal with an aspect of MR in which I lack experience and expertise. When this happens, I volunteer to try to get them their answer. I do this for a couple of reasons. First I know how frustrating it can be for someone seeking an answer to get passed along from one person to another trying to get an answer. Second, at this point, I am usually interested in learning the answer myself and if I just pass the questioner to someone else, I fail to learn and don’t satisfy my curiosity.
Recently, I was called by someone who not only posed a question outside of my area of expertise, but which was so unusual, it had never even been posed to the practitioners who I called. It dealt with CATI interviewing.
The question I was asked was, “Is there an ‘industry’ accepted standard for recordation errors for CATI interviews?” What the questioner was asking about was specifically to what degree does the interviewer make a mistake when recording the answer the respondent provided? This is a more unusual question then you might think. Most of the telephone interviewing facilities I spoke with concentrate on quality control during the interview by monitoring the audio and only occasionally both the audio and video. They usually actively police their interview staff with regard to interview length (to prevent abuse), and they train their interviewers not to lead the respondent or interpret the answers they are provided. However none of those I spoke with measure recordation errors and have come up with the rate at which they occur. Several of them even questioned how they would measure this.
I told the questioner that I couldn’t imagine it would exceed one-two percent based on my knowledge of data entry errors from paper surveys, and the interviewing companies I spoke with thought that that was probably a reasonable answer.
As researchers, we understand that there is an error factor associated with research. Although we frequently only report the error rate associated with the sample size, error is cumulative and can occur at many stages in the process. We need to always be alert to all the places errors occur so that we can minimize their impact on our work.
Do you ever get asked unusual research questions?
What is your opinion about the question I was asked?
Peter Van Brunt, PRC is the President of ReData, Inc.
This is a good topic. With self-administered online surveys, we are definitely seeing respondent errors, which would be similar to the errors you described. The most common one is giving a negative rating, then in the follow-up open-end, they explain that they meant to give a positive response. It makes me want to have an open-end after every rating!
- Ken Roberts 09/09/2010
Peter....An interesting question and one that could be expanded to focus group recruting errors. As you know I field a lot of research around the country and in Europe...I'm constantly amazed by the spreadsheets I receive that list respondents who are clearly not qualified, sometimes it's a typo, sometimes just an unqualified recruit. Although quality control is an essential part of the service I provide to my clients, it's disconcerting to find so many errors on spreadsheets that have supposedly been editted by the recruting facility. Because of this experience the facilties I operate are required to triple edit spreadsheets prior to sending to the client, even then some errors do occur. In a time when there are multiple threats to the future of traditional focus groups, we must be even more vigilant about the quality of recruiting information sent to clients.
- Carl Iseman 09/09/2010