A brief history . . .
In the 1770’s, prisoners were surveyed to determine prison conditions across Europe. In 1880, Karl Marx mailed out 25,000 questionnaires across France to assess employee exploitation.
Mechanisms of Survey Research
Ø Face-to-face surveys
Ø Mail surveys
Ø Telephone surveys
Ø Intercept surveys
Ø Dual survey mechanisms
Ø Rider survey – a limited number of questions are added to a larger, more extensive survey conducted by another researcher(s).
Individuals and groups (focus groups) [Group dynamics may augment responses but the dynamics may also suppress participation.] Respondents are typically interviewed in their home, work place, or research center. Relative to other mechanisms it yields a high response rate and permits a relatively long survey instrument.
Ten advantages:
1. respondents who are visually impaired, illiterate, or not fluent in English may be included;
2. interviewers can explain the meaning of words;
3. respondents can not look ahead to other questions in instrument (following questions may suggest answers to preceding questions);
4. respondents can not see sensitive questions towards the end (e.g., income) before completing survey;
5. the answer to one question can be used as part of the next;
6. missing information and skipped questions can be eliminated;
7. specific respondents can be selected (oldest male in household);
8. open-ended questions are more likely to elicit a response;
9. visual effects such as maps and photos can be used; and
10. allows observations by interviewer such as age, race, family size, housing type, etc..
Eight disadvantages:
1. interviewer effect during response to sensitive questions (income) and administration of the survey in the filed (interviewers avoid central cities, high rise buildings, night surveys);
2. labor intensive = expensive;
3. spatial coverage in the field may increase expenses and time;
4. respondents may be unreachable due to doormen, guards, etc.;
5. time consuming (door-door);
6. highly dispersed population may be impossible to interview face-to-face (university alum);
7. interviewers must be trustworthy; and
8. ethical considerations (intrusion into someone’s home, may put interviewer into dangerous situations).
Mail surveys
A self-administered instrument that is capable of surveying a large population or sample. The mail survey is typically mailed out and then mailed back by the respondent. It is the least expensive survey mechanism with low labor requirements.
Six advantages:
1. absence of interviewer eliminates interviewer effects;
2. allows large spatial coverage;
3. more complicated questions can be used in a mail survey than a telephone;
4. allows consultation among household members;
5. respondents have time to think; and
6. visual cues can be included (maps, photos).
Twelve disadvantages:
1. lower response rate than face-to-face (mail response rate can be increased through reminder cards, incentives such as money, tickets, etc., and self-addressed stamp envelopes);
2. visually impaired, illiterate, and those who cannot comprehend English may not respond;
3. respondents can look ahead and select out questions;
4. complicated wording can not be clarified;
5. missing information;
6. mail surveys are put aside and forgotten or treated as junk mail;
7. address is required for sampling;
8. long questionnaires may be discarded (keeps it short);
9. respondents may find it difficult to respond in writing to open-ended questions;
10. problems locating respondents in offices or companies;
11. difficulty to survey different respondents with different instruments at a given address (drivers vs. mass transit users); and
12. no control over respondent in household.
Combines the characteristics face-to-face and mail with high response rates. Instrument length can be longer than a mail survey but should be shorter than a face-to-face. Twenty minutes is a long interview, ideally it should be less than fifteen minutes.
Thirteen advantages:
1. control over whom is to respond in household;
2. open-ended questions can be asked effectively;
3. skip patterns may be employed (i.e., skip and go back);
4. non-response on question is lower than mail survey;
5. completion time is relatively short;
6. visually impaired and illiterate can participate;
7. respondents can not see ahead;
8. question-reading demeanor can influence response (slight interviewer effect);
9. easy to find interviewers;
10. survey director can oversee procedures and remedial problems;
11. central cities and rural residents can be included;
12. minimal costs; and
13. sampling techniques are simplistic (random digit dialing).
Three disadvantages:
1. one has to have a telephone to be surveyed;
2. difficulties in finding some respondents at home (schedule interview through mail or prior call), screening through answering machines and caller ID; and
3. respondent may have trouble following answer choices (over 4).
Involves the distribution of self-administered instruments or the conduct face-to-face interviews of users of a given facility (e.g., malls, transit system, park, trailhead). Response rates depend on type of facility and population, and the form of the survey (length, type of questions etc.). Intercept surveys allow for randomization.
Four advantages:
1. it may be the only mechanism for reaching a given population;
2. intercepts respondent engaged in the behavior under study;
3. interviewer can observe other information about respondent (age, sex, race, activity [fishing, hunting], direction of travel, etc.) resulting in a shorter survey; and
4. respondents can evaluate an environment in which they are currently located.
Three disadvantages:
1. respondents may pass by more quickly than the time needed to complete the survey;
2. interviewer effect; and
3. interviewer may purposely select respondents on similar race or opposite sex (biases).
Combines two or more of the four procedures (face-to-face, mail, telephone, intercept) and allows the advantages of one to negate the disadvantages of another.
Four advantages:
1. a brief intercept or telephone survey can be used to build a database for a more extensive survey;
2. follow-up reminders and prior involvement in the study will increase the response rate on the second survey (e.g., face-to-face or telephone):
3. response on the longer survey can be compared to the shorter survey to eliminate biases and unreliability; and
4. second survey can include more complicated questions or open-ended questions requiring more detail.
What mechanism is the best? Well, that depends. based on the research question one mechanism may be more workable than another and obvious. Mail surveys are the most common but the response rate is low. If the time frame to data acquisition is short a telephone survey would be most expedient. If the instrument is complicated, a face-to-face interview would be best. Obviously an intercept survey is best for a given population. Budget constraints will certainly factor into the decision. Finally, there is an inverse relationship between the length of a questionnaire and the number of elements in the responding population or sample. Thus, the type of mechanism should consider the size of the population or sample.
In a qualitative design, survey questions have no reference to statistical relationships and are not designed for numeric measurement. In a quantitative design, survey questions are designed to measure critical variables
Open-ended – respondent fills in the answers. Open-ended questions are exploratory and tend to be unstructured. Bracketing occurs in which a situation, reality, or alternative is defined and the response is elicited. The wording is such that there is no right or wrong response (i.e., horizontalization). Open-ended questions facilitate probing (i.e., expansion on a point/issue through probing).
Open-ended questions are beneficial if:
1. limited knowledge exists;
2. the range of responses is large;
3. the range of responses is large and reading or listing would lengthen the procedure; and
4. in variables such as age it is better to have the respondent fill in than ask a closed-ended question with age categories (Note: a higher level of data measurement, ratio scale. It is always easier to collapse a ration to nominal. Again, there is nothing more worthless than a runway behind you.)
The drawbacks to open-ended questions are:
1. respondents do not read the questions or do not read them carefully;
2. in a face-to-face interview the response must be taken down verbatim slowing the process;
3. interviewers must be experienced;
4. self-administered open-ended questions with lengthy responses may be discouraging;
5. self-administered open-ended questions for the semi-literate may be difficult; and
6. problems arise in coding and categorizing the responses.
Closed-ended – the respondent is provided a series of answers from which to choose. Two advantages of closed-ended questions are the categorization and classification of elements and the measurement at a numerical value for each question. However, closed-ended questions may miss certain dimensions of a variable and often a respondent will respond to a question that they have not considered fully. Closed-ended question answer items that can either be a nominal list or an ordinal scale (e.g., Likert scale). A nominal list should be exhaustive and mutually exclusive.
The kinds of answer to be obtained have to do with attitudes (scaled; agree - - disagree, semantic differentials; extremely dangerous - - nothing to worry about), belief (true or false), attribute or fact (e.g., age, race, sex, income, education, occupation, marital status, number of vehicles, crime victim, etc.), behavioral (e.g., smoking, drinking, commuting, etc.), and behavioral intent (i.e., to predict future behavior, if there was a new grocery store at this location would you use it?).
Questions must be:
Ø relevant – i.e., are measuring critical variables or exploring an attitude, behavior or phenomenon;
Ø brief; and
Ø clear – avoid double-barreled questions, jargon, inconsistency in question and/or answer set, overlapping answers, incomplete answer categories, leading questions, secondhand-opinion questions
Format:
1. begin with an explanatory paragraph or cover letter that states the purposes of the study, stresses the importance of the individual subject’s participation, assure confidentiality, and establish cordiality;
2. start off with “warm up” questions that are general and not too sensitive but spark the subject’s interest;
3. introduce sensitive questions toward the middle of the instrument
4. close with “let down” questions to leave the subject with a sense of closure; and
5. thank the subject for their time and participation
Questionnaire design should consider the needs of the interviewer and coder for an interview survey, and the respondent and coder for self-administered forms. The design should linear with good progression through the questions. The aesthetics are very important in self-administered forms and may mean the difference between a response and a refusal.
Development:
1. list data needs (i.e., operationalization, and measurement levels;
2. draft preliminary questionnaire;
3. submit questionnaire to panel of experts and/or focus group;
4. conduct in-house pretest;
5. finalize pilot study instruments;
6. submit instrument of institutional review (human subjects review); and
7. conduct pilot study (using small sample of population).
Cross-sectional – data are collected at a single point in time (Spring 1999 - MSC 516 students)
Longitudinal – data are collected over time
Ø Trend – surveying a particular category (different group) over time (MSC 516 students each semester for the next five years
Ø Cohort – surveying a particular category (population) but individuals in the category may vary (different sample of cohort over time) (Spring 1999 MSC 516 students, with different individuals sampled each time of survey)
Ø Panel – collecting data from the same group (sample) over time (surveying Spring 1999 MSC516 students each spring for the next five years)