It is not that I’m so smart. But I stay with the questions much longer. ― Albert Einstein
Survey Instrument Guidelines
When developing a survey instrument, be sure to adhere to the following best practices:
- Clearly define the question(s) to be answered by the survey.
A survey is most effective if its purpose can be clearly and succinctly stated. Surveys that are vague or overly-broad can become too long or difficult to analyze. Additionally, depending on the question(s) to be answered, a survey may not be the best method
- Focus on using structured, fixed-response questions.
Open-ended questions allow the greatest variety of responses, but are time consuming to answer and require a lot of work to analyze. Fixed-response questions (questions that use pre-populated answer choices for the respondent to choose from) are easier for respondents to answer, easier to analyze, and when well designed, ensure that respondents interpret questions the same way.
- Avoid questions using leading, emotional, or evocative language.
Question prompts must be asked in a balanced and neutral way to avoid bias. Leading questions influence responses in a way that does not reflect respondents’ true opinions/experiences. “The choice of words and phrases in a question is critical in expressing the meaning and intent of the question to the respondent… Even small wording differences can substantially affect the answers people provide” – Pew Research Center.
- Ask questions that can be answered.
A common mistake is to ask questions that most people cannot easily remember offhand.
- Keep your answer choices balanced and include all possible responses.
Attitude/Rating scales should be at 4- or 5-points. Include “I don’t know” and “N/A” options.
- Stay away from double-barreled questions.
Questions should measure one thing. Double barreled questions try to measure two or more things.
- Branch the survey when necessary
Respondents should only be asked questions that apply to them.
- Keep question order in mind:
Survey responses can be impacted by previous questions. Think about the context in which respondents are reading your questions.
- Ask for personal information only if you need it.
If the OIE fields a survey on your behalf, there is no need for demographic items since we can tie confidential survey responses to administrative data. Anonymous surveys that include demographic items MUST ensure these questions are voluntary. Additionally, any race/ethnicity items must include “Italian decent” or “Italian American” as a response option since Italian Americans are included among protected groups at CUNY.
- Don’t let your survey get too long.
The shorter and simpler the survey, the more likely respondents are to answer all the questions and answer them honestly. Complex or long surveys can annoy respondents and cause them to stop responding part-way through or to pick arbitrary answers to get to the end quickly.
You can learn more about survey design here: Writing Survey Questions (Pew Research Center).
Faculty engaging in program assessment can see examples of appropriate survey questions and answer options here:
The OIE can tailor the surveys above for your program and field such surveys on your behalf. Learn more about the kinds of assistance OIE can offer here: OIE Assessment Support
Important Considerations
Institutional Review Board Approval
If the data to be gathered and used are part of an academic research project, the project will need to be reviewed and approved by the IRB to ensure it meets their requirements. If the data are to be used for administrative purposes, it may be exempt from a formal review by the IRB, but the survey must be voluntary and the results kept confidential. You and the College may incur legal liability if the treatment of survey participants is unethical, if data resulting from your survey are misused, or if any part of the survey violates certain protected rights of individuals. Survey researchers should be aware of their responsibilities and make every effort to protect the rights of survey respondents.
The Rights of Respondents
- Survey invitations must clearly identify the group or person who is conducting the survey and provide contact information (typically name and email address) should respondents have questions about the content of the survey or about the use/publication of survey results.
- Participants must be notified that their participation in the survey is voluntary.
- Participants must be notified whether their responses will be anonymous or confidential.
Confidentiality and Anonymity
An anonymous survey means that the researcher cannot determine who participated in the survey and who has not. Anonymous surveys typically do not collect identifying information, and cannot restrict responses to one response per person.
A confidential survey collects information that can identify participants either alone or in combination (e.g., race/ethnicity, gender, major, class year) and allows the researcher to analyze responses along a variety of demographic factors, but also means researchers have a responsibility to protect the privacy of the participant. When data are collected confidentially, it is helpful to let participants know that results will be presented in aggregate only and that any results that can potentially identify participants will not be presented.
If your survey results will include academic or contact information for HMC students, you may be subject to Federal Family Educational Rights and Privacy Act (FERPA) regulations. Non-directory student data (e.g., GPA, Major(s), race/ethnicity) can be used without a student’s consent by college employees for “legitimate educational” purposes, provided the data are not reported in such a way that individual students can be identified. Otherwise, users must receive written consent from students.
Sample vs Census
It is typically not necessary to survey all QC students or all QC employees (census surveys). A sample needs to be large enough to protect the privacy of respondents and large enough for your results to be considered representative, but that does not mean you need to survey everyone. If you wish to target all students or all employees, OIE will typically employ a random sampling method to ensure that these populations are not over-surveyed. To calculate the minimum sample size to be considered representative within a given margin of error, visit Sample Size Calculator.
Pilot Testing
It is always best to pilot your survey to a small group from the population you intend to survey. As an assesment of the survey’s face validity, this can help you understand how the questions you have written will be perceived and answered and whether the survey is too long or overly complicated. Be sure to ask your pilot group whether any survey question was difficult to understand or difficult to answer.
If your survey project requires more rigorous testing, please consult with OIE. We can estimate the construct validity of an instrument using confirmatory factor analysis, and the internal consistency and reliability of the instrument can be estimated with Cronbach Alphas. Factor analysis and Cronbach Alphas can determine whether items in a survey should be edited or removed.
Race/Ethnicity Items
If the OIE fields a survey on your behalf, there is no need for demographic items on the survey such as a race/ethnicity question, as we can tie confidential survey responses to administrative data. Anonymous surveys that require demographic information should ask these types of questions near the end of the survey and these questions MUST be voluntary. Please also note that as per QC and CUNY policy, any survey that includes a race/ethnicity item must include “Italian decent” or “Italian American” as a response option since Italian Americans are included among protected groups at CUNY.
Survey Invitations
We’re more likely to partipate in a survey when (A) the purpose of the survey is clear and (B) we know who is asking the questions. Survey invitations should be short and concise, but explain why particpants’ reposnes are important and how the survey results will be used. Also, be sure to sign the survey invite as the PI. If you are sending a survey to faculty or staff at QC, it can be helpful to send out a longer email to your target poulation prior to the survey invitations so that people know what to expect.
Incentives
Depending on the population and the survey, you may offer incentives to survey respondents. Incentives must be positive (i.e., no negative consequences for failing to participate). They should also be relevant to the topic of the survey, and small enough so as not to be coercive (e.g. gift cards to Amazon, Starbucks, or the bookstore). Keep in mind that while offering incentives can increase a survey’s response rate, it can also invite less meaningul responses. Please consult with OIE if you are considering offering an incentive.
Timing
It is important to request assistance with a survey well in advance. Please understand that your preferred dates of administration may be delayed due to our availability to assist you with your project, or due to a pre-existing survey. Concurrent surveys that target the same population lower response rates, overburden respondents, and lead to poor quality data.
Sharing results
We urge you to share your results in a way that is accessible to the community you surveyed, unless your results are for very small groups. By being transparent with results, the populations we survey can see the value in particpating in surveys.