Scripps College Survey Administration, Use, and Reporting Guidelines

Overview

Surveys can be a powerful tool for gathering information directly from populations of interest. However, surveys sent out to Scripps’ population raise important ethical and practical concerns regarding issues of privacy, protection of human participants, and questionnaire fatigue from over-surveying certain groups. While surveys that are part of research that may be reported outside of the university (e.g., to academic journals or professional associations) must receive Scripps College Institutional Review Board approval (for more information, please visit http://www.scrippscollege.edu/academics/irb), surveys that do not require IRB approval should also consider and address many of the same issues and concerns. Thus, this document is endorsed by the College and designed to provide guidance for those creating surveys that do not require IRB approval. Specifically, this document offers a resource for minimizing some of the risks of surveys by summarizing best practices, while maximizing the benefits through improving protections of privacy, questionnaire quality, and reliability and validity of data.

Developing a “new” survey & survey administration

• Before you start developing your survey and collecting data, you should explore whether Scripps already has a survey developed and/or data collected that might answer your research question(s). Please contact the Office of Assessment and Institutional Research to identify possible surveys or data that may be made available to you so you do not have to build a new survey or gather data.
• Surveys can be administered in person (e.g., hard copy) or online (e.g., Qualtrics).
• When designing your survey, please think about the ethical issues.
  • Who are your survey respondents? Consider whether your respondents are vulnerable and/or have the ability to voluntarily choose whether or not to complete your
    • If respondents are minors, you may need to gain consent from the minor’s legal guardians.
    • Be mindful of power differentials between survey administrator and survey respondents and avoid coercion or the appearance of coercion. For instance, when a boss asks employees to complete a survey about their workplace, the employees may not truly feel as though they can choose whether or not to participate and/or provide honest responses.
    • If respondents are members of smaller groups, be mindful of possible survey fatigue as these individuals may be asked to complete surveys from many different sources.
  • Surveys should minimize risks to respondents and maximize benefits. Before conducting a survey, you should determine that the benefits of conducting the survey (e.g., knowledge to be gained, direct benefits to respondents) outweigh risks (e.g., psychological distress, embarrassment, harm to social standing or reputation, threat to one’s employment or financial standing, physical harm) to respondents.
  • Survey respondents should be able to give voluntary informed consent to participate in the survey. An introduction page to your survey should include information on the following nine components so that respondents have a working knowledge of what their participation entails.
    • Who is conducting the survey (e.g., person or department)?
    • The general purpose of the survey.
    • A description of what participation in the survey entails (e.g., respondents should be given a sense of what topics they will be asked about during the survey or what they will be asked to do during the survey), including an estimate of how long the survey may take.
      • If you will be asking about sensitive topics, please be sure to explain this to respondents in the consent form
    • What are the potential risks to the respondent in completing the survey? Please be sure to outline any potential risks including such things as psychological distress, harm to social standing or reputation, threat to one’s employment or financial standing, physical harm, etc. If a survey is anticipated to have little risk, then please state this for potential respondents.
    • What, if any, are the potential benefits to the respondent for completing the survey? For instance, will respondents receive any compensation (e.g., payment or entry in a raffle for a prize) for participating in the survey?
    • Information on type of data collected. Please indicate whether respondents’ responses will be anonymous, confidential, or identifiable. For some topics, potential respondents may feel uncomfortable in responding or they may not provide accurate or useful information if they can be personally identified.
      • Anonymous surveys do not collect any identifying information (e.g., name, student ID, email address) about the
        • For online studies, the survey creator should be sure that the survey program that they use does not collect IP addresses, as IP addresses can be traced to particular computers, and if the computer is a personal computer, then back to a particular For example, the default option on SurveyMonkey is to collect IP addresses, so if one is planning to conduct an anonymous survey, one should change the option so that data are collected anonymously (i.e., not collecting IP addresses).
      • Confidential surveys do collect identifying information, but the identifying information is protected so that only the survey administrators collecting the survey data will be able to link specific respondents to their responses.¹ When the survey results are shared, it will be done at an aggregate level or without sharing identifying information.
      • Identifiable surveys collect identifying information, and may report this information with responses linked to a specific respondent. These surveys are rare, but require that you notify respondents that they will be identifiable in the informed consent process as some people may not wish to participate if they can be identified.
      • Please note that if you have indicated that you will be collecting confidential or anonymous data, then you should be cautious when you collect data from only a few respondents or when combinations of responses to non-identifying questions can result in identifying For example, a survey that asks for a student’s race, number of years at the college, and major could end up linking responses to a specific student, even though each individual question does not ask for identifying information. In cases like this, be careful about how you report the data (see Reporting Data) and you should explain to potential respondents what you plan to do in this situation.
      • Please also note that there are some limits to the confidentiality that you as the survey administrator can offer. And respondents should be made aware of these limitations as relevant to your survey.
        • If you were served with a subpoena from law enforcement, there is no survey administrator-respondent confidentiality that would allow you to refuse to comply with law enforcement.
        • In the event of learning about child abuse or imminent danger to the respondent or others, you may be required to report this information to the proper authorities.
      • Information on how the data from the survey will be Who will have access to the raw and summarized data results?
      • A statement that participation in the survey is voluntary. If respondents feel coerced to participate, they may not provide you accurate or useful survey data. Allow respondents to decline to respond or opt out of taking the survey if they Allow ways for respondents to either skip sections not relevant to them or to indicate that they did not attend or do not have an opinion with a Not Applicable option
      • Contact information for the survey administrator in the event of questions or problems.

Data Access, Collection, and Storage

• Survey administrators or the designate stated in the survey introduction should be the only ones who have access to the “raw” survey data files.
• Survey administrators must protect the privacy of respondents by securely collecting and storing the data.
  • Paper surveys should be stored in locked cabinets and should not be accessible to anyone other than the survey administrators.
  • Online platforms like Survey Monkey must be secured with a password to access the survey data.
  • Data containing identifying information should be saved in computers or drives that belong to the university and must be stored securely.
• If any survey data is to be shared beyond what the consent form of the survey indicated, the appropriate VP that oversees the population surveyed should be notified to obtain permission for the data release; all identifying information should be stripped from the data before it is released.

Reporting Survey Data

• Unless individual survey respondents have given their consent, data reports should not disclose confidential or personally identifiable information about respondents.
• Regardless of whether a survey is anonymous, confidential, or identifiable of the respondent, survey respondents could identify someone else in the Scripps community in their responses in a way that may impact their reputation, employability, or safety. Owners of the survey should mask references to specific people made by respondents, and aim to present data primarily in summarized format or make the report confidential with notification that there is identifiable information included to protect those identified.
• Consider the audience for your survey data.
  • When there are small numbers of responses in particular categories, one should consider presenting the data at a higher aggregate level to protect the identity of survey respondents. For instance, instead of reporting data from 3 students in a small major (e.g., Classics) where doing so could allow the audience to identify particular students, you could report the data from students majoring in the Humanities, which includes these students but allows you to protect their identities within a larger group of respondents.
  • Designate the report as confidential (e.g., for a designated internal audience) if you have small response counts that you want to include; otherwise do not report results with fewer than five respondents. Reporting very small response counts can jeopardize the privacy of those respondents.

Some practical tips:

• Shorter is generally better. Respondents are more likely to complete your survey if it is brief and focused.
• Timing. Many surveys, particularly surveys of students, take place at the end of the academic semester. For better response rates, avoid busy times of year, like midterms, finals, or the end of the fiscal and academic year. Consider launching your survey at other times of the year.
• Keep the language simple. Avoid using jargon.
• Present questions on a similar topic together. Do not switch back and forth between topics.
• Avoid double-barreled questions. For example, a question that asks “The professor arrived for class well-prepared and on-time” is double barreled as it is possible for a professor to have been well-prepared but late, or on-time but not prepared.
• Consider the strengths and weaknesses of both open-ended and closed response options.
  • Open ended: Open ended questions pose a question and leave a comment field for respondents to respond. An example would be “What was most helpful about the workshop?”
    • Open ended questions can be a source of rich data that raises themes you might not have seen otherwise. However, they typically take much more time to process and summarize, make comparisons across respondents more difficult, and can provide more opportunities for respondents to identify themselves or others, jeopardizing privacy.
  • Close ended questions provide responses that respondents can select among. An example would be a question that asks respondents “Please indicate how helpful you found the workshop: Not at all Helpful, Somewhat Helpful, Helpful, Very Helpful”.
    • Close ended questions are easier to summarize (e.g., 70% of respondents indicated that the workshop was Very Helpful) and can provide more opportunity for direct comparison (e.g., between one workshop and another, or between the same workshop this year and last year). But close- ended questions can limit the information that respondents give you. This can be mitigated somewhat by providing an “Other” box along with the other responses, or a comment box for respondents to elaborate on their responses if they wish.
• In an anonymous survey, remind respondents not to provide personally identifying information in their responses.
• If you will be asking respondents about sensitive information, before these questions add a reminder that respondents can skip questions that they do not feel comfortable answering.
• Given the limits to confidentiality, you may wish to avoid asking about potential illegal And if there is a reasonable expectation that such information could be offered in response to a survey question, you may wish to include a reminder about the limits of confidentiality before such questions.
• If there is a reasonable expectation that respondents may discuss other people in their responses, one should remind respondents that they should avoid providing information about others. They instead could use pseudonyms or other similar tactics to avoid identifying others.
• When creating your informed consent/introduction page, you may wish to consult the “Sample Consent” forms available on the Scripps IRB website for sample language to address the various issues described above (deleting all references to the IRB or IRB approval).

1 At Scripps, most institutional surveys are confidential, rather than anonymous. This allows us to analyze responses along a variety of demographic factors, but also gives us the responsibility of protecting private information.