OSU Web Accessibility Center

A Survey of Survey Tools

This page was last updated: 25 July 2008.
Dates of evaluation of survey tools are in the “Roll-Call of Tools” section.
A “Changelog” at the end of the document lists changes for the updates to this page.
The content and comments on this page are the professional opinion solely of the Web Accessibility Center and do not represent the opinion of The Ohio State University or any organizations or units affiliated with the WAC.

This document presents an overview of a number of popular tools for conducting surveys online. We have no hard statistics on which of these tools are currently being used at OSU. In all but one case, however, we have either taken surveys delivered by OSU entities via one of these tools or have been contacted by web administrators who have used or currently deploy the tool within their department or unit. At the end of this document we mention a few other tools we came across in our research.

“In all but one case”: The only survey tool examined here that we have not heard mention of at OSU is SurveyGizmo. We discovered SurveyGizmo through an email list archive at WebAIM, an organization that promotes web accessibility. It got favorable mention. Hence our other selection criterion: high usability, especially for people with disabilities.

In the comparisons and analysis below, we attempt to assess the degree to which the survey tools—for both the administrator and survey-taker—are accessible to keyboard alone and to a screen reader. Also important in this capacity, particularly for people with cognitive disabilities, are a logical page organization (visually and at the level of the HTML), easy recognizability of both the progress through the survey for the survey-taker and the various feature sets available for the administrator, and visual usability factors such as use of open/white space, proximity of related elements, and visual contrast between elements that might help in survey-takers making selections on multiple-choice and matrix question types.

For accessibility testing:

• We ran a screen reader (JAWS) against all of the interfaces, both survey-taker and administrator. How difficult was it to navigate the interfaces and accurately fill in information?
• We attempted to perform all input with keyboard alone, specifically using the tab key to navigate form elements. Can a user tab and arrow through the interfaces? Is it clear where the cursor is while doing so, that is, do form or other elements clearly visually indicate focus? Is it necessary to use the mouse for essential functionality?
• We set the computer into a high-contrast rendering mode. Are all essential page elements still visible and is contrast uniformly enhanced?
• We enlarged the screen fonts. Does enlarging fonts distort or “break” the interfaces?
• We made subjective appraisals of the clarity, organization, and contrast of the visual presentation. Does does styling and layout aid visual recognition of the functionality or purpose and significance of the various elements of the interfaces?

It is relatively easy to make static form elements accessible. And, by and large, survey-taker interfaces in all of the tools lack dynamic interface elements. That is, there is little AJAX or use of JavaScript to manipulate display, though some use JavaScript for tooltips and to trigger selection of elements in question types. We found more extensive use of dynamic techniques on the administrator interfaces, which tended to make them less accessible. Here is some of the markup we looked for:

• Use of headings to introduce questions or sections of content
• Use of labels to associate descriptive text with form elements
• Use of fieldsets and legends to group and identify related form elements
• Proper markup of tables with headers in matrix question types

Roll-Call of Tools

Without further ado, the patients currently on the operating table:

• SurveyGizmo (commercial, hosted, reviewed 25 July 2008)
• SurveyMonkey (commercial, hosted,reviewed 25 July 2008)
• Zoomerang (commercial, hosted, reviewed 25 July 2008)
• Checkbox (commercial, server installed or hosted, reviewed 4 April 2008, version 4.4)
• LimeSurvey (open-source, server installed, reviewed 4 April 2008, version 1.71)
• Snap Survey Professional Edition (commercial, survey configuration via desktop application (Windows only), manual publishing to web for data collection, reviewed 25 July 2008)

For our tests, we composed a survey in each tool. The first seven questions in each survey are either identical or as close to identical as we could muster, given the idiosyncrasies of the tools and question types. We then imagined 10 survey-takers and had “them” fill in each of the five surveys overlapping questions identically. We did this to get a sense of how the statistics collected by each tool stack up head to head.

Question Types

We are listing only those features available at all package levels and try to constrain our choices to question types that are common across more than one survey tool. All of the remotely managed (non-locally installed) survey tools provide more question types and, in some cases, greater flexibility in collection of data (uploads of data files, for instance) at the higher cost package levels.

Basic Question Types Available in the Survey Tools

	SurveyGizmo	SurveyMonkey	Zoomerang	LimeSurvey	Checkbox	Snap Survey
Text Field/Short Answer
Text Area/Long Answer
Table of Text Fields
Radio Button Multiple Choice (high-contrast comparison of radio and checkbox controls)
Checkbox Multiple Choice
Drop-down Multiple Choice
Table of Radio Buttons
Table of Checkboxes
Table of Drop-downs
Rating Scale	(customizable)
Rank In Order of Preference
Star-based Ranking (high-contrast view of star ranking)						(Snap star-ranking is a horizontal group of radio buttons with stars above each increment. Because of this implementation it is accessible, though it doesn't behave like the JavaScript-enabled star ranking most users are familiar with.)
Pre-Set Contact/Demographic Information Fields	(customizable, with validation)
Choose Between Images	(alt text and captioned, screen reader but not keyboard-only accessible)					(alt text and captioned, screen reader and keyboard-only accessible)
Continuous Sum	(percentages, numbers toward a total)					(percentages, numbers toward a total)
Admin Can Add Instructions or Other Copy
Admin Can Add Video & Images	(with alternative text descriptions and styling hooks)	(images)		(images, by editing template)	(images)	(images and video)
Progress and Other Feedback	(feedback in charts or graphs (in pay-for packages) and progress indicator)
Make Answers Mandatory			(not all question types)
Field-Level Validation	(some limited in free package, extensive in pay-for packages)	(limited to specific question types)		(through limits and regular expressions)	(pre-defined formats, such as social security number, postal codes, etc.)

Special Question Types/Features Available in Pay-For Packages

Three of the survey tools (SurveyGizmo, Checkbox, and Snap Survey) are able to “pipe” responses from one question to other questions in the survey, thus controlling the flow of the survey. SurveyMonkey, Snap, SurveyGizmo (at pay-for package levels), and Checkbox can randomize the question order, and SurveyGizmo will even randomize rows within a matrix. SurveyGizmo and Checkbox have APIs for customized integration.

Reporting Features

Packages and Costs

Administrator and End-User Accessibility

The survey-taker experience was our primary focus in evaluating the survey tools, but we also tested the administrator interfaces accessibility with a screen reader and attempted all input using only the keyboard. We assign letter grades to the online surveys. We rate relative levels of accessibility for the administrator interfaces.

Survey Accessibility Grade Scale

For the survey-taker interface to get a grade of “A”, all of the question types must be fully accessible to screen reader and keyboard-only users. Means of determining progress through the survey must also be accessible. All question types must be usable when screen fonts are enlarged—and the fonts must be enlargeable via the browser—and when high-contrast settings are enabled on the Windows OS. Matrix tables should use table headings on both column and row. All form elements must have proper labels. Headings should be used appropriately to aid navigation through the survey. Error reporting, including indication of required fields, must be unambiguous and put focus in the offending field or to an error reporting box with links to bad fields.

For the survey-taker interface to get a grade of “B”, the majority of key question types must be fully accessible to screen reader and keyboard-only users. Means of determining progress through the survey must also be accessible. All question types must be usable when screen fonts are enlarged—and the fonts must be enlargeable via the browser—and when high-contrast settings are enabled on the Windows OS. Matrix tables should use table headings on at least the top row of the matrix table. All form elements must have proper labels or the screen reader must not make any errors in determining label text for elements. Headings should be used appropriately to aid navigation through the survey. Error reporting, including indication of required fields, must be clear enough for the user to easily locate the offending question and understand the error.

For the survey-taker interface to get a grade of “C”, some key question types must be fully accessible to screen reader and keyboard-only users. All question types must be usable when screen fonts are enlarged—and the fonts must be enlargeable via the browser—and when high-contrast settings are enabled on the Windows OS. Some form elements need to have proper labels or the screen reader must not encounter fatal errors in determining label text for elements.

For the survey-taker interface to get a grade of “F”, most types of questions present severe access problems via keyboard or screen reader. Screen fonts do not enlarge or break the visual design when they enlarge and high-contrast settings make some inputs impossible to use. Proper labels are not used for identification of form elements.

Administrator Accessibility Levels

Practically Accessible —
For the administrator interface to be “practically accessible”, a keyboard or screen reader reliant user must be able to get to all controls involved in setting up and publishing a survey with little effort, and there must be no ambiguity introduced in the process by dynamic interfaces, unexpected cursor focus changes, etc.

Functionally Accessible —
For the administrator interface to be “functionally accessible”, a keyboard or screen reader reliant user must be able to get to all key controls involved in setting up and publishing a survey with little effort. There may be some ambiguity in using dynamic interfaces, but these must be surmountable via documented workaround or through a reasonable amount of experimentation by the user.

Technically Accessible —
For the administrator interface to be “technically accessible”, a keyboard or screen reader reliant user must be able to get to all key controls involved in setting up and publishing a survey. That is, it must be technically possible to use the key interfaces—access must be feasible, though not necessarily practical.

Not Accessible —
For the administrator interface to be “not accessible”, a keyboard or screen reader reliant user would be unable to set up and publish a survey.

Summaries of Survey Tool Accessibility

SurveyGizmo

grade

B+

Most of the question types in SurveyGizmo are screen reader and keyboard accessible, with the exception of the “image choice” and “star-ranking” types, both of which can be worked around using other question types. The “rank in order of preference” type is accessible to the screen reader but not the keyboard. Text fields highlight to show cursor focus but not radio, checkbox, or other question types. Images are all given proper alternative text, labels are properly associated with form elements, and matrix tables have column headers that are properly scoped, though row headers are not proper HTML table headers. Validation or required question errors reload the page and display a high-contrast error box at the top of the screen. However, the error box is not focused on page reload (it should be) and the only way we have to determine the required fields is by browsing the page. Adding to this difficulty is the fact that the “required” star indicator is not within the label tag (so it does not get read by the screen reader when tabbing to the control.

Though conscious regard was given to accessibility in the design of SurveyGizmo’s, in our opinion, most of the accessibility comes from the fact that SurveyGizmo tends to closely follow web standards. They have a page dedicated to the accessibility of SurveyGizmo surveys, which links to a short review comparing SurveyGizmo and SurveyMonkey, by Karen Mardahl and Lisa Pappas in the Usability Professionals Association Voice. (Note the information on SurveyMonkey appears to be out of date, or at least does not reflect the current status of screen reader accessibility in SurveyMonkey.)

Admin. Interface: Between Functionally and Practically Accessible — The administrator interfaces in SurveyGizmo are a model of usability. Interfaces are clean, navigation elements are easy to find, and “flow” through the creation process seemed to us more intuitive than any of the other survey tools. There are some limitations to accessibility but they can be worked around. Specifically, when choosing question types, a right-hand panel updates to allow the administrator to enter data. There is no indication of this screen update, and screen reader users would need prior instruction or experimentation to figure it out.

SurveyMonkey

grade

B

SurveyMonkey has put significant effort toward making their standard survey interfaces screen reader accessible. They have made very clever use of labels within most of their question types. A representative example: In the radio grouping type of question, the first response repeats the question before announcing the label. So, if the question were “Whom do you like better?” and the responses were either Batman or Robin. The label on Batman would read “Whom to you like better Batman”. The first part of the label is hidden from the visual user but provides excellent orientation for screen reader users. SurveyMonkey uses a similar trick in matrix tables and goes beyond SurveyGizmo in providing row headers in tables. Though they are not scoped, screen readers are pretty adept at handling two-dimensional tables, so long as TH’s are available. Another smart feature that is not implemented in SurveyGizmo is the focusing of question errors. If you “continue” from a page and have an error in your response, SurveyMonkey not only reloads the page, it focuses the question with the error. Better would be focusing a box that listed the errors and filled the box with hyperlinks to the problematic questions, but we feel that SurveyMonkey’s approach is closer to the mark than SurveyGizmo’s.

Where SurveyMonkey falls down, however, is in keyboard accessibility for for users who can either see the screen or need to set high-contrast settings. Though at the level of HTML, SurveyMonkey is using actual radio buttons and checkboxes, as opposed to Zoomerang, SurveyMonkey covers the checkboxes and radios with images that provide a unique look to those common form elements. Unfortunately, this disrupts keyboard accessibility, as one cannot see when the group of elements has the focus solely by tabbing. More dramatic a problem arises when the user turns on high-contrast settings in Windows. Because of the way these settings are implemented in Windows, background images are disabled though the physical space they occupy is still blocked out. Thus, with high-contrast on, we no longer see any indication that checkboxes or radio buttons either have focus or are selected.

Note: a help topic on SurveyMonkey accessibility states that, to address screen reader access, the designers put a hidden link in the surveys that would point screen reader users to an accessible version. SurveyMonkey appears to have dropped this feature and instead incorporated screen reader accessibility into all surveys.

Admin. Interface: Not Accessible — It is not possible to use the SurveyMonkey administrator interfaces with keyboard alone. There is no highlighting of elements and the user has no idea where she is or what she can do to make actions occur. Though the screen reader can read the contents of many interfaces, the modal dialogs that are used very widely throughout the administrator interfaces do not seem to get the screen reader focus when spawned. Thus it is virtually impossible for the screen reader user to figure out where she is in the dialogs (or if she is even in them at all).

Zoomerang

grade

D-

Zoomerang is accessible only to mouse users. It was not possible with the keyboard to get through all of the questions. In tabbing, the tendency was to get stuck in a question, moving over and over its options, throwing JavaScript errors with each press of the tab key. The tool uses no labels or headings, leading to ambiguity in selecting answers and a general inability to easily navigate the survey. While it was technically possible to get to all of the questions using a screen reader, since the tool does not implement checkboxes and radio buttons using standard HTML form elements but instead uses custom widgets, it is very difficult or impossible to determine what checkboxes or radio buttons have been selected. Additionally, since the interface is not tab navigable, the survey-taker has to move in and out of form input mode. We could not imagine either a keyboard or screen reader user being able to complete any of the very simplest of Zoomerang surveys.

Admin. Interface: Not Accessible — Some of the initial administrator interfaces can be navigated via keyboard and screen reader. However, once the user moves deeper into the interfaces to begin either editing or creating new surveys, all screen reader and keyboard access breaks down entirely. The tools implements three or four frames per page. Typically, the user gets stuck in the topmost frame and cannot get to the content of main body/survey-editing frame. This was the case both via screen reader and keyboard. For instance, though JAWS can typically pull up a list of frames or move through them by repeatedly pressing the “m” key. However, in the Zoomerang administrator interfaces it was not possible to use these navigation techniques to get to the survey creation editing tools.

LimeSurvey

grade

C+

LimeSurvey users proper HTML labels for text fields and checkbox and radio groupings. A table header row is used in one- and two-dimensional matrix questions, but the left heading column is not implemented with table headers. Like Checkbox, errors in filling in a survey are given through a JavaScript pop-up that does not indicate which questions have problems. Also like Checkbox, we get no help from the page. It is not updated to indicate errors and it is up to the user to attempt to figure out. The “ranking” question type is neither screen reader nor keyboard accessible and lacks elegance in appearance. Similarly awkward is the pop-up calendar for date inputs. It is not accessible to screen reader or keyboard users and is amateurish in appearance.

Admin. Interface: Between Technically Accessible and Not Accessible — The administrative interface uses very few dynamic effects and is easily tab navigable. Some key images lack alternative text descriptions and, though there are headings on the pages, they are few (many interfaces have only a single heading, despite the fact that there are many logical sections on the interfaces). One of the biggest problems for keyboard and screen reader users is the mandatory use of the FCKEditor (or variant) for input of questions. The editor cannot be disabled and is neither screen reader nor keyboard-only accessible.

Checkbox

grade

C

Checkbox uses proper HTML labels for checkboxes and radio button groupings. The interface uses deeply nested tables for arranging questions, which can cause problems with screen readers. Table-based matrices use no table headers and are confusing for a screen reader to navigate. Error reporting is through a generic JavaScript pop-up message, indicating only that there are errors on the page. It’s up to the user to determine which questions have problems.

Admin. Interface: Technically Accessible — The administrative interface uses very few dynamic effects and is easily tab navigable. However many images lack alternative text and there are no headings used.

Snap Survey Professional Edition

grade

B+

The Snap home page has a prominent link to information on Snap Survey accessibility. The page notes that “Snap Professional is able to produce questionnaires which meet or exceed the highest level of web accessibility. It is able to produce questionnaires which meet Level 3 compliance (sometimes known as AAA compliance). Producing questionnaires to such a demanding standard can largely be done automatically by choosing the appropriate web publication options. Some though require care from questionnaire authors in avoiding certain layouts and applying appropriate settings.” The page links to pages that benchmark the survey tool features against both WCAG 1.0 and U.S. Section 508.

Basic user configuration recommendations having to do with accessibility on the site's pages on Section 508 and WCAG 1.0 include adding alternative text to images, ensuring link text is understood out of context, and allowing for input of “custom HTML,” such as language changes and abbreviations. More rigorously accessible “strict rules” can be enabled when producing “plain text” versions of surveys, which the tool can create automatically. The plain text versions are not text-only, rather they are simplified HTML versions. The strict rules affect positioning of field labels, expand grids into a single column, and wrap related fields—from a row of an expanded table, for example—in a fieldset. The strict rules plain text export does not appear, however, to use fieldsets to wrap groups of related checkboxes and radio buttons—the ideal usage of fieldsets, in our opinion. The strict rule on label positioning attempts to ensure that labels get placed in the code in such as way that a screen reader would automatically make the association, even in cases where “there is no support for explicit labelling.” We could not determine when this was the case; proper HTML labels are used in every question type we tested. As mentioned, the strict rule on expanding grids busts a grid-style question into separate question groups for each row in the grid and then wraps the question groups in fieldsets to disambiguate each form control. Strict rules also remove all use of font tags, allow for external style sheets to be used, a tab order to be set, place-holder text to be added to fields, and adds accesskeys to reset and submit the survey questions.

We were impressed that the tool makes accommodations for screen reader input even for complex question types. The horizontal slider-based rating question type has text input fields with proper labels. So, while a screen reader or keyboard user cannot access the sliders themselves, it is possible to fill in the text inputs—though the fields only allow whole numbers to be entered where using the slider with a mouse generates floating point numbers. JavaScript validation on the ranking (one to N) question type guarantees unique numbers in each field. However, while the error responses (in JavaScript alert boxes) are technically precise, they are not always user-friendly. For example, putting a bogus response in the second field of a question, for example, the fourth in the survey, produces this error response: ”Number invalid or out of range for question Q4.b.” Errors which referenced specific fields within questions were all similar to this and difficult to comprehend. However, other sorts of error messages were clear. There is no visual or screen reader cue of required fields by default. When moving through the survey, you would only find out whether a field was required after you submitted the page.

When a page loads with a new question on it, Snap puts the focus in the first input field on the page. This is a convenience for most users, but screen reader users will need to navigate up out of the field to read the question.

The image choice question type uses radio buttons and an image. Because of this implementation, the question type is accessible both through keyboard and screen reader. SurveyGizmo's implementation is more elegant looking, using mouse highlighting and a green check overlay for the selected image, but it is not screen reader or keyboard accessilble.

In matrix-/grid-style questions, column and row headers are true HTML table headers and have the “scope” attribute set properly. This potentially is very beneficial for screen reader users. However, matrix-style questions insert extra, empty columns inbetween answer columns to padd out the design. This may cause screen reader users some confusion, as table based navigation is made more difficult.

Overall, throughout the surveys, Snap has made overt decisions to achieve high accessibility. And its plain text export is unique. We have to wonder how often it would be used, though. We believe the goal should be a single accessible product, not two versions, though we could see the plain text version being published, along with the regular version, as an alternative for screen reader users.

Snap was developed and the company is located in England. England's laws on web accessibility cover private and public entities, requiring both to comply, unlike in the U.S. where 508 covers only entities fully funded by the Federal government. We can speculate that Snap's extensive work toward accessibility is motivated by its primary market.

Admin. Interface: Not Accessible — The administrative interface is a Windows-only, installed, GUI application, which facilitates creating survey questions, outputting the surveys into HTML forms (for the web and hand-held devices) and to PDF, importing survey results from an online service, and creating sophisticated reports. The interface is complex but appears to be efficient for creating and interpretting surveys once its wealth of features have been learned. In the JAWS screen reader and using the keyboard alone, however, it was not possible to use the interface beyond accessing the main menu system.

Summary of Findings

SurveyMonkey has made strides toward screen reader accessibility. However, the difficulty survey-takers using the keyboard alone would have in completing surveys and the problems with high-contast display prevent us from scoring it above the low “B” grade range. Overall, administrator-side usability and the high accessibility of the survey-taker interface (with some exceptions for certain question types) compels us to score SurveyGizmo best of the group, overall. Snap Survey Professional Edition also merits a high “B”. All of its question types are accessible and it has the ability to produce alternative versions that are geared specifically toward high accessibility. It has a few problems with error handling and focusing of pages. But outside of that the surveys it produces are highly usable, and even some complex, JavaScript-reliant question types (slider, for example) are accessible. Snap's installed GUI is currently not accessible. Zoomerang scores lowest. It is inaccessible to the keyboard alone and not practically accessible for a screen reader user. Out of the box, LimeSurvey provides decent accessibility for most basic question types, including matrix-style questions, which include at least a single header row. Error reporting is insufficient. (On this score, SurveyMonkey is currently the best for screen reader users.) Checkbox has a more polished appearance than LimeSurvey but is roughly equivalent in accessibility to LimeSurvey, though its matrix questions lack proper markup.

Changelog

A summary list of changes to this document.

Changes for 1 May 2008 update

• Original composition of this web page was finalized. The first round of testing was completed April 4 2008.

Changes for 25 July 2008 update

• SurveyGizmo claims in June 3 2008 change log post that they now group questions with fieldsets. We could not find this behavior for any of the question types where we figured it would make the most sense--groups of checkboxes or radio buttons.
• SurveyMonkeyupdated their FAQ dealing with accessibility in June 2008. The software was "certified" 508 compliant by RampWeb. We were contacted by a representative from SurveyMonkey via email who indicated their 508 certification. We have retested the product and do not see any changes to either the survey creation interfaces or the use of background images in radio and checkbox questions, from when we tested in April. As we note in our review, the latter cause the surveys to be impossible to fill when Windows is in high-contrast mode. They state on their accessibility FAQ that two aspects of 508 compliance are “Keyboard access for mobility impaired users” “Color contrast for users with low vision”. However, with high-contrast mode enabled, it is not possible to determine what checkbox or radio button has been selected.
• Their have been no changes to Zoomerang since we tested in April.
• Checkbox has come out in a new version (4.5) that promises more customization and styling of questions using a graphical editor. We have not re-evaluated Checkbox. Our review is based on version 4.4.
• LimeSurvey is at version 1.71, the version we tested in April, though they say they are working on version 2.0. We will try to review that when it comes out. Note that the LimeSurvey Project Leader wrote us and said that we had some of our information “just plain wrong“ regarding LimeSurvey accessibility. We responded but have not heard back.
• Added material on Snap Survey Professional Edition

Other Tools We Looked At

We examined a number of other survey tools. As was discussed above, we feel that our final picks have the most relevance to OSU because they are either currently used or have a degree of usability that merits mention. But the sea is full of fish. We also considered:

• EZQuestionnaire (markets itself as the survey tool for non-technical people and provides help setting up and running surveys and a questionnaire library of pre-fab survey types, for a fee, offers a free package)
• FeedbackFarm (hosted, completely free (no pay-for packages), limited question types and analysis tools)
• QuestionPro (hosted, geared toward enterprise and advertisers, sophisticated analysis packages, limited free accounts for non-profit and students, all typical packages are pay-for, 30-day demo available)
• KeySurvey (hosted, geared toward enterprise, API allows for integration into call center, HR, and CRM systems, 30-day trial but no free packages)