Ensuring quality responses with multi-layered protection

Conjointly uses the Swiss Cheese model for response quality management, with multiple defensive layers working together to prevent unreliable data from reaching your final results.

The Swiss Cheese model of quality management

The Swiss Cheese model, developed by Professor James Reason, likens organisational systems to multiple slices of Swiss cheese stacked together. Each slice represents a protective barrier, and the holes represent potential weak points in the layer. Because the holes in different slices don't align perfectly, even if one barrier fails, the other layers prevent problems from getting through.

Rather than relying on one single control for every issue, Conjointly adopts the Swiss Cheese model to safeguard your response quality. Each layer catches different issues that others might miss, and you can combine multiple layers to create a robust defense system that minimises the chances that poor responses slip through.

Below is an illustrative example of layered protection with various response quality management options:

When you recruit respondents through Conjointly, all response management options are enabled by default.

When you invite your own respondents, you can configure the response management options within the Response quality management and Response quality warnings tabs under Advanced Settings .

Response quality management options

Prevention of repeat completions

• Disable respondents from the same IP address
  Prevent duplicated or fraudulent responses from the same IP address. Note that this setting should not be used when surveying employees of the same company.
• Disable respondents from the same cookie
  Prevent duplicated or fraudulent responses by assigning unique “cookies” to each respondent.
• Required unique GET variables
  Specify a unique GET variable to allow only one unique entry. The survey cannot be accessed without this GET variable.

Filtering for low-quality responses

• Terminate low-quality respondents early
  Prevent respondents marked as low-quality by the system from continuing with the survey.
• Detect duplicate text responses
  Detect and filter duplicated text responses from different participants. This setting applies to all open-ended responses of more than 50 characters.
• Extra hard exclusion rules
  Enable this option to filter out suspicious responses from online panels. The set of extra hard rules is continuously updated and purposely kept undisclosed to maintain their effectiveness against evolving fraudulent behaviors. Note that this option is not compatible with anonymise responses and cannot be used simultaneously.

Set accepted device

Specify the device type respondents can use to answer the survey by choosing an option from the dropdown menu next to Accepted device. By default, any mobile device and desktop are accepted.

Geofencing

Specify the geographic locations where respondents are allowed to participate in your survey.

• Ask respondents to share browser location
  Browser location can be more precise than location determined from respondents’ IP addresses. However, it requires additional permissions from respondents that they may refuse to provide. This setting is automatically selected when you select the Only allow respondents from within certain geographic area option.
• Check time zone based on location
  Automatically detect respondents’ time zones to crosscheck the veracity of their locations.
• Allowed locations of respondents
  This option allows you to limit your survey to respondents from specific countries based on their location determined from IP addresses. You can use the field as a search bar to find and select your desired countries. This setting is automatically activated for paid sample sources. Please note that the check will only apply after the experiment is launched.

  

• Only allow respondents from within certain geographic area
  This option provides an alternative approach to geofencing your survey based on respondents’ device location. It allows you to define a rectangular area on the map where only respondents within this region can participate in the survey. This check will only apply after the experiment is launched.

Detection of sloppy open-ended responses

Easily identify and exclude low-quality open-ended responses using a large language model (LLM).

Pricing:

• This option is available for free when you source respondents through Conjointly’s Predefined panels and Self-serve sample.
• When you bring your own respondents or send an email campaign, you have the option to enable this quality check. A fee of USD 0.10 per complete response with an open-end will be charged from your team balance. In order to activate this option, a minimum team balance of USD 100 or its equivalent is required.

When this option is enabled, the system will analyse and categorise the open-ended responses in your experiment. To review the categorisation, navigate to the Quality Checker page by clicking the Quality Checker button on the experiment settings page.

At the bottom right of the Quality Checker page, you will find a table that lists all risk signals recorded for each respondent. To identify the categories of the open-ended responses, find the on-review label in the Occasion column. The corresponding categories will be listed in the Signal column next to this label, as shown in the table below.

To exclude responses, use one of the following options located in the top-right corner of the Quality Checker page:

• 👎 icon: Click this icon below Assessment of quality to exclude low-quality responses.
• Exclude button: Click this button to exclude responses due to non-quality issues, such as quota fulfilment or ineligibility.

Detection of anomalous network activity

This option allows you to track respondents’ network activities in the background. Respondents with suspicious network activities such as the usage of VPNs, browser spoofing, and other network tricks are still allowed to complete the survey but are automatically marked and excluded from analysis, with “Anomalous network activity detected” displayed in the status column of your Review Participant dashboard and data exports.

Pricing:

• This quality check is complimentary and automatically enabled when you source respondents through Conjointly.
• When you bring your own respondents or send an email campaign, you have the option to enable this quality check. A fee of USD 0.01 will be charged from your balance for each survey entry attempt (not per completion). In order to activate this option, a minimum team balance of USD 100 or its equivalent is required.

Warnings to respondents

• Warn respondents who do not scroll enough
  Prompt respondents to attentively consider the questions and choices.
• Warn at-risk respondents once
  Warn low-quality respondents once before termination.
• Force respondents to read conjoint questions
  Force respondents to stay on each choice set for a specified duration before they can make a choice.

Redirect for low-quality responses

You can also specify a separate redirect for respondents who have given low-quality responses:

• Open the Redirects upon survey completion module under the Advanced settings tab.
• Insert the redirect URL in the URL for low quality responses field.
• Click the Save changes button at the bottom of your screen.