Avoiding Common Mistakes in Conjoint Studies

Posted on 24 November 2020 Kirill Zaitsev & Catherine Chipeta

If you are setting a conjoint study, read this guide to avoid making common mistakes which can negatively impact the quality of your results.

Conjoint analysis is an ideal way to test product features and pricing as it closely mirrors the way consumers make purchasing decisions. It is quick to set up using the Conjoint.ly platform’s state-of-the-art methodology which performs smart analytics to create insightful reporting.

To make the most out of the tool, users should take great care to ensure their inputs are accurate and suitable for the study. Below, we outline some of the most common mistakes users make when setting up a conjoint study so you can avoid them.

1. Testing too many features in a single conjoint experiment

Typically, consumers only focus on the most important features when making purchases, so it is not necessary to include a large number of features.

Points to consider:

  • The more features you add, the harder it is for respondents to compare each option in your conjoint study.

  • Your experiment’s complexity and sample size will increase with every feature you add, making your experiment more expensive without necessarily improving the reliability of your findings.

  • Be careful to give attributes and levels recognisable names, and avoid industry-specific lingo, which can confuse consumers.

2. Exceeding the limits of what a single conjoint can handle

Similar to the mistake of adding too many features, including too many other inputs will also make the survey too complex, increasing the sample size and cost.

Points to consider:

  • Include up to 7 attributes with enough levels to be reflective of the market. You can also include additional questions, but keep these to a minimum to avoid respondent fatigue.

  • Price points should range from ~80% to 120% of the realistic price. If you are unsure of the realistic price, conduct a pricing study to narrow it down.

  • Ask a moderate number of questions (between 10-14). Having more than 14 questions can cause respondent fatigue.

3. Not distinguishing between features clearly enough

Respondents can often get confused about features when one feature is a combination of several parameters. For example, if a promo offer is included in the total price, respondents may get confused with the additional promotional price parameter.

If survey results show $10 to be the optimal price but with an included promo price of 10% off, you would need to consider whether or not to include it, as it was not visible to all respondents.

This confusion can also arise among products with technical variations. For example, laundry detergent is available in several varieties (e.g. powder, liquid, concentrate). To clearly differentiate between each substance and create a fair comparison, respondents would need the specific characteristics of each explained to them.

Points to consider:

  • Product features should be clearly distinct to avoid confusing respondents.

  • Similar features can create bias, reducing the quality of the results.

  • Additional information for each feature type can clear any potential confusion.

  • Using visuals to represent products makes it easier for respondents to understand.

4. Relying exclusively on average utility scores

Conjoint studies provide rich datasets of individual respondents’ preferences for the levels tested. Looking exclusively at high-level average scores prevents you from taking full advantage of this data.

Points to consider:

  • Use simulations in addition to average utility scores for more accurate analysis and conclusions.

  • Analysis should be done through thoughtful sets of simulations.

5. Preference share unadjusted for market share

Estimated preference share should not be used in isolation as it is almost impossible to include every competitors’ market offering, and factors such as shelf availability of products can pose an issue.

Practical tips for setting up conjoint studies for pricing

Sample definition

  • Aim for a representative sample of category buyers by household income and brand usage (gender and age are less relevant, but can be useful for balancing the sample).

  • Sample more than just buyers of specific brands to avoid underestimating price elasticity.

Study set-up

  • Ensure that images are sized proportionally to pack size.

  • Check consistency of image quality, especially on retina display (use Conjoint.ly image optimisation tool).

  • Do not place more than 5 options per screen.

  • Generally, virtual shelf display is not essential.*

*We have not seen any evidence in academic literature or in practice that virtual shelf display predicts market outcomes better than simple conjoint, however, it is more costly.

Working with outputs

  • It is important to play with the simulator to get a feel for the data.

  • Market share adjustments can be performed easily with a limited number of SKUs.

  • Analysis of promo mechanics is also possible, but requires elaborate work to simulate various scenarios across different weeks.

See also: How to Specify Attributes & Levels