How do I interpret preference scores in claims tests?
Preference scores used in claims tests are essentially the same as partworth utilities in conjoint or MaxDiff studies. They measure how much customers liked each claim or product variant. Items that are strongly preferred by customers are assigned higher scores, items that perform poorly (in comparison to other items) are assigned lower scores. In this article, we deep-dive into how preference scores work.
Identifying a significant difference between claims
The “Summary of preferences and diagnostics for each claim” module displays your claim’s score and ranking, and it’s here that you can determine if there is a significant difference between any two of your claims. We use 90% bootstrap confidence intervals across all significance checks on the system.
To check the significance of the difference between each of your claims:
- Select your experiment under the “My experiments” option from the left-hand menu.
- Click the “View report” button on the bottom right-hand side of your screen.
- Select two values under the row labelled “Score”.
- Refer to the text displayed in the blue box which appears in your left-hand menu.
You can also view the confidence interval of each value by hovering your cursor over it.
Changes in claims’ rankings in your survey report
Even when there is not a significant difference in the average score of your claims in Market Overview, you may still find major differences of claim rankings across separate segments of respondents.
In the example above, there is not a significant difference between Claims 3 and 5, but there may be a larger variance of claim rankings within the individual segments.
Comparing a claim across different segments
In a claims tests, scores should be compared within each segment (becasue each score is relative to the other scores within each segment). That said, you can try to compare scores across segments at a high level, as the difference in sample size alters the confidence interval across segments.
This question from our users was answered on 30 January 2020. If there is anything else you'd like to know, please do not hesitate to contact us.