How to check significant differences in Claims Tests

In Claims Test experiments, each of the claims receives an individual score which is used to rank the claims. Although the ranking depends on the score, a claim that is higher ranked than the others isn’t necessarily higher performing since there is still noise in the data. Therefore, we compute if there are statistically significant differences between the scores and diagnostics. There are two ways to check for significant differences:

Highlight significantly different values for a specific claim

1. Click on the value of the claim you want to compare (it can be the score or any diagnostic).

   

   
   

2. On the left corner of your screen a blue notification box will appear:

   

   
   

3. Click on highlight button to mark all the values that are significantly different.

   

   
   

4. The report will show the following message as well, to let you know that the highlighted values are significantly different at the confidence level used:

   

   
   

5. Undo the highlighting by clicking on Undo highlighting button.

Check for significant difference between two specific values

1. Click on the value of the claim you want to compare (it can be the score or any diagnostic).

   

   
   

2. On the left corner of your screen a blue notification box will appear:

   

   
   

3. Click on a second value to check for significant differences:

   

   
   

4. On the left corner of your screen a blue notification box will appear with one of the following messages depending on if values are significantly different:

   

   

   
   

5. Undo the highlighting by clicking on Undo highlighting button.

The same procedure may be used for the Diagnostic scores:

You can mark all the scores that are significantly different from one diagnostic score: Multi-highlighting to show all differences from one value:

You can also check the significant difference between two diagnostic scores: