Understanding prediction market reports
Prediction markets on Conjoint.ly are designed for product and concept testing through aggregating the audience’s predictions about future success or failure of product launches, modifications, and other decisions. However, respondents need to be trained to use the interface of predictions in the first instance. This guide covers how to read the outputs of both calibration and prediction questions.
Summary of calibration
The calibration accuracy score is an overall guide to how reliable the predictions made by participants will be.
- A score of 65% and above suggests that the respondents were diligent, knowledgeable, and capable of using the survey to make good predictions for the future.
- Low accuracy of calibration (50% to 65%) should make you cautious of relying on the predictions.
- When responses to calibration questions were inaccurate on average (below 50% score), predictions should not be considered trustworthy.
The percentage of skipping of calibration questions (i.e., choosing the "I don't know" option) should not exceed 20%: otherwise you are dealing with lazy respondents.
Calibration questions
For each calibration question, we calculate the percentage of bets placed on correct responses. Scores over 65% are commonly sufficient to conclude respondents were "correct on average".
The betting summary table contains the number of answers (i.e., respondents) who made a particular choice, the average number of points they bet, and the total number of points bet.
| Number of answers | Average points bet | Total points bet | |
|---|---|---|---|
| Correct answer: Mumbai Indians | 87 | 68 | 5920 |
| Incorrect answer: Rising Pune Supergiants | 24 | 54 | 1285 |
| Skipped question | 5 | N/A | N/A |
| Total | 116 | 62 | 7205 |
The consensus history chart plots the percentage of correct bets against the timeline of responses. Convergence of scores suggests that the summary is firm and that a market consensus has emerged.
For example, in this chart the first several responses were correct. Response 3333185 was wrong and moved consensus to 93% correctness. Response 3370385, on the other hand, was correct and moved consensus from 83 to 84% correctness.
Predictions and commentary
Each prediction will be displayed together with the confidence in prediction (in addition to a betting summary table and a consensus history chart).
In interpreting predictions, we recommend conservatism: If the decision at stake is product launch, only a strong prediction (over 80%) in support of the launch would be enough to support the launch.
For each side of the argument, you will see respondents' rationale for making a particular prediction. Comments that were voted by other respondents as convincing will be displayed on top with a higher conviction score.
Comments in support of Yes, it will
| Comment | Conviction score |
|---|---|
| Becasue the Apple store app is very popular | 90% |
| it will be 10 mil. People watch IPL!! | 70% |
| i would download it myself | 60% |
| because most people think so | 30% |
| it think fans will follow IPL. it is a popular code | 30% |