The post evaluates and compares the performance of forecasting sites like PredictIt, FiveThirtyEight, and Manifold Markets for the November 2022 midterm elections.
Data collection was done manually by checking the sites before the election date, and the analysis includes calibration plots and Brier scores to assess forecast accuracy.
Results show the importance of not solely relying on calibration plots, as Brier scores may provide a more accurate measure of forecast performance.
There are 2 types of forecast accuracy: Correct Prediction Accuracy and Uncertainty Judgment Accuracy.
Brier scores are a common metric to evaluate forecasts, balancing Correct Prediction Accuracy and Uncertainty Judgment Accuracy.
Brier scores can show how a partially-psychic forecaster with high correct predictions may still have a worse score than a forecaster with good judgment.