NFP: Superior Derivatives-Based Forecasts - Confirmed

Refet S. Gürkaynak and Justin Wolfers compared the Consensus of Economists or survey-based forecasts with the economic derivatives or market-based forecast using data from Oct. 2002 to Jul. 2005 (33 NFP observations). The results shown are shown below (in the GW columns).

I have replicated their study using more, and overlapping, data from Oct. 2002 to Mar. 2007 (64 observations). My results are shown in the table by the JCP columns.

The conclusion? Again the economic derivatives or auction market-based forecast dominates the Economist survey or Consensus forecast.

Details are found in the table below which looks at measures of forecast accuracy, the mean absolute error (MAE) and the root mean squared error (RMSE). There is also a correlation of each forecast with the actual (NFP release) and a regression-based test of the information content of each forecast using the Fair and Shiller method.

As with the smaller sample in GW, the MAE and RMSE are lower for the economic derivatives forecast. The correlation with the actuals is also higher than the Consensus-based forecast.

The coefficient in the regression should be unity for a good forecast. For the Derivatives or auction market-based forecast the test of the coefficient being equal 1 could not be rejected by GW. The evidence is not as strong now, as the test statistic is: F(1, 51) = 0.235477, with p-value = 0.62957.

The test that the Consensus or survey-based forecast is zero (that is that this forecast adds nothing to explanatory power of the other forecast, or conditioning on the market-based forecast renders the survey forecast uninformative) is: Test statistic: F(1, 51) = 5.62732, with p-value = 0.0214933. So the Consensus adds no information beyond the economic derivative forecast.

Not only that but the perverse negative coefficient found by GW persists with the longer data set.

Again, it seems “likely that the improved performance is due to the market effectively weighting a greater number of opinions, or more effective information aggregation as market participants are likely more careful when putting their money where their mouth is.”

	JCP	JCP	GW	GW
	Consensus	Economic Derivatives	Consensus	Economic Derivatives
Mean Absolute Error (MAE)	0.812	0.809	0.743	0.723
Root Mean Squared Error (RMSE)	1.036	1.023	0.929	0.907
Correlation of Forecast with Actual	0.7025	0.7026	0.677	0.700
Horse Race Regression (Fair-Shiller)	-0.42	1.26	-0.14	1.06
standard error	0.56	0.53	0.89	0.78
t-statistics	-0.75	2.38	-0.16	1.36
significant at 10% (*), 5% (**), or 1% (***) level		**
R2		0.50		0.46
obs.		54		33
range of data	Oct. 2002 - Mar. 2007		Oct. 2002 - Jul. 2005

Forecast errors normalized by historical (Oct. 2002 to Mar. 2007) standard deviation of survey-based forecasts of 90.31.

Fair-Shiller - Fair, Ray C. and Robert J. Shiller (1990), “Comparing Information in Forecasts from Econometric Models,” American Economic Review, 80(3), 375-89.

GW - Refet S. Gürkaynak and Justin Wolfers (2005), "Macroeconomic Derivatives: An Initial Analysis of Market-Based Macro Forecasts, Uncertainty, and Risk"