Charts for Survey vs. Derivatives Based Forecast Comparisons

In the last few posts I showed the statistics that show that the economic derivatives forecast outperforms the consensus for the nonfarm payrolls announcement. Here are the charts that go with the stats. First, here is the chart of Actual (U.S. nonfarm payrolls or NFP) versus the economic derivatives or market-based forecast for the last 54 months (2002:01 - 2007:03). Here is the same chart for the Consensus or survey-based forecast: And side-by-side:And here are all three as a time series plot: Here are the forecast errors for the economic derivative or auction market forecasts compared with a fitted normal distribution (you'd hope that the errors were random and reasonably normal in their distribution): And here are the Consensus or survey-based forecast errors: Conclusion? You can't see much from the charts as the two forecast series both track the actual quite well and eye-balling the charts does not suggest one is better than another. However, the statistics of MAE, RMSE, correlation and especially the horse-race regression confirm that the economic derivative or auction market-based forecast outperforms the survey or Consensus forecast and that the latter adds nothing once you have the former.

NFP: Superior Derivatives-Based Forecasts - Confirmed

Refet S. Gürkaynak and Justin Wolfers compared the Consensus of Economists or survey-based forecasts with the economic derivatives or market-based forecast using data from Oct. 2002 to Jul. 2005 (33 NFP observations). The results shown are shown below (in the GW columns).

I have replicated their study using more, and overlapping, data from Oct. 2002 to Mar. 2007 (64 observations). My results are shown in the table by the JCP columns.

The conclusion? Again the economic derivatives or auction market-based forecast dominates the Economist survey or Consensus forecast.

Details are found in the table below which looks at measures of forecast accuracy, the mean absolute error (MAE) and the root mean squared error (RMSE). There is also a correlation of each forecast with the actual (NFP release) and a regression-based test of the information content of each forecast using the Fair and Shiller method.

As with the smaller sample in GW, the MAE and RMSE are lower for the economic derivatives forecast. The correlation with the actuals is also higher than the Consensus-based forecast.

The coefficient in the regression should be unity for a good forecast. For the Derivatives or auction market-based forecast the test of the coefficient being equal 1 could not be rejected by GW. The evidence is not as strong now, as the test statistic is: F(1, 51) = 0.235477, with p-value = 0.62957.

The test that the Consensus or survey-based forecast is zero (that is that this forecast adds nothing to explanatory power of the other forecast, or conditioning on the market-based forecast renders the survey forecast uninformative) is: Test statistic: F(1, 51) = 5.62732, with p-value = 0.0214933. So the Consensus adds no information beyond the economic derivative forecast.

Not only that but the perverse negative coefficient found by GW persists with the longer data set.

Again, it seems “likely that the improved performance is due to the market effectively weighting a greater number of opinions, or more effective information aggregation as market participants are likely more careful when putting their money where their mouth is.”

	JCP	JCP	GW	GW
	Consensus	Economic Derivatives	Consensus	Economic Derivatives
Mean Absolute Error (MAE)	0.812	0.809	0.743	0.723
Root Mean Squared Error (RMSE)	1.036	1.023	0.929	0.907
Correlation of Forecast with Actual	0.7025	0.7026	0.677	0.700
Horse Race Regression (Fair-Shiller)	-0.42	1.26	-0.14	1.06
standard error	0.56	0.53	0.89	0.78
t-statistics	-0.75	2.38	-0.16	1.36
significant at 10% (*), 5% (**), or 1% (***) level		**
R2		0.50		0.46
obs.		54		33
range of data	Oct. 2002 - Mar. 2007		Oct. 2002 - Jul. 2005

Forecast errors normalized by historical (Oct. 2002 to Mar. 2007) standard deviation of survey-based forecasts of 90.31.

Fair-Shiller - Fair, Ray C. and Robert J. Shiller (1990), “Comparing Information in Forecasts from Econometric Models,” American Economic Review, 80(3), 375-89.

GW - Refet S. Gürkaynak and Justin Wolfers (2005), "Macroeconomic Derivatives: An Initial Analysis of Market-Based Macro Forecasts, Uncertainty, and Risk"

Nonfarm payroll employment - whose better?

Nonfarm payrolls for February 2007 (released today, Friday March 9th at 8:30am) continued to trend up (+97,000). Details are here. Yesterday I noted that as of yesterday, the Economists' consensus was 100,000 and the CME Auction Market participants' consensus was 82,500. So who is the better forecaster? In this instance the Economists were better (error of 3,000 versus the derivatives auction results error of 14,500). For the Nonfarm payrolls (NFP) there are several forecasts that come from the CME auctions There was one more auction and therefore one more forecast before the release at 8:30am this morning. This auction gave an implied market forecast that was even more off the mark (75,500). I have maintained that, based on my research, over time the CME Auction Market participants' consensus outperforms the Economists' consensus. Gürkaynak and Wolfers (2006) conclude that “The evidence presented … shows that economic derivatives option prices are accurate and efficient predictors of the densities of underlying events” (p. 29). Seems it is time to test that hypothesis again. Here's what I'd like to do. Test:

• Which forecast best predicts the actual outcome, CME economic derivatives auctions or the Economists' consensus?
• Since, for the NFP there are several forecasts from economic derivatives auctions, do auctions closer to the release perform better than forecasts that are more stale?
• Does averaging the auction results produce a superior forecast?

There are a couple of definitions of a better forecast that may be pertinent here. One might be, as implied above, that is which forecast predicts the outcome better. However this assumes that the variable you are interested in is in fact the economic release (such as the NFP). But who really cares about the NFP? What most people care about is what it means to them. These economic statistics are indicators. Most investors care about how these indicators affect their portfolio of holdings. So one definition of a better forecast of NFP might be one that more accurately explains (or forecasts) movements in the financial variable of interest. I'm not sure the data will allow for a definitive test on all of these points, but this is my goal.

1. Gürkaynak, Refet S., Wolfers, Justin, (2006) “Macroeconomic Derivatives: An Initial Analysis of Market-Based Macro Forecasts, Uncertainty and Risk” NBER Working Paper Series, NBER Working Paper 11929, January 2006.

Consensus Forecasts and Herd Mentality

This is the third and final note about a post by the FX guru of Nova Scotia, Tom Yeomans, entitled “Is news trading dead?”

Tom talks about how FX trading around economic announcements has changed over the last few years. He then outlines a view of the future of FX news trading that involves forecasting the announcement.

Since it is news (that is difference between the actual release and the market’s expectation just before the release) that moves the market you need not just a forecast but also a read on what the market thinks.

Tom discusses the use of consensus forecasts. He notes that “Usually they had 18-21 people making guesses. That always seemed a little suspect to me …”.

I have noted in another post about the problems with consensus forecasts.

However consensus forecasts can be made useful. In a paper on how forecasts can be used by financial institutions for risk management purposes I noted that surveys of forecasts can be used to develop scenarios for risk management and how …

“This allows risk managers to understand their potential losses conditional on a range of forecasts. The average forecast and the dispersion in forecasts can be used to build a model of the distribution of market participants' expectations. The fitted model can then be used to extrapolate to large moves and thereby address the problems of sparse and clustered data. Finally, conditional scenarios can be used to mitigate the lack of coverage in forecast surveys.”

Note that scenarios, based on forecasts, can also be used for speculating as well as risk management since the focus is on the entire set of possible outcomes and so scenarios paint a picture of the return and the associated risk.

Of course, for certain announcements getting a read on the market is easy since a full distribution of expectations can be had from auctions of economic derivatives.