Charts for Survey vs. Derivatives Based Forecast Comparisons

In the last few posts I showed the statistics that show that the economic derivatives forecast outperforms the consensus for the nonfarm payrolls announcement. Here are the charts that go with the stats. First, here is the chart of Actual (U.S. nonfarm payrolls or NFP) versus the economic derivatives or market-based forecast for the last 54 months (2002:01 - 2007:03). Here is the same chart for the Consensus or survey-based forecast: And side-by-side:And here are all three as a time series plot: Here are the forecast errors for the economic derivative or auction market forecasts compared with a fitted normal distribution (you'd hope that the errors were random and reasonably normal in their distribution): And here are the Consensus or survey-based forecast errors: Conclusion? You can't see much from the charts as the two forecast series both track the actual quite well and eye-balling the charts does not suggest one is better than another. However, the statistics of MAE, RMSE, correlation and especially the horse-race regression confirm that the economic derivative or auction market-based forecast outperforms the survey or Consensus forecast and that the latter adds nothing once you have the former.

NFP: Superior Derivatives-Based Forecasts - Confirmed

Refet S. Gürkaynak and Justin Wolfers compared the Consensus of Economists or survey-based forecasts with the economic derivatives or market-based forecast using data from Oct. 2002 to Jul. 2005 (33 NFP observations). The results shown are shown below (in the GW columns).

I have replicated their study using more, and overlapping, data from Oct. 2002 to Mar. 2007 (64 observations). My results are shown in the table by the JCP columns.

The conclusion? Again the economic derivatives or auction market-based forecast dominates the Economist survey or Consensus forecast.

Details are found in the table below which looks at measures of forecast accuracy, the mean absolute error (MAE) and the root mean squared error (RMSE). There is also a correlation of each forecast with the actual (NFP release) and a regression-based test of the information content of each forecast using the Fair and Shiller method.

As with the smaller sample in GW, the MAE and RMSE are lower for the economic derivatives forecast. The correlation with the actuals is also higher than the Consensus-based forecast.

The coefficient in the regression should be unity for a good forecast. For the Derivatives or auction market-based forecast the test of the coefficient being equal 1 could not be rejected by GW. The evidence is not as strong now, as the test statistic is: F(1, 51) = 0.235477, with p-value = 0.62957.

The test that the Consensus or survey-based forecast is zero (that is that this forecast adds nothing to explanatory power of the other forecast, or conditioning on the market-based forecast renders the survey forecast uninformative) is: Test statistic: F(1, 51) = 5.62732, with p-value = 0.0214933. So the Consensus adds no information beyond the economic derivative forecast.

Not only that but the perverse negative coefficient found by GW persists with the longer data set.

Again, it seems “likely that the improved performance is due to the market effectively weighting a greater number of opinions, or more effective information aggregation as market participants are likely more careful when putting their money where their mouth is.”

	JCP	JCP	GW	GW
	Consensus	Economic Derivatives	Consensus	Economic Derivatives
Mean Absolute Error (MAE)	0.812	0.809	0.743	0.723
Root Mean Squared Error (RMSE)	1.036	1.023	0.929	0.907
Correlation of Forecast with Actual	0.7025	0.7026	0.677	0.700
Horse Race Regression (Fair-Shiller)	-0.42	1.26	-0.14	1.06
standard error	0.56	0.53	0.89	0.78
t-statistics	-0.75	2.38	-0.16	1.36
significant at 10% (*), 5% (**), or 1% (***) level		**
R2		0.50		0.46
obs.		54		33
range of data	Oct. 2002 - Mar. 2007		Oct. 2002 - Jul. 2005

Forecast errors normalized by historical (Oct. 2002 to Mar. 2007) standard deviation of survey-based forecasts of 90.31.

Fair-Shiller - Fair, Ray C. and Robert J. Shiller (1990), “Comparing Information in Forecasts from Econometric Models,” American Economic Review, 80(3), 375-89.

GW - Refet S. Gürkaynak and Justin Wolfers (2005), "Macroeconomic Derivatives: An Initial Analysis of Market-Based Macro Forecasts, Uncertainty, and Risk"

A Consensus of Economists

How well do economists predict the U.S. nonfarm payrolls? I picked up the consensus forecast for the NFP from the NASDAQ website (from Econoday). The latest report is here.

There are 63 monthly reports back to January 2002 (there may be more but this is all I pulled for analysis), all have actual and the consensus forecast. The most recent 4o reports have the range of the forecasts as well.

The average error was 24,000.

This appears to be a biased forecast (on average the error should be zero. With a standard error or 10,000, this is significantly different from zero at the 95% level). So despite studying the dismal science, economists are overall an optimistic bunch. In fact they are overly optimistic.

The biggest oopsie was 328,000 (in March 2003 when the actual was -308,000 and the expected was 20,000).

In the last 40 months the actual fell within the range of the expectations 20 times. So 50% of the time the actual was inside the range. That is not a great record.

Next, I'll look at the economic derivatives forecast performance and provide some comparisons.

Nonfarm payroll employment - whose better?

Nonfarm payrolls for February 2007 (released today, Friday March 9th at 8:30am) continued to trend up (+97,000). Details are here. Yesterday I noted that as of yesterday, the Economists' consensus was 100,000 and the CME Auction Market participants' consensus was 82,500. So who is the better forecaster? In this instance the Economists were better (error of 3,000 versus the derivatives auction results error of 14,500). For the Nonfarm payrolls (NFP) there are several forecasts that come from the CME auctions There was one more auction and therefore one more forecast before the release at 8:30am this morning. This auction gave an implied market forecast that was even more off the mark (75,500). I have maintained that, based on my research, over time the CME Auction Market participants' consensus outperforms the Economists' consensus. Gürkaynak and Wolfers (2006) conclude that “The evidence presented … shows that economic derivatives option prices are accurate and efficient predictors of the densities of underlying events” (p. 29). Seems it is time to test that hypothesis again. Here's what I'd like to do. Test:

• Which forecast best predicts the actual outcome, CME economic derivatives auctions or the Economists' consensus?
• Since, for the NFP there are several forecasts from economic derivatives auctions, do auctions closer to the release perform better than forecasts that are more stale?
• Does averaging the auction results produce a superior forecast?

There are a couple of definitions of a better forecast that may be pertinent here. One might be, as implied above, that is which forecast predicts the outcome better. However this assumes that the variable you are interested in is in fact the economic release (such as the NFP). But who really cares about the NFP? What most people care about is what it means to them. These economic statistics are indicators. Most investors care about how these indicators affect their portfolio of holdings. So one definition of a better forecast of NFP might be one that more accurately explains (or forecasts) movements in the financial variable of interest. I'm not sure the data will allow for a definitive test on all of these points, but this is my goal.

1. Gürkaynak, Refet S., Wolfers, Justin, (2006) “Macroeconomic Derivatives: An Initial Analysis of Market-Based Macro Forecasts, Uncertainty and Risk” NBER Working Paper Series, NBER Working Paper 11929, January 2006.

What Can You Expect from the March Nonfarm Payroll Release?

As I have discussed before in this blog, the best information about an upcoming nonfarm payrolls release comes from the economic derivatives auctions held by the Chicago Mercantile Exchange, Longitude, and ICAP. From the CME: March Nonfarm Payroll Release - Friday, March 9 at 8:30 a.m. EST

Economists' consensus - 100,000

CME Auction Market participants' consensus - 82,500

Based on previous auctions that occurred Tuesday, March 6, Wednesday, March 7 and earlier this morning, CME Auction Market participants predict U.S. employers added 82,500 jobs last month, which is significantly lower than economists' forecasts of 100,000 jobs.

Chart from this morning's CME Auction (March 8, 7:30 a.m. - 8:15 a.m. EST)

Chart from yesterday's CME Auction (March 7, 7:30 a.m. - 8:15 a.m. EST)

Chart from Tuesday's CME Auction (March 6, 7:30 a.m. - 8:15 a.m. EST)

Consensus Forecasts and Herd Mentality

This is the third and final note about a post by the FX guru of Nova Scotia, Tom Yeomans, entitled “Is news trading dead?”

Tom talks about how FX trading around economic announcements has changed over the last few years. He then outlines a view of the future of FX news trading that involves forecasting the announcement.

Since it is news (that is difference between the actual release and the market’s expectation just before the release) that moves the market you need not just a forecast but also a read on what the market thinks.

Tom discusses the use of consensus forecasts. He notes that “Usually they had 18-21 people making guesses. That always seemed a little suspect to me …”.

I have noted in another post about the problems with consensus forecasts.

However consensus forecasts can be made useful. In a paper on how forecasts can be used by financial institutions for risk management purposes I noted that surveys of forecasts can be used to develop scenarios for risk management and how …

“This allows risk managers to understand their potential losses conditional on a range of forecasts. The average forecast and the dispersion in forecasts can be used to build a model of the distribution of market participants' expectations. The fitted model can then be used to extrapolate to large moves and thereby address the problems of sparse and clustered data. Finally, conditional scenarios can be used to mitigate the lack of coverage in forecast surveys.”

Note that scenarios, based on forecasts, can also be used for speculating as well as risk management since the focus is on the entire set of possible outcomes and so scenarios paint a picture of the return and the associated risk.

Of course, for certain announcements getting a read on the market is easy since a full distribution of expectations can be had from auctions of economic derivatives.

Market Expectations

It is not news that news moves financial markets. This blog will publish research on how, when, why, and which news moves what financial markets.

The best news depends on two factors:

1. The best measurement of the actual market move; and,
2. The best measurement of the market's expectation.

A market expectation based on information from derivative auctions outperforms survey measures.

Apart from proving more information the Economic Derivatives or market-based forecasts are found by Gürkaynak and Wolfers to outperform the survey data. They:

“… establish that the Economic Derivatives forecast dominates the survey forecast (although survey forecasts perform quite well) both in predicting outcomes and in predicting market responses to economic news.” (p. 13)

And,

· “… that central tendencies of market-based forecasts are very similar to, but more accurate than surveys. Further, financial market responses to data releases are also better captured by surprises measured with respect to market-based expectations than survey-based expectations, again suggesting that they better capture investor expectations. Some behavioral anomalies evident in survey-based expectations – such as forecastable forecast errors – are notably absent from market-based forecasts.” (p. 1)

The Federal Reserve Board of San Francisco (2006) (Wolfers) took data from the first 153 of these Economic Derivatives auctions and compared them with an alternative forecast aggregator: the survey of the expectations of financial market analysts taken on the Friday prior to the data release. They asked “which better predicts the actual data?” The Economic Derivatives forecasts were slightly (5%–10%) more accurate, although these differences were not statistically significant. They also found, more interestingly, once one knows the Economic Derivatives forecast, there is no useful information in the survey-based forecast.

They also analyze the change in stock and bond prices from 5 minutes before the announcement to 25 minutes later for the two alternative measures of news. In each case, they confirm that the Economic Derivatives market better predicts financial market responses to economic data than does the alternative survey-based measure.

Derivatives data was available for 7 series (now the CME only maintains 4) with the longest history going back monthly to September 2002. Survey data is available for some 170 series with the longest history going back weekly to 1980.

While the Economic Derivatives data is superior in terms of information content and usefulness for measuring the announcement effect, the survey data has a longer history and broader coverage. I use the economic derivatives data.