Another look at measures of forecast accuracy

doi:10.1016/j.ijforecast.2006.03.001

International Journal of Forecasting

Volume 22, Issue 4, October–December 2006, Pages 679-688

https://doi.org/10.1016/j.ijforecast.2006.03.001 Get rights and content

Abstract

We discuss and compare measures of accuracy of univariate time series forecasts. The methods used in the M-competition as well as the M3-competition, and many of the measures recommended by previous authors on this topic, are found to be degenerate in commonly occurring situations. Instead, we propose that the mean absolute scaled error become the standard measure for comparing forecast accuracy across multiple time series.

Introduction

Many measures of forecast accuracy have been proposed in the past, and several authors have made recommendations about what should be used when comparing the accuracy of forecast methods applied to univariate time series data. It is our contention that many of these proposed measures of forecast accuracy are not generally applicable, can be infinite or undefined, and can produce misleading results. We provide our own recommendations of what should be used in empirical comparisons. In particular, we do not recommend the use of any of the measures that were used in the M-competition or the M3-competition.

To demonstrate the inadequacy of many measures of forecast accuracy, we provide three examples of real data in Fig. 1. These show series N0472 from the M3-competition,² monthly log stock returns for the Walt Disney Corporation, and monthly sales of a lubricant product sold in large containers. Note that the Disney return series and the lubricant sales series both include exact zero observations, and the Disney series contains negative values. Suppose that we are interested in comparing the forecast accuracy of four simple methods: (1) the historical mean using data up to the most recent observation; (2) the “naïve” or random-walk method based on the most recent observation; (3) simple exponential smoothing and (4) Holt's method. We do not suggest that these are the best methods for these data, but they are all simple methods that are widely applied. We compare the in-sample performance of the methods (based on one-step-ahead forecasts) and the out-of-sample performance (based on forecasting the data in the hold-out period using only information from the fitting period).

Table 1, Table 2, Table 3 show some forecast error measures for these methods applied to the example data. The acronyms are defined below and we explicitly define the measures in 2 A critical survey of accuracy measures, 3 Scaled errors. The relative measures are all computed relative to a naïve (random walk) method.

In these tables, we have included measures that have been previously recommended for use in comparing forecast accuracy across many series. Most textbooks recommend the use of the MAPE (e.g., Hanke & Reitsch, 1995, p.120, and Bowerman, O'Connell, & Koehler, 2004, p.18) and it was the primary measure in the M-competition (Makridakis et al., 1982). In contrast, Makridakis, Wheelwright, and Hyndman (1998, p. 45) warn against the use of the MAPE in some circumstances, including those encountered in these examples. Armstrong and Collopy (1992) recommended the use of GMRAE, MdRAE and MdAPE. Fildes (1992) also recommended the use of MdAPE and GMRAE (although he described the latter as the relative geometric root mean square error or GRMSE). The MdRAE, sMAPE and sMdAPE were used in the M3-competition (Makridakis & Hibon, 2000).

The M-competition and M3-competition also used rankings amongst competing methods. We do not include those here as they are dependent on the number of methods being considered. They also give no indication of the size of the forecast errors. Similarly, both competitions included measures based on the percentage of times one method was better than a benchmark method. Again, such measures are not included here as they do not indicate the size of the errors.

To our knowledge, the MASE has not been proposed before. We consider it the best available measure of forecast accuracy and we argue for it in Section 3.

Note that there are many infinite values occurring in Table 1, Table 2, Table 3 due to division by zero. Division by numbers close to zero also results in very large numbers. The undefined values arise due to the division of zero by zero. Some of these are due to computations of the form Y_t / (Y_t − Y_t−1) where Y_t−1 = Y_t = 0, and others are due to computations of the form (Y_t − Y_t−1) / (Y_t − Y_t−1) where Y_t = Y_t−1. In the latter case, it is algebraically possible to cancel the numerator and denominator, although numerical results will be undefined. Also note that the sMAPE can take negative values although it is meant to be an “absolute percentage error”.

Note that with random walk forecasts, the in-sample results for MASE and all results for MdRAE and GMRAE are 1 by definition, as they involve comparison with naïve forecasts. However, some of the values for MdRAE and GMRAE are undefined as explained above.

Of the measures in Table 1, Table 2, Table 3, only the MASE can be used for these series due to the occurrence of infinite and undefined values. These three series are not degenerate or unusual—intermittent demand data often contain zeros and many time series of interest to forecasters contain negative observations. The cause of the problems with M3 series N0472 is the occurrence of consecutive observations which take the same value, something that very often occurs with real data.

Section snippets

A critical survey of accuracy measures

Let Y_t denote the observation at time t and F_t denote the forecast of Y_t. Then define the forecast error e_t = Y_t − F_t. The forecasts may be computed from a common base time, and be of varying forecast horizons. Thus, we may compute out-of-sample forecasts F_n+1,…, F_n+m based on data from times t = 1,…, n. Alternatively, the forecasts may be from varying base times, and be of a consistent forecast horizon. That is, we may compute forecasts F_1+h,…, F_m+h where each F_j+h is based on data from times t = 1,…,

Scaled errors

Relative measures and measures based on relative errors both try to remove the scale of the data by comparing the forecasts with those obtained from some benchmark forecast method, usually the naïve method. However, they both have problems. Relative errors have a statistical distribution with undefined mean and infinite variance. Relative measures can only be computed when there are several forecasts on the same series, and so cannot be used to measure out-of-sample forecast accuracy at a

Application to M3-competition data

We demonstrate the use of MASE using the M3-competition data (Makridakis & Hibon, 2000). Fig. 2 shows the MASE at each forecast horizon for four forecasting methods applied to the M3-competition data. The errors have been scaled by the one-step in-sample forecast errors from the naïve method, and then averaged across all series. So a value of 2 indicates that the out-of-sample forecast errors are, on average, about twice as large as the in-sample one-step forecast errors from the naïve method.

Conclusion

Despite two decades of papers on measures of forecast error, we believe that some fundamental problems have been overlooked. In particular, the measures used in the M-competition and the M3-competition, and the measures recommended by other authors, all have problems—they can give infinite or undefined values in commonly occurring situations.

We propose that scaled errors become the standard measure for forecast accuracy, where the forecast error is scaled by the in-sample mean absolute error

Acknowledgments

We thank Michelle Hibon for kindly providing the forecasts submitted to the M3-competition, and two anonymous referees for providing some thoughtful comments.

Rob J. Hyndman is a Professor of Statistics in the Department of Econometrics and Business Statistics at Monash University, Australia. He has published extensively in leading statistical and forecasting journals, and is Editor-in-Chief of the International Journal of Forecasting. He is also co-author of the well-known text on business forecasting, Forecasting: methods and applications (Makridakis, Wheelwright & Hyndman, 3rd ed., 1998, Wiley).

References (25)

J.S. Armstrong et al.
Error measures for generalizing about forecasting methods: Empirical comparisons
International Journal of Forecasting
(1992)
V. Assimakopoulos et al.
The theta model: A decomposition approach to forecasting
International Journal of Forecasting
(2000)
B. Billah et al.
Exponential smoothing model selection for forecasting
International Journal of Forecasting
(2006)
C. Chatfield
Apples, oranges and mean square error
International Journal of Forecasting
(1988)
R. Fildes
The evaluation of extrapolative forecasting methods
International Journal of Forecasting
(1992)
P. Goodwin et al.
On the asymmetry of the symmetric MAPE
International Journal of Forecasting
(1999)
R.J. Hyndman et al.
Unmasking the Theta Method
International Journal of Forecasting
(2003)
R.J. Hyndman et al.
A state space framework for automatic forecasting using exponential smoothing methods
International Journal of Forecasting
(2002)
M. Lawrence et al.
Judgmental forecasting in the presence of loss functions
International Journal of Forecasting
(2005)
S. Makridakis
Accuracy measures: Theoretical and practical concerns
International Journal of Forecasting
(1993)

S. Makridakis et al.

The M3-competition: Results, conclusions and implications

International Journal of Forecasting

(2000)

P.A. Thompson

An MSE statistic for comparing forecast accuracy across series

International Journal of Forecasting

(1990)

Cited by (3551)

Forecasting seasonally fluctuating sales of perishable products in the horticultural industry
2024, Expert Systems with Applications
Accurately forecasting demand is a potential competitive advantage, especially when dealing with perishable products. The multi-billion dollar horticultural industry is highly affected by perishability, but has received limited attention in forecasting research. In this paper, we analyze the applicability of general compared to dataset-specific predictors, as well as the influence of external information and online model update schemes. We employ a heterogeneous set of horticultural data, three classical, and twelve machine learning-based forecasting approaches. Our results show a superiority of multivariate machine learning methods, in particular the ensemble learner XGBoost. These advantages highlight the importance of external factors, with the feature set containing statistical, calendrical, and weather-related features leading to the most robust performance. We further observe that a general model is unable to capture the heterogeneity of the data and is outperformed by dataset-specific predictors. Moreover, frequent model updates have a negligible impact on forecasting quality, allowing long-term forecasting without significant performance degradation.
Good practices for surplus production models
2024, Fisheries Research
Surplus production models (SPMs) have a long history in fisheries ecology and are important for the assessment and management of marine species and stocks. However, the implementation, application, and usage of these models vary across regions and case studies. Good practice guidelines can streamline modelling workflows, inform the acceptance or rejection of an assessment, and facilitate the derivation of management advice from accepted assessments. This paper discusses current practices in the application of SPMs and proposes good practice guidelines for their use in stock assessment. We complement our recommendations with results from a simulation study examining the performance of an age-based operating model and a SPM assessment model under 60 scenarios with various assumptions regarding data quantity, quality, and model priors. We provide specific good practice guidelines for two widely used state-space SPMs: SPiCT and JABBA. Finally, we discuss current limitations and suggest avenues for future developments for SPMs.
Inflation prediction in emerging economies: Machine learning and FX reserves integration for enhanced forecasting
2024, International Review of Financial Analysis
The present study makes two significant contributions to the extended body of literature in the context of International Finance. First, it forecasts the inflation in an emerging economy by employing a combination of traditional forecasting and Machine Learning models to test whether machine learning models outperform traditional forecasting models. Second, it explicitly includes an often-neglected variable i.e. foreign exchange reserves into the forecasting models to ascertain whether its inclusion enhances predictive accuracy. The outcomes of the study revealed interesting findings. It is observed that machine learning models consistently outperform traditional models, with Random Forest and Gradient Boosting are the top performers across different sets of determinants. Moreover, the study unveils that the inclusion of foreign exchange reserves into the models as a determinant has a positive impact on the predictive effectiveness of both traditional and machine learning-based inflation forecasting models.
Exploring device physics of perovskite solar cell via machine learning with limited samples
2024, Journal of Energy Chemistry
Perovskite solar cells (PSCs) have developed tremendously over the past decade. However, the key factors influencing the power conversion efficiency (PCE) of PSCs remain incompletely understood, due to the complexity and coupling of these structural and compositional parameters. In this research, we demonstrate an effective approach to optimize PSCs performance via machine learning (ML). To address challenges posed by limited samples, we propose a feature mask (FM) method, which augments training samples through feature transformation rather than synthetic data. Using this approach, squeeze-and-excitation residual network (SEResNet) model achieves an accuracy with a root-mean-square-error (RMSE) of 0.833% and a Pearson's correlation coefficient (r) of 0.980. Furthermore, we employ the permutation importance (PI) algorithm to investigate key features for PCE. Subsequently, we predict PCE through high-throughput screenings, in which we study the relationship between PCE and chemical compositions. After that, we conduct experiments to validate the consistency between predicted results by ML and experimental results. In this work, ML demonstrates the capability to predict device performance, extract key parameters from complex systems, and accelerate the transition from laboratory findings to commercial applications.
Ship order book forecasting by an ensemble deep parsimonious random vector functional link network
2024, Engineering Applications of Artificial Intelligence
Efficient forecasting of ship order books holds immense significance in the maritime industry, enabling companies to optimize their operations, allocate resources effectively, and make informed decisions. However, volatile characteristics within historical order books pose challenges in achieving reliable, intelligent, and precise forecasts. This paper presents a novel ensemble deep random vector functional link (edRVFL) algorithm to anticipate future ship order book dynamics. The edRVFL leverages deep feature extraction and ensemble learning to enhance forecasting performance. To further elevate its capabilities, we introduce a discontinuous and parsimonious embedding strategy, which deviates from the conventional dense collection of continuous time steps used in vanilla edRVFL. This parsimonious embedding approach limits the model’s complexity and boosts its generalization ability. We extensively evaluate the proposed method using ship order book data, and comparative studies demonstrate its superiority over alternative approaches. Our proposed edRVFL offers a promising solution for accurate and efficient ship order book forecasting, making it a valuable asset in the maritime industry’s decision-making processes. The source codes utilized in this research are openly available on GitHub at the following link: https://github.com/crkkkaa/Ship-order-book-forecasting-by-an-ensemble-deep-parsimonious-random-vector-functional-link-network-.
A landscape scale model to predict post-fire debris flow impact zones
2024, Geomorphology
Post-fire debris flows pose a significant hazard in mountainous regions, but accurate assessment of the associated risk at large scales, particularly in the context of mega-fires, is limited. Although numerous models exist, few are functional at the landscape scale, and none integrate assessments of downstream effects with the likelihood of flow initiation to define both the frequency and magnitude of potential flows. The aim of this work, therefore, was to develop a post-fire debris flow model that could characterise both the occurrence frequency and associated runout of debris flow events and which could be rapidly applied to fire events at the landscape scale. We achieve this using a model integration approach that included the development of a reduced-complexity debris flow runout model, which was calibrated and independently tested for its application at large scales with a dataset of 1377 individual post-fire flows across south-eastern Australia. This model was then integrated with previously published methods determining debris flow source areas and likelihood of initiation specific to our study area. The developed model is intentionally designed for rapid response hazard management. Therefore, it predominantly incorporates only the fundamental, lowest parameter form drivers of debris flow initiation and runout termination, thus minimising computational complexity while ensuring prediction accuracy sufficient for hazard assessment even at application scales as large as mega-fires. Accounting for debris flows initiated from single headwaters and cases where flows simultaneously converge from multiple initiation points, our integrated model effectively characterises debris flow hazard across a geomorphologically diverse landscape encompassing >200,000km² (capturing ~640,000 potential debris flow initiation locations across Victoria, Australia). Model testing using an independent validation dataset resulted in debris flow runout predictions with root mean square error (RMSE) of 176 and 308 m for non-convergent and convergent flows respectively. Crucially, these predictions can be computed rapidly for large (up to landscape scale) fire events with minimal input requirements and the use of a remotely-sensed observation dataset for domain calibration. As such, the model provides a critical means of addressing the capacity gap in assessments of post-fire debris flow hazard by efficiently integrating measures of occurrence frequency and runout for functional application at large scales. This new work empowers land managers to rapidly predict the hazard arising from post-fire erosion processes and represents an important applied step towards better understanding and mitigating the impacts of wildfire-induced debris flows.

View all citing articles on Scopus

Anne B. Koehler is a professor of decision sciences and the Panuska Professor of Business Administration at Miami University, Ohio. Professor Koehler has numerous publications, many of which are on forecasting models for seasonal time series and exponential smoothing methods. She is co-author of the fourth edition of Forecasting, time series, and regression: an applied approach, published by Duxbury in 2005.

¹: Tel.: +1 513 529 4826.

View full text