1 / 22

Evaluation

Evaluation The evaluation of Naive Forecasting Techniques relies primarily on the comparison of the forecasts with the corresponding actual values. Evaluation Methods Mean Error (ME) Mean Absolute Error (MAE) Mean Squared Error (MSE) Mean Percentage Error (MPE)

saxon
Download Presentation

Evaluation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Evaluation • The evaluation of Naive Forecasting Techniques relies primarily on the comparison of the forecasts with the corresponding actual values

  2. Evaluation Methods • Mean Error (ME) • Mean Absolute Error (MAE) • Mean Squared Error (MSE) • Mean Percentage Error (MPE) • Mean Absolute Percentage Error (MAPE)

  3. The ME can be very misleading. A ME value of zero can mean that the method forecasted the actual values perfectly (unlikely) or that the positive and negative errors cancelled each other out. It tends to Understate the error in all cases.

  4. Evaluation Methods • Mean Error (ME) • Mean Absolute Error (MAE) • Mean Squared Error (MSE) • Mean Percentage Error (MPE) • Mean Absolute Percentage Error (MAPE)

  5. MAE is a way of dealing with the Understatement of ME. By using the Absolute values of the error, the mean gives a better indication of the model’s fit.

  6. Evaluation Methods • Mean Error (ME) • Mean Absolute Error (MAE) • Mean Squared Error (MSE) • Mean Percentage Error (MPE) • Mean Absolute Percentage Error (MAPE)

  7. The MSE eliminates the positive/negative problem by squaring the errors. The result tends to place more emphasis on the larger errors and, therefore, gives a more conservative measure than the MAE.

  8. The previous three measures are “series specific;” i.e., they only allow evaluation of the series that generated the errors. • The next two measures, by using the percentage of the error relative to the actual, are designed to allow comparison of the results with different models.

  9. Evaluation Methods • Mean Error (ME) • Mean Absolute Error (MAE) • Mean Squared Error (MSE) • Mean Percentage Error (MPE) • Mean Absolute Percentage Error (MAPE)

  10. The MPE is a relative measure of the forecasting error. It is subject to the “averaging” of the positive and negative errors.

  11. Evaluation Methods • Mean Error (ME) • Mean Absolute Error (MAE) • Mean Squared Error (MSE) • Mean Percentage Error (MPE) • Mean Absolute Percentage Error (MAPE)

  12. MAPE is a comparative measure that does not have the problem of averaging the positive and negative errors. It is relatively easy to use to communicate a model’s effectiveness.

  13. Measurement of Forecasting Error • Mean Error (ME): The average of all the errors of forecast for a group of data. • Mean Absolute Error (MAE): The mean, or average of the absolute values of the errors. • Mean Square Error (MSE): The average of the squared errors. • Mean Percentage Error (MPE): The average of the percentage errors of a forecast. • Mean Absolute Percentage Error(MAPE): The average of the absolute values of the percentage errors of a forecast.

  14. Year Actual Forecast Error 1 1402 2 1458 1402.0 56.0 3 1553 1441.2 111.8 4 1613 1519.5 93.5 5 1676 1584.9 91.1 6 1755 1648.7 106.3 7 1807 1723.1 83.9 8 1824 1781.8 42.2 9 1826 1811.3 14.7 10 1780 1821.6 -41.6 11 1759 1792.5 -33.5 Example: Nonfarm Partnership Tax Returns: Actual and Forecast with  = .7

  15. Year Actual Forecast Error 1 1402.0 2 1458.0 1402.0 56.0 3 1553.0 1441.2 111.8 4 1613.0 1519.5 93.5 5 1676.0 1584.9 91.1 6 1755.0 1648.7 106.3 7 1807.0 1723.1 83.9 8 1824.0 1781.8 42.2 9 1826.0 1811.3 14.7 10 1780.0 1821.6 -41.6 11 1759.0 1792.5 -33.5 524.3 Mean Error for the Nonfarm Partnership Forecasted Data

  16. Year Actual Forecast Error |Error| 1 1402.0 2 1458.0 1402.0 56.0 56.0 3 1553.0 1441.2 111.8 111.8 4 1613.0 1519.5 93.5 93.5 5 1676.0 1584.9 91.1 91.1 6 1755.0 1648.7 106.3 106.3 7 1807.0 1723.1 83.9 83.9 8 1824.0 1781.8 42.2 42.2 9 1826.0 1811.3 14.7 14.7 10 1780.0 1821.6 -41.6 41.6 11 1759.0 1792.5 -33.5 33.5 674.5 Mean Absolute Errorfor the Nonfarm Partnership Forecasted Data E

  17. Year Actual Forecast Error Error2 1 1402 2 1458 1402.0 56.0 3136.0 3 1553 1441.2 111.8 12499.2 4 1613 1519.5 93.5 8749.7 5 1676 1584.9 91.1 8292.3 6 1755 1648.7 106.3 11303.6 7 1807 1723.1 83.9 7038.5 8 1824 1781.8 42.2 1778.2 9 1826 1811.3 14.7 214.6 10 1780 1821.6 -41.6 1731.0 11 1759 1792.5 -33.5 1121.0 55864.2 Mean Square Error for the Nonfarm Partnership Forecasted Data

  18. Year Actual Forecast Error Error % 1 1402 2 1458 1402.0 56.0 3.8% 3 1553 1441.2 111.8 7.2% 4 1613 1519.5 93.5 5.8% 5 1676 1584.9 91.1 5.4% 6 1755 1648.7 106.3 6.1% 7 1807 1723.1 83.9 4.6% 8 1824 1781.8 42.2 2.3% 9 1826 1811.3 14.7 0.8% 10 1780 1821.6 -41.6 -2.3% 11 1759 1792.5 -33.5 -1.9% 31.8% Mean Percentage Error for the Nonfarm Partnership Forecasted Data

  19. Year Actual Forecast Error |Error %| 1 1402 2 1458 1402.0 56.0 3.8% 3 1553 1441.2 111.8 7.2% 4 1613 1519.5 93.5 5.8% 5 1676 1584.9 91.1 5.4% 6 1755 1648.7 106.3 6.1% 7 1807 1723.1 83.9 4.6% 8 1824 1781.8 42.2 2.3% 9 1826 1811.3 14.7 0.8% 10 1780 1821.6 -41.6 2.3% 11 1759 1792.5 -33.5 1.9% 40.3% Mean Absolute Percentage Error for the Nonfarm Partnership Forecasted Data

  20. Use of Error Measures To identify the best forecasting method • Use error measure to identify the best value for the parameters of a specific method. • Use error measure to identify the best method. • Use MSE and MAE for both of these situations. Note that MSE tends to emphasize large errors.

  21. Use of Error Measures, continued Forecast bias is the tendency of a forecasting method to over or under predict. The mean error, ME, measures the forecast bias.

More Related