Automated Analysis of Simulation Output Data and the AutoSimOA Project

1. Automated Analysis of Simulation Output Data and the AutoSimOA Project Stewart Robinson and Katy Hoad and Ruth Davies Warwick Business School Simulation Group Seminar, 5 May 2006

2. Outline The problem An automated output Analyser: • Warm-up analysis • Replications analysis • Run-length analysis: batch means method Example (demonstration) Discussion The AutoSimOA Project

3. The Problem Prevalence of simulation software: ‘easy-to-develop’ models and use by non-experts. Simulation software generally have very limited facilities for directing/advising on simulation experiments. Main exception is directing scenario selection through ‘optimisers’. With a lack of the necessary skills and support, it is highly likely that simulation users are using their models poorly.

4. The Problem Despite continued theoretical developments in simulation output analysis, little is being put into practical use. • There are 3 factors that seem to inhibit the adoption of output analysis methods: • Limited testing of methods • Requirement for detailed statistical knowledge • Methods generally not implemented in simulation software (AutoMod/AutoStat is an exception) A solution would be to provide an automated output ‘Analyser’.

5. Simulation model Output data Analyser Warm-up analysis Obtain more output data Use replications or long-run? Replications analysis Run-length analysis Recommendation possible? Recommend- ation An Automated Output Analyser • For this project the Analyser looked at: • Warm-up • Run-length • Number of replications • Scenario analysis could be added.

6. An Automated Output Analyser A prototype Analyser has been developed in Microsoft Excel. At present it links to the SIMUL8 software, but it could be used with any software that can be controlled from Excel VBA.

7. Warm-up Analysis The Analyser uses 3 procedures from which the user can select the desired warm-up period: MSER-5, Batch Means Bias Detection, Welch’s Method. The 3 procedures were chosen on the basis of: • Accuracy • Reliability • Generality • Ease of implementation (in Excel) • Requires minimum user intervention • Varied (e.g. not all graphical procedures)

8. Warm-up Analysis Adaptation of Welch’s Method Smoothness Criterion ith jump: Average jump: Increase window size until average jump is reduced to 10% of its value in the raw data.

9. Warm-up Analysis Adaptation of Welch’s Method Convergence Criterion (average difference rule) Suppose the moving average plot becomes smooth at observation Xj. Then Cjshould have a low value: Obtain a value for Cj such that Cj/M<L M is the difference between the max and min Xifor i>=j Tests showed that a value of L=0.0025 gave convergence close to that chosen by visual inspection.

10. Replications Analysis Option to run normal streams or mixed normal and antithetic streams. Set significance level and confidence interval half width (%).

11. Run-Length Selection: Batch Means Method Three procedures are used for selecting the batch size: • Fishman’s algorithm (Fishman, 1978) • Law and Carson’s algorithm (Law and Carson, 1979) • ABATCH algorithm (Fishman and Yarberry, 1997)

12. Example The Analyser is applied to an M/M/1 queuing model in SIMUL8: Arrival rate = 1 Service rate = 0.67 Queue limit: 100 Output statistic: customers in the system Demonstration!

13. Example Run Length: Batch Means example

14. Example Run Length: Batch Means example results

15. Discussion It is possible to link an Automated Analyser in Excel to a simulation software tool. At present this is just a proof of concept. • Key issues to address: • More thorough testing of output analysis methods for their accuracy and their generality. • Adaptation of methods to sequential procedures and to minimise the need for user intervention.

16. The AutoSimOA Project A 3 year, EPSRC funded project in collaboration with SIMUL8 Corporation. • Objectives • To determine the most appropriate methods for automating simulation output analysis • To determine the effectiveness of the analysis methods • To revise the methods where necessary in order to improve their effectiveness and capacity for automation • To propose a procedure for automated output analysis of warm-up, replications and run-length • Only looking at analysis of a single scenario

17. The AutoSimOA Project Programme of work:

18. The AutoSimOA Project • CURRENT WORK: • Literature review of warm-up, replications and run-length methods * • Development of artificial data sets (Auto-Regressive; Moving average; M/M/n/p Queues…) • and collection of ‘real’ simulation models • Produce output data • Analyse and categorise output: Auto-correlation; Normality; M.A.D…..

19. Initial Bias Functions: Exponential Mean shift 2 Under Damped oscillations Example artificial models: 1.Auto-Regressive (2) series

21. Example artificial models: 2.E4 ~ Erlang(4) / M / 1 Queue mean 1.8 Traffic Intensity = 0.8

22. Example ‘ real ’ models: 1.Coventry Train Station – Queuing Times

23. Example ‘ real ’ models: 2.Argos – Queuing Times

24. Example ‘ real ’ models: 3.Tesco petrol Station – Queuing Times

25. Example ‘ real ’ models: 4.Café Library – Queuing Times

26. Auto Correlation Spread round mean In/out of control Terminating Group B Non-terminating Trend Normality Transient Steady state Seasonality