Steady-State Statistical Analysis

1. Steady-State Statistical Analysis By Dr. Jason Merrick

2. Warm Up and Run Length • Most models start empty and idle • Empty: No entities present at time 0 • Idle: All resources idle at time 0 • In a terminating simulation this is OK if realistic • In a steady-state simulation, though, this can bias the output for a while after startup • Bias can go either way • Usually downward (results are biased low) in queueing-type models that eventually get congested • Depending on model, parameters, and run length, the bias can be very severe Simulation with Arena - Steady-state Output Analysis

3. Warm Up and Run Length (cont’d.) • The period up to 1500 minutes is less congested • Thus average output measures will be biased down • How can we get rid of this bias? Simulation with Arena - Steady-state Output Analysis

4. Intelligent Initial Conditions • Collect data • Observe an actual state of the real system that has been running for a reasonable period of time • Use this state as the initial conditions • Not possible if system does not exist or you are modifying the system • Use another model • Queuing models, inventory models etc. • Give steady-state results under more restrictive assumptions than simulation • Use these results as initial conditions Simulation with Arena - Steady-state Output Analysis

5. Warm-up • Define some time tWuntil which no statistics are collected • Suppose mW observations are collected up to time tW • Suppose m observations are collected after time tW • The idea is that Ymw+1,…,Ym+mw are drawn from the “steady state” distribution, while Ym,…,Ymw are from a different warm-up distribution • So truncating the warm-up observations removes the bias Simulation with Arena - Steady-state Output Analysis

6. Series Averages Ensemble Averages Determining Warm-up Times • Ensemble averages • The average across replications of the first, second, third, … observations • Each ensemble average is an iid sample from the distribution of that observation • Put t-distribution confidence interval around each average • See when the ensemble averages settle down Simulation with Arena - Steady-state Output Analysis

7. Determining Warm-up Times Simulation with Arena - Steady-state Output Analysis

8. Determining Warm-up Times Simulation with Arena - Steady-state Output Analysis

9. Truncated Replications • If you can identify appropriate warm-up and run-length times, just make replications as for terminating simulations • Only difference: Specify Warm-Up Period in Simulate module Simulation with Arena - Steady-state Output Analysis

10. If model warms up very slowly, truncated replications can be costly Have to “pay” warm-up on each replication Throw away Batching in a Single Run Simulation with Arena - Steady-state Output Analysis

11. Batching in a Single Run • Alternative: Just one R E A L L Y long run • Only have to “pay” warm-up once • Problem: Have only one “replication” and you need more than that to form a variance estimate (the basic quantity needed for statistical analysis) • Big no-no: Use the individual points within the run as “data” for variance estimate • Usually correlated (not indep.), variance estimate biased throw away sample Simulation with Arena - Steady-state Output Analysis

12. Batching in a Single Run (cont’d.) • Break each output record from the run into a few large batches …... Warm-up Batch 1 Batch 2 Simulation with Arena - Steady-state Output Analysis

13. Batching in a Single Run (cont’d.) • The batch means will not actually be independent • The idea is to reduce the correlation to a level that it will not introduce a significant bias in the estimate of the standard deviation • The individual observations in the series are correlated with the previous observations • The correlogram shows that the correlation reduces the higher the lag • So if the batch size is long enough then most of the observations making up batch 1 will be approximately independent of those making up batch 2 • Only the observations near the end of the batches will be correlated Batch 1 Batch 2 Simulation with Arena - Steady-state Output Analysis

14. Batching in a Single Run (cont’d.) • Rules of thumb • Schmeiser (1982) found that for a given run length, there was little benefit in more than 30 batches • However, less than 10 batches was too few • There may well be correlation between all lags, looking at the lag 1 correlation is usually enough to ascertain independence • Auto-correlation estimates are not very good for sample sizes like 30 • Use smaller batches (say c > 100) and if the independence test is passed then the bigger batches will be fine Simulation with Arena - Steady-state Output Analysis

15. Examining # of Batches • Consider the Simple Processing System • t = 1,000,000 minutes, increase number of batches CI too small CI doesn’t change much Simulation with Arena - Steady-state Output Analysis

16. Examining Run Length • Consider the Simple Processing System • n = 25 replications, increase the run length Simulation with Arena - Steady-state Output Analysis

17. Examining # of Replications • Consider the Simple Processing System • t = 15 minutes, increase the number of replications Simulation with Arena - Steady-state Output Analysis