SIMULATION MODELING AND ANALYSIS WITH ARENA T. Altiok and B. Melamed Chapter 8 Model Goodness: Verification and Valid

1. SIMULATION MODELING AND ANALYSIS WITH ARENA T. Altiok and B. Melamed Chapter 8 Model Goodness: Verification and Validation Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

2. Verification and Validation • Verification assesses the correctness of the formal representation of the intended model (in our case, a computer simulation program), by inspecting computer code and test runs, and performing consistency checks on their statistics • Validation assesses how realistic the modeling assumptions are, by comparing model performance metrics (predictions), obtained from model test runs, to their counterparts in the system under study • obviously, validation is possible only if the simulation program exists Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

3. Model Verification via Test Runs • Verification activities via inspection of test runs include • input parameters and output statistics • using a debugger • using animation • sanity checks Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

4. WIP inventory Processor Job Departures Job Arrivals Model Verification via Performance • A simple workstation • A Generic workstation can be described as a queueing system • the flow of incoming jobs, either singly or in batches, forms the arrival stream • the time a job is delayed for processing at the workstation server is its service time. • if an incoming job cannot start processing right away (because servers are busy), then it is held in a buffer (finite or infinite), and in due time will be removed from the buffer and assigned a server (real-life workstations it is, of course, finite) • a job that finds the buffer full on arrival is usually rerouted to another workstation or simply assumed to be lost • the order in which jobs queue up in the buffer is called a queueing discipline. • a workstation model may have additional wrinkles, such as server failure and repair, routinely modeled via random uptimes and random downtimes Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

5. Queueing Conventions and Notation • The arrival process is denoted by , where is the i-th interarrival interval, separating customers i-1 and i • for stationary arrival processes, the rate is denoted by • The service process is denoted by , where is the i-th service time • for stationary service processes, the rate is denoted by • A standard notation has been developed to specify a queueing system succinctly, employing slash-separated symbols, representing arrival process, service process, number of servers and queue capacity in this order (an omitted capacity is understood to be infinite) • the symbol GI stands for a general iid distribution (special cases are M for an exponential distribution and D for a deterministic distribution) • the symbol G stands for a general distribution non- iid distribution • M/M/1 specifies a queue with iid exponential distributions for inter-arrival and service times (Poisson process), a single server and infinite buffer capacity • M/D/k/C specifies a queue with iid exponential inter-arrival times, deterministic service times, k servers and a system of capacity of C, of which the first k positions are occupied by servers Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

6. Service Disciplines • Common service disciplines: • FIFO (First-In-First-Out), also known as FCFS (First-Come-First-Served) is the most common discipline, where jobs are served in order of arrival • LIFO (Last-in-First-Out), also known as LCFS (Last-Come-First-Served), serves the most recent arrival first • SIRO (Service-In-Random-Order) selects the next job in the buffer randomly, with equal probabilities • RR (Round Robin) has associated with it a (fixed) service time, often referred to as the time quantum, and jobs are served cyclically, one quantum at a time, until attaining their requisite service time • PS (Priority-Service) assumes that jobs have priorities associated with them, and selects a job with the highest service priority jobs, such as FIFO, LIFO, SIRO, etc. Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

7. Queueing Performance Measures • Common queueing performance measures: • average number of jobs in the queue (buffer only) • average number of jobs in the system (buffer and servers) • average job waiting time (buffer only delay) • average job sojourn time (buffer and service delays) • server utilization (fraction of time a server is busy) • throughput (output rate of the system) Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

8. Average Number of Jobs • Denote the following quantities as follows: • is the average number of jobs in the system, including queue (buffer) and servers • is the average number of jobs in the queue (buffer) only • For a queueing system in steady state of capacity (buffer size) (finite or infinite), • where denotes the steady-state probability of n jobs in the system • For a single-server queueing system, • where is the steady-state average number of jobs in service Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

9. Queueing System Stability • In general, the server utilization, , of a single-server queueing system with any queue capacity is given by • For infinite capacity systems, the utilization can be expressed as the ratio of the input rate and the service rate, that is, • When , then system is said to be unstable, andthe server is said to be exhausted • In this case, the server is not able to keep up with the incoming workload, and consequently the number of jobs in the system grows without bound in the long run (“system explosion”) • When , then system is said to be stable • stability is needed for long-run performance measures to exist • in the context of simulation modeling we are typically interested in stable systems • a finite-capacity system is always stable, because the number of jobs in it is bounded by its capacity. Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

10. Regenerative Queueing Systems • A regenerative queueing system is one where temporal histories can be decomposed into periods (random intervals), such that the associated stochastic processes are independent over distinct periods • the onset of a busy cycle is called a regeneration point, since the system “regenerates” itself there probabilistically • each independent period is called a regeneration cycle (or busy cycle), and extends over a pair of consecutive busy period (when the server is busy) and idle period (when the server is idle), as shown below • Regeneration cycles consisting of successive idle and busy periods Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

11. Regeneration Cycles and utilization • Consider a regeneration cycle in a GI/GI/1 queue, and define • is the length of the busy period • is the length of the regeneration cycle • Since the number of incoming and outgoing jobs in a regeneration cycle must coincide, one has • or equivalently • It follows that • For Poisson arrivals (iid exponential interarrival intervals) • Since residual interarrival intervals are exponential too, • Thus, and Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

12. Queuing System Throughput • The throughput of a queueing system is its long-run output rate (jobs served per unit time), given by • which is the service rate while the server is busy • For an infinite capacity queue, this becomes • by flow conservation Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

13. Little’s Formula • Little’s Formula is a key general conservation equation, linking the mean number of jobs in the system, , with the arrival rate, , and the mean waiting time, ,given by • note carefully that is not the offered arrival rate,but the effective arrival rate, excluding any lost jobs! • when the capacity is infinite (so no losses are incurred), then the offered arrival rate and effective arrival rate are the same • The formula holds for any queueing subsystem • for the entire queueing system (buffer plus servers), • for the buffer subsystem only, Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

14. Steady-State Flow Conservation • Define the following quantities: • is the long-run fraction of jobs that find k jobs in the system on arrival. Such quantities are referred to as customer-average probabilities arrival point probabilities. Note carefully that the number of jobs in the system is computed only at the (random) times of arrival and excludes the arriving job. For this reason this kind of system state is referred to as the state embedded at arrival times, or as the state seen by arriving jobs. • is the long-run fraction of time that the system has precisely k jobs in it. Such quantities are referred to as time-average probabilities or arbitrary time probabilities. • Generally • For single-server queues with finite capacity, , Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

15. The PASTA Property • While generally , under certain conditions the equality • does hold • The most common case where the equality above holds is when the arrival stream is a Poisson process • This case is known as PASTA (Poisson Arrivals See Time Averages) • The general case when this equality holds is known as ASTA (Arrivals See Time Averages) • It is possible to have ASTA which is not PASTA… Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

16. Model Verification: Single Workstation • As an example, consider an Arena model of a single workstation below • Arena model of a single workstation model Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

17. Single Workstation Queue Statistics • Arena Queues summary statistics report for the single workstation model Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

18. Single Workstation Resource Statistics • Arena Resources Summary statistics report for the single workstation model Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

19. Single Workstation User Statistics • Arena User Specified statistics report for the single workstation model Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

20. Single-Station Verification • For a replication r, define • is the number of departures (jobs processed) during replication r • is the length (duration) of replication r • the throughput estimator over that period in that replication is • The verification procedure to be outlined uses various sections of the Arena summary reports to estimate the throughput in multiple alternative computations, to show that they are all in reasonable agreement • note that is just circumstantial evidence of verification • any major discrepancy among the computations would suggest an errorin the simulation program (bug) and will preclude verification Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

21. Verification via Throughput • Throughput estimation from the Queues report and the Counter section in the User Specified report • an examination of the previous report shows that replication 1 ran for 100,000 hours, and the Counter section reveals that D(1) = 20,736 jobs were processed and 1,503 jobs were rejected. • the first estimate of the throughput is jobs/hour • Throughput estimation from the Resources section and the throughput equation • an examination of the Resources report shows that the utilization was estimated as , and while the workstation was busy, it processed jobs at the rate of 0.25 jobs per hour • by the throughput equation , a second estimate of the throughput is jobs/hour Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

22. Verification via Throughput (Cont.) • Throughput estimation from the Tally section in the User Specified report • using the mean inter-departure time from the Tally section, a third estimate of the throughput is jobs/hour • Throughput estimation from the Counter section in the User Specified report and the effective arrival rate • the Counter section shows that the number of jobs that succeeded in entering the workstation was 20,736 out of a total of 20736+1503 jobsthat arrived at the workstation, so that we estimate • since the throughput is the effective arrival rate, a fourth estimate is just jobs/hour Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

23. Verification via Little’s Formula • Little’s formula provides another common verification check, using the summary statistics in the previous report as follows: • for the buffer subsystem, Little’s formula is • from the Queues report of Figure 8.4, we have • from the Queues report of Figure 8.4, we also have • from the Counter Section of the User Specified report, the effective arrival rate (equal to throughput ) is • computing the right-hand side of Little’s formula yields • since the values of and are close, we conclude that the simulation agrees with Little’s formula Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

24. Assembly Painting Model Verification: A Tandem Network • As an example, consider the Arena model of a network of two workstations (assembly and painting) in tandem, shown below • suppose that neither workstation has buffer capacity limitations • assume further that the arrival stream consists of two types of jobs: type 1 and type 2, each with its own assembly and painting operations • job inter-arrival times are exponentially distributed with means 4 hours for type 1 and 10 hours for type 2. • for type 1 jobs, assembly times (in hours) are distributed according to the Tria(1,2,3) distribution, while painting times are deterministic lasting 3 hours • for type 2 jobs, assembly times (again in hours) are distributed according to the Tria(1,3,8) distribution, while painting times are deterministic lasting 2 hours • this model is simulated for 100,000 hours to obtain resource utilizations and mean flow times by job type, and average WIP (work-in-process) levels in each buffer Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

25. Tandem Network Arena Model Arena model of the tandem flow line Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

26. Tandem Network Arena Model (Cont.) Spreadsheet view of the Resource module with StateSet module names Dialog box for the StateSet module defining the states of each resource Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

27. Tandem Network Arena Model (Cont.) Dialog box and spreadsheet view for the Record module tallying flow times Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

28. Tandem Network Arena Model (Cont.) Dialog box for the Statistic module with Frequency statistics Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

29. Tandem Network Resources Statistics Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

30. Tandem Network Frequencies Statistics Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

31. Tandem Network Queues Statistics Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

32. Tandem Network User Specified Statistics Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

33. Verification via Utilization • Theoretical utilizations at the Assembly workstation • the total workstation utilization is • the partial utilizations of type 1 jobs is • the partial utilizations of type 2 jobs is • the sum of utilizations checks, since • Simulation utilizations at the Assembly workstation • the simulation results from the report estimate the utilization of the assembly workstation as . • The two figures, and , are close, but would be even closer if we were to run the model longer Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

34. Verification via Little’s Formula • A verification via Little’s formula is carried out by the following procedure: • the total arrival rate is • The simulation estimates of average flow time of type 1 and 2 jobs • are, respectively, and • The estimated combined average flow time over all job types (weighted by the relative arrival rates) is • The average number of jobs in the system is estimated from the simulation run as the sum of average buffer sizes plus workstation utilizations over all job types, namely, • Thus, we find that andagree closely, in agreement with Little’s formula Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

35. Model validation • Validation activities are critical to the construction of credible models • The standard approach to model validation is to collect data (parameter values, performance metrics, etc.) from the system under study and compare them to their model counterparts • the data collection effort of Input Analysis can provide the requisite data from the system under study • data collected is classified into input values and corresponding output values Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

36. Model validation (Cont.) • Consider a generic system • Raw input data is obtained by observing the system over a time period • A model is constructed from the observed data and run to produce simulation data (that serve as input data sets) from which the corresponding output data (performance metrics) are then estimated • The correspondence of the generic inputs, , and generic outputs, ,is shown below Generic input and output data Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

37. Model validation (Cont.) • Each set of output metric values constitutes a sample of random values (say, daily mean delay times) • two sets (samples) of output daily mean delay data are collected for days • 1. a sample of observed daily mean delays,collected from the real-life system under study • 2. a sample of estimated daily mean delays, • collected from runs of the simulation model of the system under study • Thus, • for the real-life system under study, • for the simulation runs, Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

38. Model validation (Cont.) • To test that the observed and simulated delays have the same means, define the statistics ,assumed approximately normally distributed with mean and variance • Perform hypothesis testing of the formwith the test statistic , where • has a t distribution with degrees of freedom • and are the sample mean and sample standard deviation,respectively, of • The confidence interval of the test at significance level is • The null hypothesis cannot be rejected if the confidence interval above contains 0, supporting the model is valid Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8