This lecture section is a cumulative review of material from the semester and is meant to serve as a study guide for students preparing for the upcoming final exam. Topics start at modeling fundamentals (what is the purpose of a model in general) to the specifics of designing statistical experiments with stochastic simulations.
[ due to an instructor error, the lecture from 2021-11-09 was not recorded, and the archived 2020-12-01 lecture is re-used here instead ]
In this lecture, we wrap up our discussion of Variance Reduction Techniques. We introduced Common Random Numbers (CRNs) last time, which we review in this lecture. We then introduce Control Variates (CVs), Antithetic Variates (AVs), and Importance Sampling. These four methods are all examples of amplifying signals in a statistical experiment either by manipulating the simulation execution or using information about known sources of variance to increase statistical power.
In this lecture, we wrap up our discussion of the movement from point estimation (sample means) to interval estimation for: (a) estimating absolute performance of a system, (b) estimating relative performance of two systems, and (c) estimating relative performance of more than 2 systems. We then pivot to discussing Variance Reduction Techniques (VRT's), starting with Common Random Numbers (CRN's).
In this lecture, we further review the use of confidence intervals to summarize empirical results from simulation as we move from thinking about absolute performance estimation (i.e., using one model system to estimate one parameter) to relative performance estimation (i.e., comparing two model systems to make an inference about whether they differ). This allows us to discuss how confidence intervals are used in regression analysis and start to motivate how to build confidence intervals that are summarizes of 2-sample (instead of 1-sample) t-tests. We had to stop a little early, and so the next lecture will discuss how to convert paired t-tests and two different types of 2-sample t-tests into 2-sample confidence intervals (which are compared to 0).
This lecture continues to discuss issues related to estimating absolute performance from transient and steady-state simulations (of terminating and non-terminating systems, respectively). We continue to emphasize the importance and utility of interval estimations (over point estimates). We then move on to discuss experimental methodologies useful for steady-state simulations, particularly related to eliminating estimator bias and reducing computational time.
[ due to an instructor error, the lecture from 2021-11-09 was not recorded, and the archived 2020-11-010lecture is re-used here instead ]
In this lecture, we continue to introduce terminating and non-terminating systems and difference methods for estimating performance from simulation models of them (using transient and steady-state simulations). This involves a description of various types of point estimators (mean and quantile) as well as related interval estimators (confidence intervals and prediction intervals, as well as the relationship to standard error of the mean (SEM)). We start to discuss issues involving making inferences from pseudo-replicated within-replication samples versus across-replication samples (which are independent and often normally distributed). We will continue this in the next lecture, as we start focusing more on steady-state simulations of non-terminating systems.
[ For some strange reason, the in-room video camera was not recorded as the speaker view despite apparently working during the class. Consequently, only the slide view is shown. ]
In this lecture, we review the fundamental tradeoffs in hypothesis testing and the concrete origins of the assumptions in both the t-test and Chi-square test. We also discuss parametric and non-parametric statistics (including exact and non-exact tests) and how non-parametric, exact statistics like the Kolmogorov–Smirnov test are derived. This culminates in a discussion of the multiple comparisons (MC) problem and the Bonferroni correction as well as alternative tests (such as a MANOVA or an ANOVA with post hoc test) that have more statistical power than the Bonferroni correction. We close with an introduction to performance inference from simulation, which we will continue discussing in the next 3 lectures.
In this halloween-themed lecture, we go into more detail on the foundations of hypothesis testing – specifically hypothesis testing with small sample sizes. This allows us to talk about where the Student's t test comes from (and why it is defined that way) as well as where the Chi-square test comes from (and why it is defined that way). Throughout the lecture, we highlight the importance of statistical power and do a power analysis example for a paired-difference t-test.
In this lecture, we review summary statistics, MLE, and goodness-of-fit tests (particularly Chi-square and Kolmogorov–Smirnov, with some mention of Anderson–Darling and Shapiro–Wilk), with a particular focus on the type-I error, type-II error, and statistical power. We then introduce verification, validation, and calibration of simulation models and close with an example for the simulation of a bank. We use rigorous statistical methods to drive the calibration process that leads to updating the model of the bank and ensuring its outputs are a good statistical match for outputs in a real bank. This involves making use of a power analysis for a one-sample, two-sided t-test. We will cover the paired t-test version of this problem in the next lecture.
In this lecture, we start out with Q-Q and P-P probability plots that we did not have time to cover from last time. We then transition to a review about type-I error and p values and try to motivate the topics of STATISTICAL POWER and EFFECT SIZES, which we will dive into more in the next few lectures. We then discuss summary statistics and how to use methods such as maximum likelihood estimation (MLE) to come up with good choices of parameters for distributions picked in the input modeling process. Next time, we will discuss testing the (goodness of) fit for those parameterized distributions.
In this lecture, we continue our discussion of input modeling in depth. We start with a more detailed example of how data collection can guide the choice of the structural features of a system. We then move to the point in the process when the structure of the model is set but the input models have to be chosen based on collected data. We cover methods for generating histograms and matching those histograms to common distributions (both discrete and continuous). We stop just before discussing Q-Q plots and P-P plots, which we will pick up next time along with discussing how to parameterize these chosen distributions.
In this lecture, we introduce the 3-lecture unit on "Input Modeling." We start with motivations from thinking about stochastic simulation models and then describe the potential problems that can occur in collecting data. We close with a set of rules that can be helpful to follow when collecting data. We will start on choosing probabilistic families, parameterizing them, and testing goodness of fit next lecture (and extending over the next lecture).
In this lecture, we review topics from the first half of the semester that will be tested over in the upcoming midterm. Most of the class involves working examples on the whiteboard.
Whiteboard notes captured for this session can be found at: https://www.dropbox.com/s/pih0wt3abwbatbb/IEE475-LectureF-2021-09-30-Midterm_Review-Whiteboard_Notes.pdf?dl=0
In this lecture, we finish covering tests of uniformity (Chi-squared and Kolmogorov–Smirnov) and independence (autocorrelation and runs (above and below) tests) for pseudo-random number generators (PRNGs). We then move on to discussing the details of inverse-transform sampling for random-variate generation. We cover how to derive a CDF from a piecewise PDF and how to invert a CDF to produce a quantile function fit for random-variate generation. We also discuss the discrete inverse-transform case.
We start the lecture covering some discrete random variables that we did not get to during Lecture D2. We also introduce the Poisson process and how it relates to the Poisson and exponential random variables. We then pivot to discussing pseudo-random number generators (PRNGs), including their required as well as desired properties and statistical tests to test for independence and uniformity. We will continue the discussion of statistical tests for independence at the start of next lecture (Lecture E2).
In this lecture, we review basic probability space concepts from the previous lecture. We then go on to discuss the common probabilistic models that we will use in stochastic simulation (e.g., uniform, triangular, normal, exponential, Weibull, Erlang, Poisson, etc.). Basic background on the structure of each distribution is given as well as practical reasons why one distribution might be picked over another.
In this lecture, we use motivation from stochastic modeling (i.e., incorporating randomness into models in order to capture realistic variation without having to specify a great many details) to formally introduce random variables and probability spaces (as a subset of measure theory). We heavily lean on the analogy between probability and mass as we introduce the sample space, probability measure, random variable, probability mass function (pmf), probability density function (pdf), cumulative distribution function (cdf), and moments (including expectation and central moments as in variance).
In this lecture, we review results from the Monte Carlo simulation lab (Lab 3) and setup motivation for the agent-based modeling/NetLogo lab (Lab 4). For the MC-lab review, we cover the estimation of pi by drawing random coordinates in the unit cube. We also discuss the possibly counter-intuitive results from estimating the length of a 3-path stochastic activity network. To prepare for Lab 4, we review the three different types of simulation methodologies (ABM/IBM, DES, and SDM) and then give a brief introduction to NetLogo. A more detailed/tutorial introduction to NetLogo will take place during Lab 4.
In this lecture, we discuss more sophisticated dynamical simulation models that can be implemented within spreadsheets. We start with a review of the M/M/1 single-channel, single-server queueing node and then show how more explicit state variables can be introduced in an M/M/2 version (i.e., with two servers). We then discuss two different popular inventory management models (implemented within a spreadsheet) -- the "Order-up-to (M,N)" model as well as the "newsvendor (single-period/perishable) model". We close with some discussion of Monte Carlo methods -- which apply simulation techniques as numerical methods to solve mathematical problems that might otherwise be intractable analytically. Despite all of these examples of the power of spreadsheets, we end with a hint that much more is possible in terms of simulation of complex systems if we use specialized simulation tools. We will introduce some of those more specialized tools starting in the next lecture.
In this lecture, we review hand-simulation/DES simulation basics. We then introduce how to simulate discrete event system simulations (which are dynamic simulation models built around the idea of "state") in declarative programming frameworks like spreadsheets (which have no "state"). We work through the relationships necessary to encode in a spreadsheet to simulate a single-channel, single-server queue. We then pivot to covering comments from Lab 2, which was a hand simulation of a system with partial batching. This allows for motivating why we use multiple replications when we do empirical work with stochastic simulations, and how tools such as common random numbers can reduce variance but require special statistical tools (such as the paired-difference t-test). We then discuss the multiple comparisons problem, some ways to solve it, and how linear models extend what we can say from empirical studies -- so we can go from statistically significant to practically significant. (i.e., we can better characterize the "effect size" of a variable we have control over).
In this lecture, we carry forward our high-level description of the event-scheduling world view to specific hand-simulation examples of a single-channel, single-server queueing network node.
In this lecture, we review modeling basics for process-centric modeling (entities, resources, events, activities, delays, etc.) and then introduce the event-scheduling world view that acts behind the scenes in any discrete event system (DES) simulation. We begin discussing hand simulation of DESS, at least in the abstract. More concrete examples are to come in the next lecture.
In this lecture, we pivot from our general introduction to (quantitative) modeling to a more specific introduction of simulation modeling. System dynamics modeling (SDM), agent-based modeling (ABM), and discrete event system (DES) simulation are introduced, with the most detail on DES that will be the focus for the course. We then motivate the approach of "stochastic modeling" -- using randomness in these models in place of deterministic details.
In this lecture, we introduce the basic motivations for quantitative modeling -- including fundamental definitions of what is a model. This definition is meant to cover all models -- from fashion models to mouse models to statistical models to simulation models.
Recorded day-1 lecture of IEE 475 (Simulating Stochastic Systems) in the Fall 2021 semester. Introduces course and its policies. Audio is poor due to microphone support in room.
