This lecture section is a cumulative review of material from the semester and is meant to serve as a study guide for students preparing for the upcoming final exam. Topics start at modeling fundamentals (what is the purpose of a model in general) to the specifics of designing statistical experiments with stochastic simulations.
[ due to an instructor error, the lecture from 2021-11-09 was not recorded, and the archived 2020-12-01 lecture is re-used here instead ]
In this lecture, we wrap up our discussion of Variance Reduction Techniques. We introduced Common Random Numbers (CRNs) last time, which we review in this lecture. We then introduce Control Variates (CVs), Antithetic Variates (AVs), and Importance Sampling. These four methods are all examples of amplifying signals in a statistical experiment either by manipulating the simulation execution or using information about known sources of variance to increase statistical power.
In this lecture, we wrap up our discussion of the movement from point estimation (sample means) to interval estimation for: (a) estimating absolute performance of a system, (b) estimating relative performance of two systems, and (c) estimating relative performance of more than 2 systems. We then pivot to discussing Variance Reduction Techniques (VRT's), starting with Common Random Numbers (CRN's).
In this lecture, we further review the use of confidence intervals to summarize empirical results from simulation as we move from thinking about absolute performance estimation (i.e., using one model system to estimate one parameter) to relative performance estimation (i.e., comparing two model systems to make an inference about whether they differ). This allows us to discuss how confidence intervals are used in regression analysis and start to motivate how to build confidence intervals that are summarizes of 2-sample (instead of 1-sample) t-tests. We had to stop a little early, and so the next lecture will discuss how to convert paired t-tests and two different types of 2-sample t-tests into 2-sample confidence intervals (which are compared to 0).
This lecture continues to discuss issues related to estimating absolute performance from transient and steady-state simulations (of terminating and non-terminating systems, respectively). We continue to emphasize the importance and utility of interval estimations (over point estimates). We then move on to discuss experimental methodologies useful for steady-state simulations, particularly related to eliminating estimator bias and reducing computational time.
[ due to an instructor error, the lecture from 2021-11-09 was not recorded, and the archived 2020-11-010lecture is re-used here instead ]
In this lecture, we continue to introduce terminating and non-terminating systems and difference methods for estimating performance from simulation models of them (using transient and steady-state simulations). This involves a description of various types of point estimators (mean and quantile) as well as related interval estimators (confidence intervals and prediction intervals, as well as the relationship to standard error of the mean (SEM)). We start to discuss issues involving making inferences from pseudo-replicated within-replication samples versus across-replication samples (which are independent and often normally distributed). We will continue this in the next lecture, as we start focusing more on steady-state simulations of non-terminating systems.
[ For some strange reason, the in-room video camera was not recorded as the speaker view despite apparently working during the class. Consequently, only the slide view is shown. ]
In this lecture, we review the fundamental tradeoffs in hypothesis testing and the concrete origins of the assumptions in both the t-test and Chi-square test. We also discuss parametric and non-parametric statistics (including exact and non-exact tests) and how non-parametric, exact statistics like the Kolmogorov–Smirnov test are derived. This culminates in a discussion of the multiple comparisons (MC) problem and the Bonferroni correction as well as alternative tests (such as a MANOVA or an ANOVA with post hoc test) that have more statistical power than the Bonferroni correction. We close with an introduction to performance inference from simulation, which we will continue discussing in the next 3 lectures.
