# Probability and Statistics: The Science of Uncertainty

## by Michael J. Evans and Jeffrey S. Rosenthal

This undergraduate-level probability and statistics textbook was published by W.H. Freeman in 2003, with a second edition in 2010 (still available from e.g. amazon.ca with solutions manual, or from amazon.com with solutions manual). However:

The entire book is now available for free in pdf format (including errata).

"The authors do an admirable job supplementing theoretical results with numerical examples and potential simulation studies. ... Ample homework problems are provided. ... The authors' organization is logical, with essential ideas from probability placed at the beginning followed by one-sample inference and then regression problems. The authors succeed in unifying a number of seemingly disparate ideas. ... The exposition is clear and uncluttered. ... In addition to an ambitious topic list and numerous examples throughout, the authors provide off-the-cuff remarks to help the reader assimilate information. ... This is a quality text."
-- Dave H. Annis, in The American Statistician 59(3), August 2005.

## Preface

This book is an introductory text on probability and statistics. The book is targeted at students who have studied one year of calculus at the university level and are seeking an introduction to probability and statistics that has mathematical content. Where possible, we provide mathematical details, and it is expected that students are seeking to gain some mastery over these, as well as learn how to conduct data analyses. All of the usual methodologies covered in a typical introductory course are introduced, as well as some of the theory that serves as their justification.

The text can be used with or without a statistical computer package. It is our opinion that students should see the importance of various computational techniques in applications, and the book attempts to do this. Accordingly, we feel that computational aspects of the subject, such as Monte Carlo, should be covered, even if a statistical package is not used. All of the computations in this text were carried out using Minitab. Minitab is a suitable computational platform to accompany the text, but others could be used. There is a Computations appendix that contains the Minitab code for those computations that are slightly involved (for example, if looping is required); these can be used by students as templates for their own calculations. If a software package like Minitab is used with the course, then no programming is required by the students to do problems.

We have organized the exercises in the book into groups, as an aid to users. Exercises are suitable for all students and are there to give practice in applying the concepts discussed in a particular section. Problems require greater understanding, and a student can expect to spend more thinking time on these. If a problem is marked (MV), then it will require some facility with multivariable calculus beyond the first calculus course, although these problems are not necessarily hard. Challenges are problems that most students will find difficult. The Challenges are only for students who have no difficulty with the Exercises and the Problems. There are also Computer Exercises and Computer Problems, where it is expected that students will make use of a statistical package in deriving solutions.

We have also included a number of Discussion Topics that are designed to promote critical thinking in students. Throughout the book we try to point students beyond the mastery of technicalities to think of the subject in a larger frame of reference. It is important that students acquire a sound mathematical foundation in the basic techniques of probability and statistics. We believe that this book will help students accomplish this. Ultimately, however, these subjects are applied in real-world contexts, so it is equally important that students understand how to go about their application and understand what issues arise. Often there are no right answers to Discussion Topics. Their purpose is to get a student thinking about the subject matter. If these were to be used for evaluation, then they would be answered in essay format and graded on the maturity the student showed with respect to the issues involved. Discussion Topics are probably most suitable for smaller classes, but there will also be benefit to students if these are simply read over and thought about.

Some sections of the book are labelled Advanced. This material is aimed at students who are more mathematically mature (for example, they are taking, or have taken, a second course in calculus). All of the Advanced material can be skipped, with no loss of continuity, by an instructor who wishes to do so. In particular, the final chapter of the text is labelled Advanced and would only be taught in a high-level introductory course aimed at specialists. Also, many proofs are put in a final section of each chapter, labelled Further Proofs (Advanced). An instructor can choose which (if any) of these proofs they wish to present to their students. As such, we feel that the material in the text is presented in a flexible way that allows the instructor to find an appropriate level for the students they are teaching. There is a Mathematical Background appendix that reviews some mathematical concepts that students may be rusty on from a first course in calculus, as well as brief introductions to partial derivatives, double integrals, etc.

Chapter 1 introduces the probability model and provides motivation for the study of probability. The basic properties of a probability measure are developed.

Chapter 2 deals with discrete, continuous, joint distributions, and the effects of a change of variable. The multivariate change of variable is developed in an Advanced section. The topic of simulating from a probability distribution is introduced in this chapter.

Chapter 3 introduces expectation. The probability-generating function is introduced as well as the moments and the moment-generating function of a random variable. This chapter develops some of the major inequalities used in probability. There is a section available on characteristic functions as an Advanced topic.

Chapter 4 deals with sampling distributions and limits. Convergence in probability, convergence with probability 1, the weak and strong laws of large numbers, convergence in distribution, and the central limit theorem are all introduced along with various applications such as Monte Carlo. The normal distribution theory, necessary for many statistical applications, is also dealt with here.

As mentioned, Chapters 1 through 4 include material on Monte Carlo techniques. Simulation is a key aspect of the application of probability theory, and it is our view that its teaching should be integrated with the theory right from the start. This reveals the power of probability to solve real-world problems and helps convince students that it is far more than just an interesting mathematical theory. No practitioner divorces himself from the theory when using the computer for computations or vice versa. We believe this is a more modern way of teaching the subject. This material can be skipped, however, if an instructor doesn't agree with this, or feels they do not have enough time to cover it effectively.

Chapter 5 is an introduction to statistical inference. For the most part this is concerned with laying the groundwork for the development of more formal methodology in later chapters. So practical issues -- such as proper data collection, presenting data via graphical techniques, and informal inference methods like descriptive statistics -- are discussed here.

Chapter 6 deals with many of the standard methods of inference for one-sample problems. The theoretical justification for these methods is developed primarily through the likelihood function, but the treatment is still fairly informal. Basic methods of inference, such as the standard error of an estimate, confidence intervals, and P-values, are introduced. There is also a section devoted to distribution-free (nonparametric) methods like the bootstrap.

Chapter 7 involves many of the same problems discussed in Chapter 6 but now from a Bayesian perspective. The point of view adopted here is not that Bayesian methods are better or, for that matter, worse than those of Chapter 6. Rather, we take the view that Bayesian methods arise naturally when the statistician adds another ingredient -- the prior -- to the model. The appropriateness of this, or the sampling model for the data, is resolved through the model-checking methods of Chapter 9. It is not our intention to have students adopt a particular philosophy. Rather, the text introduces students to a broad spectrum of statistical thinking.

Subsequent chapters deal with both frequentist and Bayesian approaches to the various problems discussed. The Bayesian material is in clearly labelled sections and can be skipped with no loss of continuity, if so desired. It has become apparent in recent years, however, that Bayesian methodology is widely used in applications. As such, we feel that it is important for students to be exposed to this, as well as to the frequentist approaches, early in their statistical education.

Chapter 8 deals with the traditional optimality justifications offered for some statistical inferences. In particular, some aspects of optimal unbiased estimation and the Neyman-Pearson theorem are discussed in this chapter. There is also a brief introduction to decision theory. This chapter is more formal and mathematical than Chapters 5, 6, and 7, and it can be skipped, with no loss of continuity, if an instructor wants to emphasize methods and applications.

Chapter 9 is on model checking. We placed model checking in a separate chapter to emphasize its importance in applications. In practice, model checking is the way statisticians justify the methods of inference they use. So this is a very important topic.

Chapter 10 is concerned with the statistical analysis of relationships among variables. This includes material on simple linear and multiple regression, ANOVA, the design of experiments, and contingency tables. The emphasis in this chapter is on applications.

Chapter 11 is concerned with stochastic processes. In particular, Markov chains and Markov chain Monte Carlo are covered in this chapter, as are Brownian motion and its relevance to finance. Fairly sophisticated topics are introduced, but the treatment is entirely elementary. Chapter 11 depends only on the material in Chapters 1 through 4.

A one-semester course on probability would cover Chapters 1-4 and perhaps some of Chapter 11. A one-semester, follow-up course on statistics would cover Chapters 5-7 and 9-10. Chapter 8 is not necessary, but some parts, such as the theory of unbiased estimation and optimal testing, are suitable for a more theoretical course.

A basic two-semester course in probability and statistics would cover Chapters 1-6 and 9-10. Such a course covers all the traditional topics, including basic probability theory, basic statistical inference concepts, and the usual introductory applied statistics topics. To cover the entire book would take three semesters, which could be organized in a variety of ways.

The Advanced sections can be skipped or included, depending on the level of the students, with no loss of continuity. A similar comment applies to Chapters 7, 8, and 11.

Students who have already taken an introductory noncalculus-based, applied statistics course will also benefit from a course based on this text. While similar topics are covered, they are presented with more depth and rigor here. For example, Introduction to the Practice of Statistics, Fourth Edition, by D. Moore and G. McCabe (W. H. Freeman, 2003) is an excellent text, and we feel that this book will serve as the basis for a good follow-up course.

Many thanks to the reviewers and class testers for their comments: Michelle Baillargeon (McMaster University), Lisa A. Bloomer (Middle Tennessee State University), Eugene Demidenko (Dartmouth College), Robert P. Dobrow (Carleton College), John Ferdinands (Calvin College), Soledad A. Fernandez (The Ohio State University), Dr. Paramjit Gill (Okanagan University College), Ellen Gundlach (Purdue University), Paul Gustafson (University of British Columbia), Jan Hannig (Colorado State University), Susan Herring (Sonoma State University), George F. Hilton, Ph.D., (Pacific Union College), Paul Joyce (University of Idaho), Hubert Lilliefors (George Washington University), Phil McDonnough (University of Toronto), Julia Morton (Nipissing University), Randall H. Rieger (West Chester University), Robert L. Schaefer (Miami University), Osnat Stramer (University of Iowa), Tim B. Swartz (Simon Fraser University), Glen Takahara (Queen's University), Robert D. Thompson (Hunter College), Dr. David C. Vaughan (Wilfrid Laurier University), Joseph J. Walker (Georgia State University), Dongfeng Wu (Mississippi State University), Yuehua Wu (York University), Nicholas Zaino (University of Rochester).

The authors would also like to thank many who have assisted in the development of this project. In particular our colleagues and students at the University of Toronto have been very supportive. Hadas Moshonov, Aysha Hashim, and Natalia Cheredeko of the University of Toronto helped in many ways. A number of the data sets in Chapter 10 have been used in courses at the University of Toronto for many years and were, we believe, compiled through the work of the late Professor Daniel B. DeLury. Professor David Moore of Purdue University was of assistance in providing several of the tables at the back of the text. Patrick Farace, Chris Spavins, and Danielle Swearengin of W. H. Freeman provided much support and encouragement. Our families helped us with their patience and care while we worked at what seemed at times an unending task; many thanks to Rosemary and Heather Evans and Margaret Fulford.

```  Preface  x
1 Probability Models  1
1.1 Probability: A Measure of Uncertainty  1
1.1.1 Why Do We Need Probability Theory?  2
1.2 Probability Models  4
1.2.1 Venn Diagrams and Subsets  7
1.3 Properties of Probability Models  10
1.4 Uniform Probability on Finite Spaces  13
1.4.1 Combinatorial Principles  14
1.5 Conditional Probability and Independence  19
1.5.1 Conditional Probability  20
1.5.2 Independence of Events  23
1.6 Continuity of P   28
2 Random Variables and Distributions  33
2.1 Random Variables  33
2.2 Distributions of Random Variables  37
2.3 Discrete Distributions  40
2.3.1 Important Discrete Distributions  41
2.4 Continuous Distributions  50
2.4.1 Important Absolutely Continuous Distributions  52
2.5 Cumulative Distribution Functions  61
2.5.1 Properties of Distribution Functions  62
2.5.2 Cdfs of Discrete Distributions  62
2.5.3 Cdfs of Absolutely Continuous Distributions  64
2.5.4 Mixture Distributions  66
2.5.5 Distributions Neither Discrete Nor Continuous  69
2.6 One-Dimensional Change of Variable   72
2.6.1 The Discrete Case  72
2.6.2 The Continuous Case  73
2.7 Joint Distributions  77
2.7.1 Joint Cumulative Distribution Functions  77
2.7.2 Marginal Distributions  79
2.7.3 Joint Probability Functions  80
2.7.4 Joint Density Functions  82
2.8 Conditioning and Independence  89
2.8.1 Conditioning on Discrete Random Variables  90
2.8.2 Conditioning on Continuous Random Variables  91
2.8.3 Independence of Random Variables  93
2.8.4 Order Statistics  99
2.9 Multidimensional Change of Variable  104
2.9.1 The Discrete Case  104
2.9.2 The Continuous Case (Advanced)  105
2.9.3 Convolution  108
2.10 Simulating Probability Distributions  111
2.10.1 Simulating Discrete Distributions  112
2.10.2 Simulating Continuous Distributions  114
3 Expectation  123
3.1 The Discrete Case  123
3.2 The Absolutely Continuous Case  135
3.3 Variance, Covariance, and Correlation  142
3.4 Generating Functions  154
3.5 Conditional Expectation  166
3.5.1 Discrete Case  166
3.5.2 Absolutely Continuous Case  168
3.5.3 Double Expectations  169
3.6 Inequalities  176
4 Sampling Distributions and Limits  189
4.1 Sampling Distributions  190
4.2 Convergence in Probability  193
4.2.1 The Weak Law of Large Numbers  195
4.3 Convergence with Probability 1  198
4.3.1 The Strong Law of Large Numbers  200
4.4 Convergence in Distribution  202
4.4.1 The Central Limit Theorem  204
4.4.2 The Central Limit Theorem and Assessing Error  209
4.5 Monte Carlo Approximations  213
4.6 Normal Distribution Theory   222
4.6.1 The Chi-Squared Distribution  223
4.6.2 The t Distribution  226
4.6.3 The F Distribution  227
5 Statistical Inference  239
5.1 Why Do We Need Statistics?  239
5.2 Inference Using a Probability Model  244
5.3 Statistical Models  247
5.4 Data Collection  254
5.4.1 Finite Populations  254
5.4.2 Simple Random Sampling  256
5.4.3 Histograms  259
5.4.4 Survey Sampling  261
5.5 Some Basic Inferences  266
5.5.1 Descriptive Statistics  267
5.5.2 Plotting Data  271
5.5.3 Types of Inference  273
6 Likelihood Inference  281
6.1 The Likelihood Function  281
6.1.1 Sufficient Statistics  286
6.2 Maximum Likelihood Estimation  291
6.2.1 The Multidimensional Case (Advanced)  299
6.3 Inferences Based on the MLE  302
6.3.1 Standard Errors and Bias  303
6.3.2 Confidence Intervals  307
6.3.3 Testing Hypotheses and P-Values  313
6.3.4 Sample Size Calculations: Confidence Intervals  320
6.3.5 Sample Size Calculations: Power  322
6.4 Distribution-Free Methods  329
6.4.1 Method of Moments  330
6.4.2 Bootstrapping  331
6.4.3 The Sign Statistic and Inferences about Quantiles  335
6.5 Large Sample Behavior of the MLE (Advanced)  342
7 Bayesian Inference  351
7.1 The Prior and Posterior Distributions  352
7.2 Inferences Based on the Posterior  361
7.2.1 Estimation  364
7.2.2 Credible Intervals  368
7.2.3 Hypothesis Testing and Bayes Factors  371
7.2.4 Prediction  377
7.3 Bayesian Computations  383
7.3.1 Asymptotic Normality of the Posterior  383
7.3.2 Sampling from the Posterior  383
7.3.3 Sampling from the Posterior Using Gibbs Sampling (Advanced)  389
7.4 Choosing Priors  397
8 Optimal Inferences  405
8.1 Optimal Unbiased Estimation  405
8.1.1 The Cramer-Rao Inequality (Advanced)  412
8.2 Optimal Hypothesis Testing  418
8.2.1 Likelihood Ratio Tests (Advanced)  426
8.3 Optimal Bayesian Inferences  430
9 Model Checking  449
9.1 Checking the Sampling Model  449
9.1.1 Residual and Probability Plots  456
9.1.2 The Chi-Squared Goodness of Fit Test  460
9.1.3 Prediction and Cross-Validation  465
9.1.4 What Do We Do When a Model Fails?  466
9.2 Checking the Bayesian Model  471
9.3 The Problem with Multiple Checks  477
10 Relationships Among Variables  479
10.1 Related Variables  480
10.1.1 Cause-Effect Relationships and Experiments  483
10.1.2 Design of Experiments  486
10.2 Categorical Response and Predictors  494
10.2.1 Random Predictor  494
10.2.2 Deterministic Predictor  497
10.2.3 Bayesian Formulation  499
10.3 Quantitative Response and Predictors  505
10.3.1 The Method of Least Squares  505
10.3.2 The Simple Linear Regression Model  507
10.3.3 Bayesian Simple Linear Model (Advanced)  521
10.3.4 The Multiple Linear Regression Model (Advanced)  525
10.4 Quantitative Response and Categorical Predictors  543
10.4.1 One Categorical Predictor (One-Way ANOVA)   543
10.4.2 Repeated Measures (Paired Comparisons)  549
10.4.3 Two Categorical Predictors (Two-Way ANOVA)   552
10.4.4 Randomized Blocks  559
10.4.5 One Categorical and One Quantitative Predictor  560
10.5 Categorical Response and Quantitative Predictors  568
11 Advanced Topic -- Stochastic Processes  579
11.1 Simple Random Walk  579
11.1.1 The Distribution of the Fortune  580
11.1.2 The Gambler's Ruin Problem  582
11.2 Markov Chains  586
11.2.1 Examples of Markov Chains  587
11.2.2 Computing with Markov Chains  590
11.2.3 Stationary Distributions  593
11.2.4 Markov Chain Limit Theorem  597
11.3 Markov Chain Monte Carlo   604
11.3.1 The Metropolis-Hastings Algorithm  607
11.3.2 The Gibbs Sampler  610
11.4 Martingales  613
11.4.1 Definition of a Martingale  613
11.4.2 Expected Values  615
11.4.3 Stopping Times  616
11.5 Brownian Motion  620
11.5.1 Faster and Faster Random Walks  621
11.5.2 Brownian Motion as a Limit  622
11.5.3 Diffusions and Stock Prices  625
11.6 Poisson Processes  629
11.7 Further Proofs  631
A Mathematical Background  639
A.1 Derivatives  639
A.2 Integrals  640
A.3 Infinite Series  641
A.4 Matrix Multiplication  642
A.5 Partial Derivatives  642
A.6 Multivariable Integrals  643
A.6.1 Nonrectangular Regions  644
B Computations  647
C Common Distributions  653
C.1 Discrete Distributions  653
C.2 Absolutely Continuous Distributions  654
D Tables  657
D.1 Random Numbers  658
D.2 Standard Normal Cdf  660
D.3 Chi-Squared Distribution Quantiles  661
D.4 t Distribution Quantiles  662
D.5 F Distribution Quantiles  663
D.6 Binomial Distribution Probabilities  672
Index  677
```