Little Apps: Proportions and zero/one variables

Teaching with the proportions littleapp

We’re going to start with basic levels and then increase in sophistication as we go. For an introductory course, you probably want to make each level its own topic in your course, covering the various topics as the course moves on. For instance, “What’s a proportion?” might be appropriate when you first introduce a proportion, “Confidence intervals” a couple of weeks later, quantitative explanatory variables at the end of the semester.

Sampling variation

The set-up is the same as previously.

There are two new controls to introduce. The first is the “Take new sample” button. Each time it is pressed, a new random sample from the entire NHANES data is displayed. Set the sample size to 50 and press the new-sample button many times. The proportion will jump around each time you press.

Now check the “Retain trials” box. This will cause the sample shown in dots to stay put, but will put up the proportion of a new sample each time the new-sample button is pressed. Press it many times. This gives a visual impression of the range of sampling variation.

Note that for small sample sizes, e.g. n=20, the proportion always falls at a discrete level: 0/20, 1/20, 2/20, and so one. This leads to overplotting in the horizontal lines. The dots to the side of the trials let you see how many trials produced each level.

Run sampling trials for large n and for small n. Note that the range of the trials is larger for smaller n.

Using a categorical explanatory variable

Now we’ll use the app with an actual explanatory variable. Pick one that’s categorical, e.g. gender, depressed, home ownership.

For small sample size, go through the counting exercise. You’ll have to count both the top and bottom levels in order to find the proportion for each group.

Try a bunch of different response and explanatory variables. Make sure to interpret the graphs in everyday terms. Does it look like the fraction of people above the poverty level is different for people who own or rent their home? Might it be that poor people can’t afford to buy a home? Or perhaps people who are less settled tend to be renters, and they also are more likely to be poor. Tell a story. Ask the students to tell a story that matches the pattern that they see.

After looking at several combinations of response and categorical explanatory variables, ask the question, “How do we know that this pattern is just an accident of sampling? Might another sample have produced a very different result?” For large and small sample sizes, press the new-sample button many times to get an idea of how much sampling variation there is. Check the retain-trials box so that each sample shows up as new lines in the graph. You can call the collection of lines the sampling distribution. Introduce the (pretty good) rule of thumb: if two levels of the explanatory variable have sampling distributions that have little or no overlap, you can be pretty sure that they data give good reason to think that there is a difference in proportion in the world from which the samples were taken. Sometimes for small n the sampling distributions do overlap while for large n they do not. This tells us that there really is a difference in proportion in the world² from which the samples were taken, but that with small n we don’t have enough data to justify that conclusion. Collecting samples with large n makes it easier to see a difference if there is one.

Confidence intervals

It’s easy to carry out sampling trials on the computer, but behind the scenes that’s because we already have a large sample from which we’re pulling subsamples. In real work we don’t have this large sample, just the sample at hand that we collected with great expense and labor. Without access to the sampling distribution itself, we need something to stand in for it.

At this point, where you go depends on whether your students have yet seen confidence intervals in any form and whether they’ve seen resampling. Turn on the confidence interval display. One approach is to generate the sampling distribution and show that it’s roughly the same width as the confidence interval. Another is to use resampling instead of sampling. Another is just to say that there is a way to know how precise our estimate of the proportion is.

Show that confidence intervals tend to get smaller as sample sizes get larger.

Quantitative explanatory variables

Some faculty may be unfamiliar with an approach to confidence intervals not grounded in \(p_0\), \(p_1\), \(p_2\), and \(\sqrt{p (1 - p) / n}\). These are part of an apparatus for statistical inference, admittedly one that is commonly found in introductory textbooks, but which is just one of the ways of doing things. A more general framework for statistical inference, regression, subsumes the textbook apparatus and enables us to have much more flexibility in the choice of explanatory variables and the incorporation of covariates.

In particular, regression enables you to use a quantitative variable (e.g. height, age) as an explanatory variable when examining proportions. The proportion indicated at any given level of the explanatory variable (e.g. a height of 170 cm) is more-or-less the proportion we would get if we took out the cases near that level (you can point to which dots are those cases) and calculated the proportion in the usual way: the number at the top level divided by the number at the bottom level.

If you’ve already talked about linear regression, you can be more precise in describing how the line is found: regression of the zero-one response variable against the explanatory variable.

As always, show several different combinations of response and explanatory variable. Some good examples are above poverty level versus height or age, or high-blood pressure versus age or smoking status. Ask the students to interpret the slope of the fitted line in everyday terms: “taller people tend to have more …”, or “there’s no evident relationship between age and …”

And, of course, turn on “retain trials” and show the sampling or resampling distribution at large and small sample sizes. For smaller samples, it can be harder to make a definitive statement about what is the relationship between the response and explanatory variables.

You can turn on the confidence intervals. Any line you can draw that stays within the band of the confidence interval is a plausible match to the data. If there are such lines with different stories, you don’t have enough data to distinguish between the stories: either or neither might hold in the world.

Proportions must be between zero and one.

As you go through various combinations of response and explanatory variables, you will find instances where the confidence interval extends below zero or above one. You’ll also see that when using a quantitative explanatory variable, the line showing proportions might itself go outside the bounds zero to one. This is a mathematical absurdity, since proportions must be between zero and one.

The problem is not in our data but in our methods for calculating confidence intervals and regression models. Using logistic regression solves the problem.

I think it’s unlikely that many instructors will want to teach extensively about logistic regression. And it’s just one of a set of modeling techniques that fall under the name “machine learning.” Some machine-learning methods are mathematically sophisticated (e.g. support vector machines), some are pretty simple (e.g. k-nearest neighbors). Logistic regression is in the middle.

I think it’s reasonable to expect that machine learning will become even more widely used in the modeling and interpretation of data. Here’s a short explanation of logistic regression that is oriented toward instructors with substantial mathematics training. It’s probably not for your students.

1. A probability p must fall in the range between zero and one.
2. For each value of p, there is a corresponding value p / (1-p). This representation is called an odds. (Odds are routinely used in gambling. They simplify calculating a payout that won’t systematically cause the “house” to lose money.) Odds can range, of course, from zero to \(\infty\).
3. Now take the logarithm of the odds, called the log odds. This is a number that can range from \(-\infty\) to \(\infty\).
4. In logistic regression, one fits a model whose output is in terms of log odds. Whatever that model might be, translating the log-odds output to a probability will necessarily produce a value between zero and one.
5. Fitting a logistic regression model isn’t done with the linear algebraic methods of linear regression. Instead, it is done by systematic search over the model parameter space. Each point in that space generates a probability for each of the cases in the data; put the explanatory variables into the model, calculate the log odds using the given parameters, then translate to probability. The likelihood of the model is the probability of the actual data given the point in parameter space. The model is fit by maximizing this likelihood using numerical optimization.

There’s much more to say about logistic regression and its interpretation. It’s very widely used in clinical research because it enables quantitative covariates to be included in a model in a natural way. (Example: What is the difference in recovery rates for the treatment and placebo groups, taking into account age, sex, blood pressure at entry to the study, …)