Investigation 1.16: Literary Digest

Section 4.7 Investigation 1.16: Literary Digest

Literary Digest was a well-respected political magazine founded in 1890. Using sampling, they correctly predicted the presidential outcomes from 1916–1932. In 1936, they conducted the most extensive (to that date) public opinion poll in history. They mailed out questionnaires (postcards) to over 10 million people (about one-fourth of voters), hand addressing more than a quarter million postcards per day) whose names and addresses they obtained from club rosters, city directories, and (mostly) vehicle registration lists and telephone books.

🔗

Literary Digest Poll images

🔗

Based on almost 2.4 million responses (2,376,523), the Literary Digest predicted that 54% of voters would vote for Republican Alf Landon (then governor of Kansas) in the upcoming presidential election, and only 41% would vote for Democrat Franklin Roosevelt (the incumbent).

🔗

Literary Digest Poll Results:

🔗

Checkpoint 4.7.1. Identify study components.

Identify the variable and population of interest, the sample, and the sampling frame in this study. Also define the parameter and statistic, in words and symbols, and indicate any values that you know.

🔗

Variable: Type:

🔗

Population of interest: Sample:

Sampling frame:

Parameter:

Statistic:

Solution.

Variable: Presidential vote preference; Type: Categorical (binary)

🔗

Population of interest: All U.S. voters in 1936

🔗

Sample: 2.4 million respondents

🔗

Sampling frame: Telephone books and vehicle registration lists

🔗

Parameter: \(\pi\) = proportion of all U.S. voters who actually voted for Alf Landon

🔗

Statistic: \(\hat{p} = 0.57\) = proportion of sample who responded they would vote for Landon

🔗

Checkpoint 4.7.2. Calculate sample statistic.

Why is \(1293669/(1293669 + 972897) = 0.57\) not equal to 0.54?

🔗

Solution.

The 0.57 represents the proportion among just Landon and Roosevelt (the two main candidates), while 0.54 includes other candidates like William Lemke (Union Party) in the denominator.

🔗

Have you ever heard of Alf Landon? He lost. By a landslide. Incumbent Democrat Franklin Roosevelt won the election, carrying 60.8% of the popular vote to Landon’s 36.5%.

🔗

Checkpoint 4.7.3. Explain polling error.

Give two plausible explanations why the Literary Digest prediction was so much in error. In particular, talk about the direction of the bias – why was this sampling method vulnerable to producing an overestimate of the parameter?

🔗

Solution.

1. Oversampled Republicans because Republicans tended to have more money for cars, phones, etc. in 1936 (bad sampling frame)

🔗

2. Voluntary response bias. More likely to hear from unhappy people (vs. supporting the incumbent)

🔗

Still, a 24% response rate is much higher than current polls and this was the Digest’s largest poll yet, shouldn’t this improve their prediction? How are current polls with smaller response rates and smaller sample sizes able to be more accurate? One reason suggested for the error in 1936 was that the Digest had consistently been having more Republican voters responding to their straw vote. Does the data support this argument?

🔗

Open the LitDigest1936.xlsx file in Excel or Google Sheets. This contains the raw counts for the three main candidates (including William Lemke, Union Party), as well as an overall total number of straw votes cast in each state.

🔗

Checkpoint 4.7.4. Calculate 1936 poll proportion.

For these 3 candidates, what percentage of the poll respondents said they would vote for Republican Landon in the 1936 election?

🔗

Hint.

Set up a column formula in row 52, using columns C and AA.

🔗

Solution.

C52/AA52 = 0.544 (approximately 54.4%)

🔗

Checkpoint 4.7.5. Calculate 1932 voting breakdown.

Set up a formula for determining the number of respondents to the 1936 poll who said they voted in 1932 for either the Republican, Democratic, or Socialist or Other candidate. What proportion of these voted for the Republican candidate?

🔗

Sum: Proportion:

🔗

Hint.

Use columns D-G, L-O, and T-W.

🔗

Solution.

Formula: (c52 + l52 + t52)/(sum(c52:g52)+sum(l52:o52)+sum(t52:w52))

🔗

Proportion: 0.507 (approximately 50.7% voted Republican in 1932)

🔗

Checkpoint 4.7.6. Compare to actual 1932 results.

Now find the actual 1932 election results, what proportion of voters voted for the Republican (Hoover) candidate (among the three major candidates)? Did the Literary Digest poll have too many or too few Republican voters? How could we account for this in our estimates of vote count?

🔗

Solution.

15,761,254/(15,761,254 + 22,821,277 + 73,158) = 0.399

🔗

The proportion who claimed to vote Republican in the survey (0.507) is noticeably larger than the proportion that actually voted Republican (0.399). This supports the suspicion that the survey overrepresented Republicans compared to the voting population.

🔗

Post Stratification.

In the 1936 poll, about 51% of respondents said they voted Republican in 1932, but only about 40% of the actual voters in 1932 voted Republican, a ratio of 0.78. So the Digest’s sample appear to overrepresent the Republican voters. Similarly, persons who said they voted Democrat or Other in the 1932 election were unrepresented in the 1936 poll (ratios: 1.197, 2.23 respectively). (Lohr and Brick, 2017, also estimated the ratio for the non-voters and missing to be 1.1275 for Democrats and 0.871 for Republicans.)

🔗

1932 Vote	Republican	Democrat	Soc./Other	Non-voters	Missing
Ratio	0.782	1.197	2.228	D: 1.1275 🔗 R: 0.871 🔗	D: 1.1275 🔗 R: 0.871 🔗

This means we want to lower the number of Republican votes in the Digest poll by multiplying by 0.78, to adjust for the overrepresentation in the sample, and increase the number of Democrat votes in the poll by multiplying by 1.197 and so forth.

🔗

Checkpoint 4.7.7. Adjust Landon votes.

Start with the Landon voters in the Digest poll, using the breakdown by how they voted in 1932, adjust the counts using the above ratios. What is the total number of Landon votes?

🔗

Hint.

Set up a formula using these weights and the breakdown of the planned Landon voters by the 1932 vote, columns D-I, to find a new count. Using 0.871 for non-voters and missing.

🔗

Solution.

Using 0.78 as the first multiplier (so watch for rounding discrepancies):

🔗

717775.5 + 299320.623 + 12541.412 + 1838.1 + 53412.333 + 48434.568 = 1,133,322.536

🔗

Checkpoint 4.7.8. Adjust Roosevelt votes.

Repeat for Roosevelt (columns L-Q, 1.1275 for non-voters, missing) and report the nubmer of Roosevelt votes.

🔗

Hint.

Use columns L-Q, with 1.1275 for non-voters and missing.

🔗

Solution.

111494.76 + 854890.218 + 41039.76 + 1608.616 + 64617.025 + 44320.8975 = 1,117,971.277

🔗

Checkpoint 4.7.9. Calculate adjusted percentage.

Between these two candidates, what is the adjusted percentage voting for Roosevelt? Is this larger or smaller than the original Literary Digest prediction?

🔗

Solution.

Proportion (for these two) voting Roosevelt is now about 0.497 (49.7%), up from 0.429 for the 2 candidates (though still below half).

🔗

This is larger than the original Literary Digest prediction.

🔗

Although this is still much lower than the actual vote share for Roosevelt between these two candidates (62%), you can see how this process tries to account for the sampling bias in the original poll. There are still a lot of assumptions, e.g., respondents accurately reported their 1932 vote, and non-respondents to the LD survey had the same relationship between the 1932 and 1936 votes as the respondents. But if you apply this technique to the individual states, 10 states change from Landon to Roosevelt, including California and New York, and Roosevelt is predicted to win 26 states (276 electoral votes), rather than 16 states (161 electoral votes).

🔗

Comparison chart 1 for different correction methods — Figure 4.7.10. Comparison of correction methods
🔗

So using this one other piece of information would have at least predicted the correct outcome, though still way underestimating the margin of victory. More complicated weighting schemes further adjust based on the size of the error in the 1928 LD poll or use a regression model based on the previous two elections. The Literary Digest also collected but did not publish data on postmarks of returned ballots and could have adjusted for rural/urban differences or county-level demographics from the 1930 census.

🔗

Below are state by state results for some different possible adjustments for the 1936 poll:

🔗

Actual state by state results
🔗

🔗
Using the relationship between predicted and actual votes from 1928 and 1932 to predict the results for 1936
🔗

🔗
Weighting the 1936 poll responses by the reported 1932 party breakdowns
🔗

🔗
Adding the LD error in the predicted Democrat proportion in 1932 to the 1936 LD prediction
🔗

🔗
Original LD predictions
🔗

🔗

🔗

Comparison chart 2 for correction methods — Figure 4.7.11.
🔗

Checkpoint 4.7.12. Compare correction methods.

Which correction method seems to help the most? How are you deciding?

🔗

Hint.

Look at which method’s distribution is most centered around the actual country proportion.

🔗

Solution.

In this case, the method using data from 1928 and 1932 seems to be the best as the center of the state by state proportions is at least centered around the country proportion, rather than most of the states being too low.

🔗

Discussion.

There were two main issues in the Literary Digest poll. One, the sampling frame did not include all members of the population of interest and in particular failed to include those that were poorer (and at that time likely to be Democrats). Second, the voluntary response nature of the poll implies the surveyors were more likely to hear from those unhappy with the status quo (incumbent candidate) or with more time on their hands to complete such surveys (e.g., retired folks), and even those more willing to pay for a stamp. Both of these probably point to an overrepresentation of Republicans. Bad sampling frames and voluntary response bias are perhaps the most common sources of sampling error. By the way, a fledgling pollster of the time, George Gallup, actually bet that he would predict the percentages more accurately. Not only did he correctly predict the Digest result with only 3,000 respondents, he also correctly predicted a Roosevelt victory! The issues with the Literary Digest poll were evident in earlier elections (overpredicting the popular vote for the winner in 1924 and 1928, but the results were so one-sided, the bias didn’t matter), and even though they could have weighted the results, they chose not to and let readers "draw their own conclusions." But Lohr and Brick (2017) point out that the largest issue was probably failing to accurately assess the uncertainty in their estimates.

🔗

References.

Lohr & Brick (2007). "Roosevelt Predicted to Win: Revisiting the 1936 Literary Digest Poll," Stat Polit Pol, 8(1): 65-85.
🔗

🔗
Lusinchi, D. (2012). "’President’ Landon and the 1936 Literary Digest Poll: Were Automobile and Telephone Owners to Blame?" Social Science History, 36:23–54.
🔗

🔗
Robinson, C. (1932). Straw Votes. New York, NY: Columbia University Press.
🔗

🔗

🔗

Subsection 4.7.1 Practice Problem 1.16A

In the mid-1980s, Dr. Shere Hite, actress and writer, and subject of a 2023 documentary, undertook a survey of women’s attitudes toward relationships, love, and sex by distributing 100,000 questionnaires in women’s groups. Of the 4500 who returned the questionnaire, 96% said that they gave more emotional support than they received from their husbands or boyfriends. An ABC News/Washington Post poll surveyed a random sample of 767 women, finding that 44% claimed to give more emotional support than they received.

🔗

Checkpoint 4.7.13.

Which of the following is the "parameter" of interest in these studies?

🔗

Those who said they gave more emotional support
All American women
The 4500 women who responded
The proportion who said they received less emotional support
Other

🔗

Checkpoint 4.7.14.

Which poll’s results do you think are more representative of the population of all American women? Explain.

🔗

Subsection 4.7.2 Practice Problem 1.16B

An article published in the June 6, 2006 issue of the journal Pediatrics describes a survey on the topic of college students intentionally injuring themselves. Researchers invited 8300 undergraduate and graduate students at Cornell University and Princeton University to participate in the survey. A total of 2875 students responded, with 17% of them saying that they have purposefully injured themselves. Suppose we are interested in the proportion of self-injuries in the population of all college students.

🔗

Checkpoint 4.7.15.

Identify the observational units and variable in this study. Also classify the variable as categorical (also binary?) or quantitative.

🔗

Checkpoint 4.7.16.

Identify the parameter and statistic. Also indicate appropriate symbols.

🔗

Checkpoint 4.7.17.

Do you think it likely that this sample is representative of the population of all college students in the world? What about all college students in the U.S.? Explain.

🔗

Checkpoint 4.7.18.

Describe a population to which you might be willing to generalize the results.

🔗

Checkpoint 4.7.19.

For which of the following variables would you suspect this sample would be representative of the population of all U.S. college students? Justify your answer.

Favorite TV show:

Height:

Parental education:

Eye color:

Subsection 4.7.3 Practice Problem 1.16C

History Matters includes a report on the vote totals. Examine the data provided on that webpage.

🔗

Checkpoint 4.7.20.

If you check the totals, they don’t quite match up. Can you find the data entry errors?

🔗

Hint.

Do any numerical values look suspicious to you? Do any states behave unusually?

🔗

Checkpoint 4.7.21.

Notice the State Unknown row. If we don’t know the state, how do we know the electoral count for those states… Based on the values given in that row, what do you think the counts for Landon and Roosevelt for individuals with unknown states actually were?

🔗

Subsection 4.7.4 Practice Problem 1.16D

Checkpoint 4.7.22.

Does your class constitute a random sample of students from your school? Explain why or why not.

🔗

Checkpoint 4.7.23.

Suggest a variable for which your class should not be considered a representative sample of all students at your school. Explain why not.

🔗

Checkpoint 4.7.24.

Suggest a variable for which it might be reasonable to consider your class to be representative of all students at your school. Justify your choice.

🔗

You have attempted of activities on this page.

🔗

Prev Top Next