This is an open access article distributed under the terms of the Creative Commons
Attribution License (In the first two decades of the 21st century, political polarization increased in many democratic countries, and this surge was especially notable in the United States (Boxell et al., 2021). Polarization can lead to the stratification of social interaction whereby citizens obtain more political information from like-minded sources than neutral or contrasting sources. This is particularly troubling when misinformation is amplified, including both out-of-context and intentionally fabricated statements. Such misinformation is resistant to corrective processes when maintained in a partisan bubble (Jerit & Barabas, 2012).
Despite these broad trends, citizens often remain interested in obtaining a range of information when making decisions to support candidates and policies. Being able to consider numerous sides makes these informed citizens more convincing when sharing views with others (Xu & Petty, 2022). Moreover, voters willing to engage with other politically dissimilar voters can, in turn, make informed decisions about voting and select representatives likely to work with politically dissimilar fellow representatives. A politically pluralistic public, by carefully considering whether to vote and whom to vote for, can push their representatives to reach cross-partisan solutions. An important question, then, is how open American electorates are to political pluralism, particularly across political divides.
We call an interest in obtaining and engaging with political information from a variety of the perspective, openness to political pluralism (OPP). People vary in OPP, with some being virtually closed to other viewpoints, and others willing to actively seek information from news and opinion sources that reflect different perspectives on an argument. In this article, we consider how to assess openness to political pluralism. We posit that this tendency can be found in persons regardless of their political orientation.
When assessing openness to political pluralism, one must recognize that engaging with persons and sources on the other side of the political spectrum can be difficult. Once a person has formed an opinion or had assumed a political identity, they perceive contradictory or inconsistent information as less credible (Metzger et al., 2020). This may lead to nonconscious attention and perceptual confirmation bias processes that make it difficult to process other points of view. To offset the challenges to deliberating possibly inconsistent information with their beliefs, people need to possess certain qualities. We consider four qualities that are of theoretical and practical interest.
One of these qualities is emotional tolerance. A notable aspect of current political polarization is that of “affective polarization”—the difference in affect felt for the in-party vs. out-party—which is driven by negative feelings for political opponents (Iyengar et al., 2019). Affective polarization is so extreme that dislike of the opposing party better predicts voting behaviors than love for one’s own party (Abramowitz & Webster, 2018). Consequently, it is crucial for people who want to be open to political pluralism to regulate negative affect when faced with challenging or inconsistent political information to their political opinions.
Not only do people need to have emotional tolerance, but they need to be able to acknowledge the fact that others may have different political opinions than themselves. We call this attribute intellectual tolerance. This corresponds to a component of intellectual humility, which has been used to predict willingness to change an opinion after talking to others (Krumrei-Mancuso & Newman, 2021).
A third attribute is non-rigidity. This attribute is related to a component of intellectual humility called “lack of intellectual overconfidence” (Krumrei-Mancuso & Rouse, 2016). People need to be non-rigid in their opinions and attitudes for them to be receptive to diverse political views.
Being emotionally tolerant, intellectually tolerant, and non-rigid all foster openness to opposing ideas, but they are passive—rather than proactive—attributes (i.e., do not mind political pluralism). We posit that, for someone to be truly high on openness to political pluralism, they need to have pluralistic motivation. That is, a willingness to proactively engage with politically dissimilar others. We suggest that these four sub-processes work together to make persons more open to political pluralism.
In sum, we view openness to political pluralism to be a combination of attitudes and motivations that are related to engaging in pluralistic dialogue in open societies where individuals can participate in democracy. An important question is how these come together to provide a coherent ordering of people with respect to the likelihood that they would engage in information gathering or direct discussions regarding divergent views. We do not conceptualize openness to political pluralism as simply a collection of constructs, or as a general construct that leads to the attributes and motivations we just discussed. Rather, we see pluralism openness as a unidimensional construct that complements the specific attributes. This conceptualization is consistent with what psychometricians call a bifactor model, and it is distinct from a second-order factor model (Chen et al., 2006). Figure 1 shows diagrams of these models, with the second-order model on the left and the bifactor model on the right (see details in the Supplementary Materials). The gist is that item responses (boxes) are related to the general OPP process in a two-stage process on the left and a one stage process in the right.
Figure 1
Comparison of Second-Order Factor (Left) and Bifactor (Right) Measurement Models of 12 Items and Four Specific Factors Related to Openness to Political Pluralism (OPP)
Relation of Openness to Political Pluralism to Other Constructs
We hypothesize that OPP is related to the general personality trait of openness to experience as well as conscientiousness (Soto & John, 2017), but not reducible to those traits. It should be moderately (inversely) related to dogmatic intolerance, which is related to rigidity (van Prooijen & Krouwel, 2017). We also expect that people who are high on OPP will tend to consider diverse, if not balanced, sources of information compared to those who are low on OPP. We further expect that having close others who hold different political perspectives could influence OPP, but we acknowledge that political differences are often a source of relationship strain (e.g., Chen & Rohla, 2018). Nonetheless, on average people whose family and loved ones have different political opinions have more incentive to learn to be open than those whose family and loved ones share the same political bubble.
We do not expect that a carefully constructed measure of OPP would necessarily be correlated with political ideology. An often-contentious debate in political psychology is the notion of “ideological asymmetry” in prejudicial attitudes and behavior across the left-right continuum, such that conservatives are more biased than liberals (e.g., Jost, 2017). We sought to create a measure of OPP that itself was unbiased with regard to the asymmetry-symmetry debate.
An important step in assessing the validity of the new OPP measure is to relate it to other measures of political open-mindedness. Social scientists have long been interested in open-minded thinking styles, (e.g., Baron, 1985; Stanovich & West, 1997). Baron (2019) argued that “actively open-minded thinking” is vital for a democracy to function effectively, because it provides a standard for evaluating the trustworthiness of information relevant to political decision-making. Generally, the open-minded thinking literature assumes that those with an open-minded style process information more rationally and arrive at more informed decisions regardless of their prior views.
There exist several measures of open-minded thinking. The oldest is actively open-minded thinking scale (AOT; e.g., Stanovich & West, 1997; Svedholm-Häkkinen & Lindeman, 2017). Influenced by the work of Baron (1985), it was designed to measure flexibility and rationality in thinking. More recently, Price et al. (2015) introduced a novel framework for measuring open minded cognitive style. They described an open-minded cognitive style as “a willingness to consider a variety of intellectual perspectives, values, attitudes, opinions, or beliefs, even those that contradict the individual’s prior opinion” (Price et al., 2015, p. 1488). They argued that open-minded cognition varies across individuals and contexts (religious, political and non-specific). Their political context measure (OMC-P) is most relevant to this article. In addition, Hackett, Gaffney, and Data (2018) created a measure of openness to diverse political discussions (ODPD) to understand intergroup anxiety between winners and losers following elections. See the Supplementary Materials for more information on measures of open-minded thinking.
Current Studies
We report data from three studies of openness to political pluralism. In Study 1, we present a 25-item measure of Openness to Political Pluralism (OPP) and a five-item version (OPP Short: OPPS) that captures the majority of the variance of the longer measure. Study 1 is a confirmatory report, testing a psychometric structure that was registered before data were collected in 2018. We predicted that OPP responses will be well fit by a bifactor psychometric structure (Figure 1 right side), and we compare this fit to the second-order factor model illustrated in Figure 1 (left side). We carry out initial tests of convergent and discriminant validity by correlating OPP with big-five trait measures of open-mindedness, conscientiousness, and openness. We examine the relation of OPP to left-right political orientation and the assumptions that OPP will be related to living in a diverse political environment and that those high on OPP will obtain information from relatively balanced sources. Finally, we estimate the test-retest reliability of both OPP and OPPS.
In Studies 2 and 3, we replicate the bifactor structure of the OPP and carry out further discriminant and convergent validity analyses. To differentiate OPP from other measures of political open mindedness, we ask participants to respond to the 17-item version of AOT (Svedholm-Häkkinen & Lindeman, 2017), the six-item OMC-P (Price et al., 2015), and the six-item ODPD (Hackett et al., 2018). We also assessed dogmatic intolerance (van Prooijen & Krouwel, 2017). We recruited participants from MTurk in 2019 (Study 2) and recruited participants via Qualtrics Panel in 2020 (Study 3). We preregistered main predictions and analyses prior to data. The data and syntax used for analyses of all three studies are available on the OSF repository (see Supplementary Materials).
The overall goal of this project is to provide researchers with tools for measuring individual differences in openness to discussions of diverse political ideas and perspectives. We hope that these tools can both be used in applied political survey research to predict who might be open to new messaging of candidates and policies, as well as in basic research on how voters are drawn to political bubbles and to partisan misinformation.
Study 1: Testing a Preregistered Bifactor Structure for Pluralism Openness Items
Study 1 tested a bifactor structure (see Figure 1, right panel) that we pre-registered based on exploratory analyses of data collected prior to the current studies (not reported here). The exploratory data were collected to examine 56 candidate items that were written by students in a doctoral class on psychometric theory and practice. After discussing the general concept of openness to political pluralism, students were each asked to think of two conceptual facets of openness to political pluralism and then write eight items spanning these facets, half of which were to be reverse-coded items. The four facets of openness to political pluralism emerged from this collaboration. We evaluated the items using data collected from Amazon Mechanical Turk (MTurk) for class use only. Because we agreed not to publish results with these data, the project was designated by the IRB as not-research. Nonetheless, we learned about the psychometric structure of the items and used this knowledge to inform predictions for the current research and select high-performing items from each facet.
As described below, the new measure of political pluralism has 25 items, reflecting pluralism motivation, emotional tolerance, intellectual tolerance, and non-rigidity in beliefs. Twenty-three of the items were grouped in only one of these facet themes, but two items had cross loadings in two facets in the analyses of pilot data that informed our preregistration. The 25 items are shown in Table 1; the item groupings are evident from the bifactor results shown in the table.
Table 1
Openness to Political Pluralism (OPP) Items, Descriptive Statistics, and Bifactor Model Estimates From Study 1 Sample of 400 MTurk Participants
| Item | M | SD | G | EmTol | PlurMot | IntTol | NRigid |
|---|---|---|---|---|---|---|---|
| It bothers me to hear about political standpoints that oppose mine.a | 2.36 | 1.19 | .69 | 0.51*** | |||
| I get annoyed when I find out someone disagrees with me on a political issue.a | 2.47 | 1.15 | .67 | 0.50*** | |||
| I find it frustrating to hear political commentary that I disagree with.a | 2.79 | 1.24 | .64 | 0.49*** | |||
| It doesn’t bother me if someone’s political views differ from mine. | 3.58 | 1.19 | .54 | 0.40*** | |||
| I don’t like the fact that there are people out there whose political views differ from mine.a | 2.15 | 1.13 | .73 | 0.28*** | |||
| I don’t like to be exposed to political perspectives that are dissimilar to mine.a | 2.16 | 1.04 | .74 | 0.14*** | 0.08 | ||
| I am willing to engage in discussions with individuals who hold opposing political views. | 3.94 | 0.98 | .61 | 0.62*** | |||
| I am willing to discuss political issues with persons that I know have very different opinions than I do. | 3.81 | 1.05 | .61 | 0.56*** | |||
| I talk with people who have different political views than mine to understand their arguments better. | 3.71 | 1.05 | .63 | 0.40*** | |||
| I am willing to receive criticism about my political positions from those who have differing views. | 3.79 | 1.04 | .61 | 0.35*** | |||
| I am open to challenging my perspective on political issues through conversation with others who disagree. | 3.84 | 0.99 | .69 | 0.29*** | |||
| I don't seek to understand the political arguments of people who have differing political positions than mine.a | 2.14 | 1.08 | .65 | 0.21** | |||
| I refuse to process and engage with political ideas that clash with my own.a | 1.98 | 1.03 | .70 | 0.19** | |||
| It is useless discussing political perspectives with individuals on the other side of the political spectrum.a | 2.40 | 1.22 | .74 | 0.11* | |||
| I would never subscribe to a media outlet that has different political leaning than mine.a | 3.05 | 1.29 | .55 | 0.04 | |||
| I am willing to hear and consider others’ political perspectives. | 4.08 | 0.93 | .72 | 0.24*** | .27*** | ||
| Other political views are as important to consider as my own. | 3.81 | 1.06 | .62 | .42*** | |||
| People from opposing political sides can learn from each other. | 4.17 | 0.92 | .70 | .40*** | |||
| I try to understand others even if they do not have the same beliefs as me. | 4.18 | 0.87 | .66 | .34*** | |||
| It is okay that some people’s beliefs are incompatible with mine. | 4.15 | 0.93 | .58 | .31*** | |||
| I think that people have a range of political views because there are many valid ways of thinking about these issues. | 3.93 | 1.03 | .57 | .31*** | |||
| The nation would be better off if everyone adopted my political perspective.a | 3.17 | 1.23 | .46 | .60*** | |||
| My political views are more valid than different political views.a | 2.87 | 1.17 | .61 | .50*** | |||
| Views from the other side of the political spectrum are usually wrong.a | 2.83 | 1.17 | .60 | .49*** | |||
| Positions taken by those on the other end of the political divide are often immoral.a | 2.60 | 1.25 | .63 | .44*** |
Note. The five items in boldface may be used as a Short Form (OPPS). The column labelled G includes bifactor loadings of the general factor, and the other four columns (EmTol-NRigid) show the four bifactor specific factor loadings. EmTol refers to Emotional tolerance, PlurMot to Pluralistic motivation, IntTol to Intellectual tolerance, and NRigid to Non-rigidity.
Superscript a indicates that an item is reverse coded before analysis.
In addition to the pre-registered bifactor model, we also test the fit a second-order factor model (Figure 1, left panel). This model suggests that the latent variable OPP leads to the four facets, and the facets in turn lead to the item response pattern. We compare the fits of these two multivariate models of item response.
We next begin to examine the validity of OPP as a measure of openness to information about political issues from diverse sources. We distinguish openness to political pluralism from general big five personality open-mindedness (Soto & John, 2017). Openness to political pluralism pertains to political views and does not necessarily involve creativity, intellectual capacity, or appreciation of adventure (John & Srivastava, 1999). However, we expect a small correlation between openness to political pluralism and open-mindedness, because those high in openness to political pluralism are expected to be curious about and open to others’ positions in multiple domains. We also expect two other dimensions of big five personality to have small correlations with openness to political pluralism because of their assumed associations with specific facets. We expect that individuals high on trait conscientiousness will be more motivated by civic duty to pursue pluralistic viewpoints (i.e., higher pluralism motivation), and we expect better emotional tolerance from those low on trait neuroticism (Weinschenk, 2014).
Method
Participants
Participants were 400 Amazon Mechanical Turk (MTurk) workers who completed the survey in September 2018. They were paid at a rate of $5.65 per hour for their participation. We originally obtained reports from 456 subjects but excluded 56 who did not meet preregistered eligibility criteria (see Supplementary Materials). We determined the sample size to provide standard errors of the factor loadings that are approximately .06 or smaller, which indicates precision of the estimates. This determination was based a simulation of the expected factor analysis results.
The final sample had a mean age of 36.8 years (SD = 10.4) and was 51% male. Approximately 38% of participants identified as Democrat, 19% as Republican, 41% as Independent, and 2% as other. Of the Independents, 39% reported leaning Democratic and 24% reported leaning Republican. A random subset of 215 participants were asked to complete the survey a second time one week after their initial response. Of those asked, 109 (51%) filled out the follow-up survey and 102 (47%) provided usable retest data (see Supplementary Materials for a comparison of the retest and no-retest samples).
Procedure
The median completion time for the initial survey was 6.2 minutes. At the end, we asked participants if they would be willing to take a repeat survey one week later. People took 6 minutes to complete the follow-up survey on average. Both surveys were administered using Qualtrics. Compensation for completing each survey was $0.75. The Institutional Review Board approved the procedures.
Measures
Openness to Political Pluralism
Openness to political pluralism was assessed with the items listed in Table 1, presented in randomized order. Participants responded on a 5-point scale, ranging from 1 (Strongly disagree) to 5 (Strongly agree). As we describe in the results section, we selected five of the 25 items to be in a short form, which we call OPPS. These items are bolded in Table 1. In the results section we report the internal consistency and test-retest reliability of both OPP and OPPS.
Big 5 Personality Traits
Open-mindedness, conscientiousness, and neuroticism were measured using a short form of the Big Five Inventory-2 (BFI-2-S; Soto & John, 2017). Participants answered six items per trait on 5-point scales ranging from 1 (Disagree strongly) to 5 (Agree strongly). Cronbach’s alpha (α) was .86 for open-mindedness, .86 for conscientiousness and .90 for neuroticism. Means and standard deviations were respectively, 3.82 (0.93); 3.82 (0.88); 2.51 (1.08).
Ideological Orientation
Left-right orientation was measured using two items that asked participants to place themselves on a 7-point scale ranging from 1 (Extremely Liberal) to 7 (Extremely Conservative). The first question simply asked, “Overall, where would you place yourself on the following scale of liberalism-conservatism?” The second was qualified: “In terms of social and cultural issues (e.g., abortion, separation of church and state, affirmative action), where would you place yourself on the following scale?” The two items were averaged (r = .89, α = .94; M = 3.30, SD = 1.78).
Diverse Political Context
Living in a diverse political context was measured with three items that asked participants to report if they 1) shared the same political views, 2) sometimes differed in their political views, or 3) had very different political views with respect to their intimate partner, family, and friends respectively (see Supplementary Materials). We averaged the responses to measure the tendency to have a social network with pluralistic views (M = 1.65, SD = 0.46).
Balanced Information Gathering
Information gathering was assessed by asking participants to indicate which of eight print sources, nine television/cable/radio sources, and five online sources they had used in the past week (see Supplementary Materials). Within each category, the sources were coded as right-leaning, left-leaning, or balanced according to www.allsides.com and Pew Research (Mitchell, Gottfried, Kiley, & Matsa, 2014). Participants were given one point for each balanced source and one point for each pair of left/right sources they accessed. In other words, participants earned a point each time they reported a balanced source (e.g., PBS NewsHour), and also when they reported both a left-leaning (e.g., The Atlantic) and a right-leaning (e.g., National Review) source. The summed score could range from 0 to 14, but the actual scores ranged from 0 to 10 (M = 2.32, SD = 2.09), with higher scores indicating more balanced information gathering. Using each domain (i.e. print, media and online) as a separate item indicating an interest in balanced information, Cronbach’s alpha was .65 (95% CI [.58, .70]). This index was not preregistered, and its relation to openness to political pluralism is provided for descriptive rather than confirmatory purposes.
Psychometric Analyses of Openness to Political Pluralism Items
In all three studies we employed confirmatory factor analysis (CFA) with maximum likelihood estimation to fit psychometric models of the item response data using Mplus (Version 8.1) (Muthén & Muthén, 1998-2017). For Study 1 we considered three competing models. Two of the three models were pre-registered: a baseline model with one factor only (Model 1) and a bifactor model with a general factor and four specific factors to account for associations among items within each facet (Model 3; Figure 1 Right). In response to reviewers, we also fit a second-order factor model (Model 2; Figure 1 Left). We compared the models using three fit indices (Kline, 2016, p. 269): Comparative fit index (CFI), Tucker-Lewis Index (TLI), and Root mean square error of approximation (RMSEA). For the first two indices, values greater than .90 are interpreted as adequate fit, and for the last, values less than .08 indicate adequate fit. We also used a statistical test based on likelihood ratio (LR) statistics to test if Model 2 provided a significantly better fit than Model 1 and if Model 3 provided a better fit than Model 2.
Results
Table 1 lists the OPP item means and standard deviations and the factor estimates from Model 3, the pre-registered bifactor model. Before fitting the models, we calculated the eigenvalues of the item correlation matrix (see Supplementary Materials) to assess the latent dimensionality of the item correlations. The first six eigenvalues are consistent with a strong first factor and three additional weaker factors (DeVellis, 2017, pp. 165-170): 11.68, 2.15, 1.46, 1.05, 0.77 and 0.69.
Although one factor was dominant, a single factor CFA model (Model 1), did not fit the data well (RMSEA = .117; CFI = .759; TLI = .738; LR = 1773.4 on 275 df). In contrast, Model 2 (the second-order factor model) and Model 3 (the bifactor model) had acceptable fit (see Supplementary Materials). Model 2, the second-order factor model, provided a significantly improved fit over Model 1 (RMSEA = .077; CFI = .897; TLI = .885; LR = 910.7 on 269 df; Δ LR(6) = 862.7, p < .001).
The bifactor model (Model 3) provides an even better fit to the data (RMSEA = .073; CFI = .909; TLI = .898; LR = 771.0 on 248 df), and the fit is significantly better than Model 2 (Δ LR(21) = 139.7, p < .001). The standardized factor loadings for the bifactor model are provided on the right of Table 1, including those for the general factor and the four specific factors. The loadings for the general factor ranged from .46 (Reverse-coded item, “The nation would be better off if everyone adopted my political perspective.”) to .74 (Reverse-coded item, “It is useless discussing political perspectives with individuals on the other side.”). The median loading was .63, suggesting a strong correspondence between the item responses and the openness to political pluralism latent variable.
Table 1 also shows factor loadings for the four pre-specified facets (specific factors). They are smaller than those for the general factor, indicating that the facets have less impact on item responses than the overall openness to political pluralism dimension. All but two loadings for the specific factors are significantly different from zero.
After rescaling reversed items so that larger values indicate more openness to political pluralism, we computed the average of the 25 items (OPP), which can range from 1 to 5. The mean OPP in the sample is 3.68 (SD = 0.74); this is significantly larger than the midpoint of 3, indicating that the majority of the sample was relatively open to pluralism (t(399) = 18.4, p < .001). Cronbach’s alpha, which is often a conservative index of reliability is .95. McDonald’s Omega Total in this case is also .95.
After establishing the 25-item OPP, we also created a short version by selecting five items that loaded highly on the general factor (all G loadings > 0.60), represented each of the four facets (with two items from pluralism motivation). We also sought to select face valid items spanning the breadth of item content (see Table 1)1 We averaged these items to create OPPS, which was highly correlated with OPP, r = .94 (95% CI [.93, .95]). Cronbach’s alpha for the OPPS items was .80 (95% CI [.77, .83]), and its correlation with the 20 items not included in the OPPS was .91 (95% CI [.90, .93]). The latter correlation is a more accurate estimate of OPPS reliability than Cronbach’s alpha.
We next estimated test-retest reliability over a one-week period. For OPP, the test-retest correlation was .86 (95% CI [.80, .90]), and for OPPS it was .84 (95% CI [.77, .89]). For comparison, we also calculated the test-retest reliability of political left-right orientation, r = .96 (95% CI [.94, .97]) and big five Open-mindedness (BFI-2-S), r = .89 (95% CI [.84, .92]). These test-retest estimates suggest excellent reliability.
OPP and big five open-mindedness were correlated .31 (95% CI [.22, .40]). This modest, but statistically significant, correlation was consistent with our construct validity prediction. The correlations of OPP with conscientiousness and neuroticism were .25 (95% CI [.16, .34]) and -.18 (95% CI [-.27, -.08]) respectively, again consistent with expectations.
Figure 2 shows a scatterplot of OPP and left-right political orientation. The linear correlation of these variables was not significantly different from zero, r = -.06 (95% CI [-.16, .04]). However, consistent with our hypothesis that the highest level of openness to political pluralism is in the center of the left-right orientation continuum, there was a significant quadratic association (B = -0.067, SE = 0.012, t(397) = -5.82, p < .001, 95% CI [-0.09, -0.05]).
Figure 2
Scatterplot of OPP to Left-Right Orientation Showing Fitted Values From Two-Segment Spline Model in Study 1
Note. Low values of left-right orientation indicate left-leaning and high values indicate right-leaning. The distribution of OPP scores is shown to the right.
To further examine this quadratic effect, we fitted a linear spline model that includes slopes for two distinct segments of L-R orientation. One goes from the extreme left to the midpoint, and the other goes from the midpoint to the extreme right. The test of the quadratic provides an overall test of linearity, whereas the two slopes provide direct evidence of the relation of openness to political pluralism to ideological extremity for liberals and conservatives. The below-midpoint slope of L-R orientation was positive (B = 0.14, SE = 0.036, t(397) = 3.99, p < .001, 95% CI [0.072, 0.212]) and the above-midpoint slope was significantly negative (B = -0.28, SE = 0.048, t(397) = -5.76, p < .001, 95% CI [-0.37, -0.18]; see Figure 2). The first slope was significantly less steep than the second slope (t(397) = 3.22, p <.001), suggesting that the pattern was more complicated than extremism in either party. For both liberals and conservatives, openness to political pluralism was lower the more extreme the L-R orientation, but the association was stronger for conservatives than liberals (Details shown in the Supplementary Materials).
Next, we checked if OPP related to living in a diverse political context. This index ranged from 1 (political bubble) to 3 (very different political views). The sample mean was 1.66 (SD = 0.46), and its correlation with OPP was significant but small, r = .17 (95% CI [.07, .26]). Similarly, we observed a small correlation between OPP and balanced information gathering, r = .18 (95% CI [ .08, .27]). Those with higher OPP scores were more likely to report obtaining information with balanced perspectives. We had no predictions that OPP would be related to gender, age, or education and found that none were significantly correlated (see Supplementary Materials).
Discussion
Using a preregistered measurement study informed by initial psychometric work, we tested a new measure of openness to political pluralism. Items in the OPP were well-fit by a bifactor model that recognized four facets: pluralism motivation, intellectual tolerance, non-rigidity in beliefs, and emotional tolerance. OPP and OPPS showed initial evidence of construct validity and excellent reliability. Furthermore, OPP bore an inverted U-shape relation to left-right orientation, such that moderates were more open than those at either extreme end of the political spectrum.
After establishing a politically relevant measure grounded in psychological and psychometric principles, we needed to determine how it relates to other measures of open-mindedness. To address this question and replicate the psychometric structure of OPP, we carried out Study 2.
Study 2: Convergent and Discriminant Validity of OPP
In Study 2, we included three existing measures of open-mindedness and a measure of dogmatic intolerance to establish convergent and discriminant validity of our OPP measure. We expected openness to political pluralism to relate to actively open-minded thinking (AOT) (Baron, 2019; Svedholm-Häkkinen & Lindeman, 2017), but we expected this correlation to be only moderate, which would be consistent with Price and colleagues (2015) findings for open-mindedness across domains. We expected higher correlations between OPP and two measures of open-mindedness that are specific to the political domain: the OMC-P (Price et al., 2015) and ODPD (Hackett et al., 2018). In addition, we re-examined the relations between each openness measure with validity measures from Study 1: left-right ideological orientation, exposure to diverse political context, and balanced information gathering.
Method
Participants
Participants were 258 MTurk workers who completed the survey in June and July 2019. We used TurkPrime to recruit more conservative participants than in Study 1. All participants were paid at a rate of approximately $10.00 per hour. Reports from 312 participants were initially obtained but 54 were excluded (see Supplementary Materials). The final sample size provided confidence intervals for correlations of .50 that were approximately +/- .10.
The sample had a mean age of 38.7 years (SD = 12.6) and 60% were male. Thirty-eight percent identified as Democrat, 35% as Independent, 24% as Republican, and 3% as other. Of Independents, 23% reported leaning Democratic and 40% reported leaning Republican.
Procedure
The Study 2 procedure was similar to Study 1. The survey was administered using Qualtrics and lasted a median of 7.73 minutes. Participants were paid $1.50. The Institutional Review Board approved the procedures.
Measures
Openness to Political Pluralism
Openness to political pluralism items were identical to those used in Study 1, but we changed the response scale to range from 1 (Strongly disagree) to 7 (Strongly agree) to make it comparable to the scales of other measures (OPP: α = .97; OPPS: α = .83).
Actively Open-Minded Thinking
Actively Open-Minded Thinking (AOT) was assessed with the 17-item AOT measure of Svedholm-Häkkinen and Lindeman (2017) (α = .90). Sample items include, “I believe that loyalty to one’s ideals and principles is more important than ‘open-mindedness’ (reverse-scored)” and “People should always take into consideration evidence that goes against their beliefs.” The response scale ranged from 1 (Strongly disagree) to 7 (Strongly agree).
Open-Minded Cognition Politics
Open-Minded Cognition Politics (OMC-P) has six items (α = .92). Example items include, “I have no patience for political arguments I disagree with,” and “When thinking about a political issue, I consider as many different opinions as possible” (Price et al., 2015; Study 2). The response scale ranged from 1 (Strongly disagree) to 7 (Strongly agree).
Openness to Diverse Political Discussions
Openness to Diverse Political Discussions (ODPD) was measured with six items (α = .92), including, “I enjoy listening to members of other political parties talk” (Hackett et al., 2018). The response scale ranged from 1 (Strongly disagree) to 7 (Strongly agree).
Dogmatic Intolerance
Dogmatic Intolerance (DI) was measured with six items (α = .82). Example items include, “I believe that everyone should think like me,” and “If everyone would think about it, they would hold the same opinions as I do” (van Prooijen & Krouwel, 2017). The response scale ranged from 1 (Strongly disagree) to 7 (Strongly agree).
Ideological Orientation
Left-right orientation was measured using the same items from Study 1. The two items were averaged (M = 3.61, SD = 1.88; α = .96). Scores ranged from 1 (Extremely liberal) to 7 (Extremely conservative).
Diverse Political Context
Diverse political context was measured as in Study 1. We averaged responses to measure the tendency to have a social network with pluralistic views (M = 1.64, SD = 0.43).
Balanced Information
Balanced information gathering was assessed and scored as in Study 1 but with four additional news sources (see Supplementary Materials). The sum score could have ranged from 0 to 17, but in the data it ranged from 0 to 13 (α = .81, M = 3.66, SD = 3.15).
Analyses
In Study 2 we compared the single-factor model and bifactor models using the same CFA methods described in Study 1. To assess convergent and discriminant validity, we computed Pearson correlations among OPP, OPPS and the four other measures reflecting open-minded thinking (or its opposite, dogmatic intolerance). We computed 95% confidence bounds on these correlations using Fisher’s transformation. We also estimated the correlations among the scales adjusting for measurement error. To do so, we fit a CFA model that included the bifactor model for OPP, and single-factor representations for the items associated with AOT, OMC-P, ODPD and DI, allowing the factors to correlate.
Results
The psychometric structure of the OPP items in Study 2 was similar to that predicted and found in Study 1. The one-factor model does not fit the data well (RMSEA = .12; CFI = .82; TLI = .80; LR = 1205.8 on 275 df), but the bifactor model fit is both acceptable (RMSEA = .07; CFI = .94; TLI = .92; LR = 570.4 on 248 df) and a significant improvement over the first model (χ2[27] = 635.4, p < .0001). We conclude that the test of the bifactor model in Study 1 replicates in Study 2.
Table 2 reports the means, standard deviations and correlations between OPP, OPPS, and the four open-mindedness measures2. Consistent with Study 1, OPP and OPPS are highly correlated. After adjusting for measurement error, the correlation is essentially perfect (upper triangle of Table 2 correlation matrix). Because of this redundancy, we focus on OPP going forward. As expected, OPP is strongly correlated with AOT (Pearson and adjusted correlations: .60 and .61). The square of these correlations suggests that about 36% of the variance of the general tendency to engage in open-minded thinking is shared with our measure of openness to political pluralism.
Table 2
Descriptive Statistics and Correlations Among Alternate Measures of Open-Mindedness From Study 2 Sample (N = 258)
| Measure | M | SD | OPP | OPPS | AOT | OMC_P | ODPD | DI |
|---|---|---|---|---|---|---|---|---|
| Openness to Political Pluralism (OPP) | 5.00 | 1.24 | 0.97 | 1.00 | 0.61 | 0.98 | 0.94 | -0.77 |
| Openness to Political Pluralism Short (OPPS) | 5.03 | 1.34 | 0.95 | 0.83 | 0.62 | 0.98 | 0.94 | -0.80 |
| Active Open-minded Thinking (AOT) | 5.05 | 1.11 | 0.60 | 0.60 | 0.90 | 0.56 | 0.44 | -0.62 |
| Open-minded Cognition, Politics (OMC_P) | 5.04 | 1.45 | 0.91 | 0.85 | 0.54 | 0.92 | 0.94 | -0.74 |
| Openness to Diverse Political Discussions (ODPD) | 4.51 | 1.50 | 0.88 | 0.81 | 0.43 | 0.87 | 0.92 | -0.68 |
| Dogmatic Intolerance (DI) | 2.88 | 1.16 | -0.74 | -0.71 | -0.60 | -0.67 | -0.62 | 0.82 |
Note. Last six columns show correlations among the six measures. In the lower triangle of the matrix are Pearson’s correlations of scale scores computed as averages of items. In the upper triangle are estimates of the correlations between latent variables, which are adjusted for measurement error. On the diagonal (underlined in italics) are Cronbach’s alpha for each scale. All correlations are significantly different from zero at p < .01 or greater.
The Pearson and adjusted correlations of OPP with OMC-P are (.91, .98), and those of OPP with ODPD are (.88, .94). The overlap of OPP with OMC-P is virtually complete, and a bootstrapped confidence bound on the adjusted correlation included 1.00 (95% CI [.96, 1.00]). The correlation with ODPD was nearly as high but was not completely redundant in our data (95% CI for adjusted correlation = [.88, .97]). We expected these correlations to be high, but, contrary to our expectations, they suggest that our measures and the two published measures of political openness are essentially alternate measures of the same construct. Table 2 also shows that OPP is strongly, though not redundantly, negatively correlated with dogmatic intolerance.
We next examined linear and nonlinear associations of political openness measures with left-right orientation (see Supplementary Materials for details). Similar to Figure 2, Figure 3 shows that all associations include nonlinear components.
Figure 3
Scatterplots of OPP, OPPS, AOT, OMC-P, ODPD and DI to Left-Right Orientation Showing Fitted Values From Two-Segment Spline Model in Study 2
Note. Low values of left-right orientation indicate left-leaning and high values indicate right-leaning.
Slightly different from Study 1, OPP and OPPS have significant, though small, linear correlations with left-right ideology (-.16 and -.15, respectively), wherein conservatives scored somewhat lower on OPP/OPPS relative to liberals. Like Study 1, there is strong evidence of a non-linear association between OPP and ideology. As ideology scores move from the most liberal to the midpoint, respondents report more pluralism, and as ideology scores move from the midpoint to strongly conservative, respondents report less openness to political pluralism. Details of these analyses are in the Supplementary Materials.
Except for ODPD, which has a nonsignificant negative correlation (r = -.09), the other openness-related variables have significant linear associations with left-right ideology. All four variables have significant quadratic effects, indicating nonlinear associations with left-right ideology. However, only OMC-P and ODPD have the same inverted U-shape pattern as OPP (see Figure 3). In contrast, the sign of both spline segments is negative for AOT. So, as ideology scores move from the most liberal to the midpoint, and from the midpoint to most conservative, actively open-minded thinking scores significantly decline. The results of DI are mixed; the relation of ideology to DI is significantly related in the segment from the midpoint to the most conservative, but not in the segment from the most liberal to the midpoint.
As in Study 1, we checked if OPP related to living in a context where partners, family and friends hold diverse political views. The correlation of diverse political context with OPP was significant but small (r = .20, 95% CI [.08, .32]). The correlations with AOT, OMC_P, and ODPD were nearly the same (.22, .21 and .20, respectively).
Similarly, we observed a small correlation between OPP and balanced information gathering (r = .22, 95% CI [.10, .33]). Those with higher OPP scores were more likely to obtain information with more balanced perspectives. Balanced information gathering was comparably related to AOT, OMC_P and ODPD (.31, .21 and .19, respectively).
Discussion
Study 2 replicated the psychometric structure of OPP that was found in Study 1. The bifactor model, with a general pluralism openness factor and four facets, fit the data well. These results show that changing the scale response format from 1-5 to 1-7 does not have an important impact on measurement.
Our predictions for the convergent validity of OPP/OPPS relative to existing measures of open-mindedness and dogmatic intolerance (DI) were confirmed, but our predictions for discriminant validity of OPP relative to OMC-P and ODPD were not confirmed. Study 2 revealed that OPP, OMC_P and ODPD are essentially alternate forms of a measure of the same construct.
A hotly debated topic in political psychology is the nature of the relationship between open-mindedness and left-right political identification, which we examined in Study 2. Actively open-minded thinking did not bear an inverse U-shaped relation to ideology, despite a significant quadratic term. Instead, predicted AOT diminishes as one moves rightward along the entire ideological continuum. When we examined the items closely, this finding was not surprising. AOT includes a cluster of items that are politically charged and represent “Liberalism”. For example, the item, “I consider myself to be broad-minded and tolerant of other people’s lifestyles,” might reflect an ideological position rather than a willingness to discuss ideas with those who might disagree. It also contains a cluster labeled “Dogmatism,” which includes items such as, “I believe we should look to our religious authorities for decisions on moral issues,” which similarly includes ideological content rather than simply an inclination to be rigid.
OPP, OPPS, OMC-P and ODPD are relatively free of ideological content. These measures differentiate the center of the left-right spectrum from the liberal and conservative extremes, illustrating that those who strongly endorse views on either end of the political spectrum can be closed-minded in their political views compared to those who are more moderate. This is noteworthy given that previous research has not investigated inverted U-shape relations between their measures of openness and left-right ideological orientation, except for those studying dogmatic intolerance (van Prooijen & Krouwel, 2017).
Study 3: Replication of Study 2
The respondents in both Studies 1 and 2 were recruited from the Amazon Mechanical Turk platform. In recent years a number of methodologists have raised questions about the validity of the data provided by MTurk workers (e.g., Peer et al., 2022). To guard against possible response bias from this specialized participant pool, we replicated Study 2 using a sample from a commercial survey organization, Qualtrics Panels. The methods and approach we used in Study 3 were identical to Study 2, with one exception. We added the general version of the six-item Open Minded Cognition scale (OMC-G) of Price and colleagues (2015); which is identical to OMC-P, except that politics is not mentioned.
Although the methods were mostly the same, the context of the two surveys were different in important ways. One is that Study 3 was carried out eight weeks before the heated U.S. presidential election between Donald Trump and Joe Biden, whereas Study 2 was surveyed a year earlier. The other is that Study 3 took place during the historic and politically charged COVID-19 pandemic year of 2020. Ideally, the measures of political pluralism will function well in this new context, but Study 3 must be considered partly a conceptual rather than exact replication of Study 2 because of these changes in the sociopolitical context.
Method
Participants
Participants were 313 survey workers recruited by Qualtrics Panel to be representative of the U.S. population in terms of age, education, region (four sections), and race (see Supplementary Materials). Qualtrics also aimed to balance the sample on the liberal-conservative spectrum. The survey was administered in the first two weeks of September 2020, and participants were provided incentives that were negotiated with Qualtrics-Panel, with the expected value of $10 per hour. Seventeen respondents were removed because of limited response variability, leading to a final sample size of 296.
The sample had a mean age of 47.6 years (SD = 17.4) and 30% were male3. Thirty-four percent identified as Democrat, 30% as Independent, 31% as Republican, and 4% as other. Of Independents, 37% reported leaning Democratic and 18% reported leaning Republican.
Procedure
The survey was administered using Qualtrics and the median time to complete was 11.14 minutes. The Institutional Review Board approved the procedures.
Measures
The Study 2 measures were included in Study 3, plus Open-Minded Cognition General (OMC-G), which has six items (α = .73). Example items include, “I have no patience for arguments I disagree with,” and “When thinking about an issue, I consider as many different opinions as possible” (Price et al., 2015).
Analyses
To assess convergent and discriminant validity, we computed Pearson correlations among OPP, OPPS and the four other measures reflecting open-minded thinking (or its opposite, dogmatic intolerance). We computed 95% confidence bounds on these correlations using Fisher’s transformation. Like Study 2, we also estimated the correlations among the scales adjusting for measurement error. In contrast to Study 2, we were unable to estimate these adjusted correlations from a CFA model, because its numerical solution was unstable. Instead, we calculated the adjusted estimates by dividing the Pearson correlations by the square root of the product of the alpha coefficients of the two scales being correlated (Crocker & Algina, 2006, p. 237).
Results
Confirmatory Replication Results
The psychometric structure of the OPP items in Study 3 was comparable to that found in Studies 1 and 2. The one-factor model does not fit the data well (RMSEA = .12; CFI = .66; TLI = .63; LR = 1386.1 on 275 df), but the bifactor model fit is better (RMSEA = .077; CFI = .87; TLI = 0.84; LR = 678.17 on 248 df) and a significant improvement over the first model (χ2(27) = 707.93, p < .0001). Although the pattern of results is consistent with the first two studies, the CFI and TLI fit statistics are less than the ideal values of .9 in Study 3. Because this is a confirmatory study, we first focus on the predicted model. In the Supplementary Materials we report exploratory analyses that provide insight into the fit results.
Descriptive statistics are reported in the Supplementary Materials as well as full correlation table. OPP is correlated with AOT (Pearson and adjusted correlation: .49 and .57), and even more highly correlated with OMC-P (.79, .90), and with ODPD (.80, .92). As in Study 2, the overlap of OPP with OMC-P and ODPD is high, suggesting that they are essentially alternate measures of the same construct. OPP is strongly, though not redundantly, negatively correlated with dogmatic intolerance (-.56, -.64).
We examined linear and nonlinear patterns of relations between the open-mindedness and political orientation as in Studies 1 and 2. The correlation of OPP with left-right orientation in Study 3 (r = -.08) was similar to that of Study 1 (r = -.06) but smaller than the significant correlation of Study 2 (r = -.16). AOT had a robust inverse correlation with left-right orientation in Study 3 (r = -.37), consistent with the result in Study 2. Those who report being more right-oriented tend to report lower AOT scores. Although OMC-P and DI were significantly associated with left-right orientation in Study 2, neither was significant in Study 3 (OMC-P r = -.05; DI r = .06). Replicating results from Study 2, all the quadratic effects in Study 3 were significant for the six open-mindedness measures previously studied (details in Supplementary Materials). OMC-G, which was added for Study 3, did not have either a linear or quadratic relation to political orientation. The spline model results replicated Study 2 in that they showed that OPP, OPPS, OMC-P, ODPD and DI had significant opposite slopes for the two segments, confirming an inverted U pattern for political orientation regarding these outcomes. For these measures, the highest openness was found in the middle of the left-right orientation continuum. Again, consistent with Study 2, AOT did not display the inverted U relation with political orientation.
As in Studies 1 and 2, we checked if OPP related to living in a context where partners, family and friends hold diverse political views. The correlation of diverse political context with OPP was again significant (r = .24, 95% CI [.08, .32]). The correlations with AOT, OMC_P, and ODPD were nearly the same (.15, .17 and .20, respectively).
Unlike Study 2, we did not observe a correlation between OPP and balanced information gathering (r = .05, 95% CI [-.07, .16]). Further, balanced information gathering was related to OMC_P (r = .18, 95% CI [.07, .29]), but not to AOT or ODPD (r = -.07, 95% CI [-.18, .05]; r = .09, 95% CI [-.03, .20]).
In the Supplementary Materials we report exploratory analyses of Study 3 response data to understand any differences.
Discussion
Study 3 was designed to replicate Study 2 using a nationally representative sample of respondents recruited by a professional survey organization. Study 3 also differed from Study 2 in the sociopolitical context. It was carried out in early September 2020, which was during the first year of COVID-19 pandemic and was seven weeks before a contentious U.S. presidential election that pitted the incumbent President Donald Trump against the challenger, former Vice President Joe Biden.
Despite the polarization, the OPP mean in Study 3 was similar to the OPP mean in Study 2, which was conducted in the summer of 2019 with an MTurk sample. The OPP measure had a strong common factor, and the response data was better fit by a bifactor model with four specific factors than a one factor model. The correlations of OPP with other measures of open-mindedness were similar to those found in Study 2, and the non-linear relationship with left-right orientation that we documented in Study 2 was replicated in Study 3.
General Discussion
In this research, we developed a new measure of openness to political pluralism that reflects the inclination toward and interest in obtaining political information from a range of perspectives. We included items that reflect four related facets, general intellectual tolerance, non-rigidity, emotional tolerance, and pluralism motivation. We illustrated the utility of the bifactor measurement model, which recognizes that items related to a single latent dimension are not necessarily homogeneous. Furthermore, we developed a short version of our measure that can be used by survey researchers and political pollsters. Finally, we also examined aspects of our measure’s validity with regard to left-right orientation, reports of actual balanced information gathering, and diverse social contexts.
All three studies provided encouraging results supporting the reliability and validity of the new 25-item OPP and the 5-item OPPS in samples from the United States. Reliability of the item responses was documented by test-retest data, internal consistency of the OPP, and alternate form reliability of the OPPS. We found that scores were higher among persons whose family and friends had different political perspectives, and we also found that those with higher scores were more likely to report gathering information from balanced news sources.
We found that both the politics version of the open-minded cognition measure (Price et al., 2015) and the openness to diverse political discussions measure (Hackett et al., 2018) were psychometrically equivalent to the OPP. All three (OPP, OMC-P and ODPD) had comparable inverted U-shaped relations to left-right ideology. Although the three measures may be comparable, we elaborated the components of the construct in a way that had not been previously described. The validity analyses reported here can inform the interpretation of all three measures.
In future research, those interested in basic information-seeking processes might find the 25-item OPP to be useful because of the item clusters associated with the four facets. For instance, the OPP subscales might illuminate mechanisms involved in ideologically motivated avoidance of exposure to divergent political opinions. Further, future research can examine the distinct consequences of each of the four subprocesses to examine whether, for instance, being high on proactive pluralism motivation can better predict behavioral outcomes than the three more passive facets.
Importantly, our results provide evidence of ideological symmetries (e.g., Kahan, 2013) in openness to political pluralism, such that those on both extreme ends of the left-right ideological spectrum are less open-minded to political dialogues than their moderate counterparts. At the same time, our results also demonstrate differences in steepness of slopes that can be interpreted as a weaker tendency for leftist (vs. rightist) extremes to be close-minded (e.g., Jost, 2017).
Our studies have important limitations. Although political polarization has increased in many democratic nations, we only studied US samples. This is because the US is viewed as a model of modern democracy, but nonetheless has experienced polarization that undermines democratic rule. Measures of political openness have promise for research in other nations, assuming an adequate translation is obtained when relevant. However, research in non-US democracies is needed to verify that the measures have the same psychometric structure as reported here.
Our report introduces the bifactor model as an alternative to a second-order factor model for multifaceted constructs. However, we limited our discussion of the psychometric methodology so that we could focus on important conceptual issues. For a discussion of methodological issues, including the consideration of the item response distributions, alternate estimation strategies and debates about the limitations of bifactor models, we recommend the article by Sellbom and Tellegen (2019).
Our studies depended on self-reports and are subject to response bias, including social desirability bias. Insofar as openness to political pluralism is viewed to be desirable, respondents might overestimate their openness in their self report; this overestimation could be in awareness or out of their awareness. More work is needed to relate self-reported OPP to behavioral indications of openness.
In conclusion, the media and press often emphasize political polarization. Nonetheless, our studies reveal that many Americans are open to political pluralism, and that such openness is found especially among ideological moderates. Insofar as measures of open-mindedness are to be used to find common ground across the political spectrum, one can measure openness to political pluralism and identify those most receptive to information facilitating compromise and bipartisanship. It is also possible to identify the persons on both sides of the political spectrum who are the most polarized and insular for targeted interventions. We believe that the openness to political pluralism construct and its measures will be useful in a variety of future research on information seeking and political psychology.