Featured post

SuAVE Data Insights Blog: Introduction

This blog will highlight SuAVE analysis capabilities and demonstrate how your insights from exploring a dataset in SuAVE can be stitched together into analysis pathways. Basically, we want to tell a data analysis story, and illustrate it with references to patterns you can examine on your own. We invite readers to click on the links and open SuAVE at the annotation points, to follow our analysis, or to take the analysis in other directions. I am not a sociologist, biologist, geologist, or art historian (as some of the datasets we are going to analyze might suggest) – I am just playing with the data. So if you take the analysis in other directions and share what you found – that will be great.

– Ilya Zaslavsky

Slicing and Dicing Facebook Ads from Russia

Many reviews of Facebook and Instagram ads purchased by the Internet Research Agency (IRA) in Saint-Petersburg, Russia, have been published by national news media since these ads were released by Democrats on the Permanent Select Committee on Intelligence of the US House of Representatives (see links below this post, and the context of the release at https://democrats-intelligence.house.gov/facebook-ads/). These reviews primarily emphasized that:

  1. The ads were meant to sow discord in the US political system by emphasizing key divisive points in the US political discourse;
  2. There is much contradictory messaging as ads focused on different target groups and presented different themes to them;
  3. Overall, the ads were favorable to the Trump campaign and negative to Clinton. At the same time, there were a number of pro-Clinton ads as well, which led some observers, on the Republican side, to suggest that the main focus was not to influence the elections in favor of Trump but to generally undermine public trust in the system.

While these points are generally valid, an additional analysis reveals a more interesting picture. With SuAVE one can analyze the ads in several ways. For example, we can now look at how various target groups were addressed over time, what types of ads were pushed stronger than others (based on their cost, number of impressions, clicks, etc.), which groups were targeted for different types of ads (based on communities built around several monikers, as well as on target interests and geographic location of the audience, and on audiences to be excluded), and the multitude of voices impersonated by the Russian actors for each group. All these factors are present in the SuAVE application that enables joint analysis of ad metadata, content and images. This approach lets us uncover a somewhat more systematic and well thought out set of strategies for influencing electoral opinions. Moreover, availability of the data on the web in an easy-to-use form lets the public to analyze the data by themselves and make their own conclusions, rather than rely on narratives as presented in competing reports from the Republican majority (https://docs.house.gov/meetings/IG/IG00/20180322/108023/HRPT-115-1_1-p1-U3.pdf ) and Democratic minority (https://docs.house.gov/meetings/IG/IG00/20180322/108023/HRPT-115-2.pdf) members of the Committee on Intelligence.

Here are some patterns one can easily see in SuAVE. Disclaimer: the information was parsed from PDF files. While every effort was made to keep the data free of errors (through a series of automatic checks and manual corrections), PDF parsing is far from fool-proof. SuAVE users can retrieve and examine the original PDF document for any ad by clicking the ad’s numeric label in the right (information) pane. Please email me (or add a comment to that ad) if you run into any discrepancies.

Let’s look at the most expensive ads first. The majority of the ads were paid for in rubles, with the most expensive one costing over 330,000 rubles (roughly over $5.5K). 113 ads, or about 3% of all ads, were over 10K rubles  Click to open in SuAVE (click the snapshot on the right, or the following link: http://suave2.sdsc.edu/main/file=suavedemos_Facebook_Ads_from_Russia.csv&view=bucket&id=602dfec57791e730201ee6e2 ). They were posted by various actors, from accounts referencing Black Matters, LGBT United, Save the 2nd, South United, Secured Borders, Army of Jesus, United Muslims, and others. In 2015, such relatively expensive ads were used to build and promote group identities. For example, they would focus on Texas identity and target Texan audience (“Time to secede”, “Best state in the US”, from actor “Heart of Texas”), promote themes of African-American identity and issues (with messages such as “Black Lives Matter!”, “I am black and I am proud!”, “Join us because we care”, from actors named “Black Matters” and “Blacktivist”), Muslim identity (from “United Muslims of America”), LGBT causes (from “LGBT United”: “We speak for all fellow members of LGBT community across the nation”). The key message of the most expensive Facebook ad, from an actor named “Being Patriotic” in June 2015, is simply “United we Stand!” (click image on the left, or open  Click to open in SuAVEhttp://suave2.sdsc.edu/main/file=suavedemos_Facebook_Ads_from_Russia.csv&view=bucket&id=602e2cca7791e730201ee6e4),  while the next most expensive ad from the same actor invites Facebook users to share their 9/11 stories under a #My911Story tag. The most expensive Instagram ads, at the same time, exploit predominantly black issues, such as black unity in the face of oppression (“People, our race is in danger!”) and police violence. It appears that these 2015 ads were mostly meant to build and expand a base of followers for each of the respective groups; for example, the ads of this period often exclude Facebook audiences who liked these groups already, focusing on potential new recruits. Many of these ads also mean to establish news centers where group members would continuously get information about the group’s activity and positions, and otherwise lay the groundwork for subsequent more targeted (and less expensive) messaging.

In 2016, and especially in the months leading to the November elections, the nature of the ads changes. Instead of community building, through the emphasis on unity around common cultural and social issues and very limited political action content, the ads from the ringleaders of the already established groups pivot to direct political messaging. For example, the “Being Patriotic” actor we saw earlier now shares a “March for Trump” event (with an additional #Hillaryforprison hashtag) or claims that “Clinton is the co-author of Obama’s anti-police and anti-Constitutional propaganda”, or shows Clinton in a prison cell. To see the evolution of “Being Patriotic” ads over time click the image on the right, Click to open in SuAVEor follow http://suave2.sdsc.edu/main/file=suavedemos_Facebook_Ads_from_Russia.csv&view=bucket&id=602e2d987791e730201ee6e5). At the same time, the actor “Donald Trump America” calls for the removal of Clinton from the presidential ballot. At this point in 2016, all of these actors have a significant following because of the earlier community building effort. At the same time, expensive ads promote anti-immigration and border security topics (e.g., from the “Stop A.I.” actor, where A.I. stands for “alien invaders”, and from the “Secured Borders” actor), and focus on freedom (the “Stand for Freedom” actor), values (“Angry Eagle”), “militarized” religion (“Army of Jesus”), gun ownership (“Defend the 2nd”). While the identity themes from the earlier period continue (e.g., “Proud to be black”, “South United”, “LGBT United”) such messages are now mixed with pro-Trump election themes and attempts to suppress or change the vote of certain traditionally pro-democratic groups (such as “Hillary Clinton Doesn’t Deserve Black Vote” from the “Black Matters” actor – seeClick to open in SuAVE http://suave2.sdsc.edu/main/file=suavedemos_Facebook_Ads_from_Russia.csv&view=bucket&id=602e2e847791e730201ee6e6 or click image on the left), or “We don’t have any other choice this time but boycott the election. This time we choose between two racists” by “Williams & Kalvin”, a YouTube group that combined anti-Clinton messages with more common issues of racism and police brutality. Click the image on the right or Click to see in SuAVEfollow  http://suave2.sdsc.edu/main/file=suavedemos_Facebook_Ads_from_Russia.csv&view=bucket&id=602e2f717791e730201ee6e7 to examine this pattern and similar boycott posts from “Woke Blacks”.

Another interesting observation is that the target communities changed after the elections. For example, the “Being Patriotic” actor disappeared for half a year after October 2016 (was that a well-deserved vacation for the operative?), and only reappeared in May 2017 with 8 ads, which were a mixture of generic “patriotism” wording with historical references and a message against US involvement in Syria (click image on the left or follow Click to open in SuAVEhttp://suave2.sdsc.edu/main/file=suavedemos_Facebook_Ads_from_Russia.csv&view=bucket&id=602e308d7791e730201ee6e8 to see these ads.) The same happened with the “United Muslims of America”: after not being seen for 6 months the actor published 9 ads during the same week as “Being Patriotic” – also against the US involvement in Syria. Several communities never reappeared after the elections (e.g., “LGBT United”, “Don’t Shoot”, “Stop A.I.”, “MusicFB”) – possibly because it wasn’t immediately clear how they could be turned into propaganda channels on issues related to Russian political interests of the day. At the same time, an entirely new and active group was created after the election, targeting Hispanics. The “Brown United Front”, posting as “Brown Power”, with 203 ads after the elections, mostly focused on community-uniting issues, such as Mexican culture and music, and Chicano history, with many ads, including the more expensive ones, Click to open in SuAVEemphasizing “Mexican Pride” (see image on the right or http://suave2.sdsc.edu/main/file=suavedemos_Facebook_Ads_from_Russia.csv&view=bucket&id=602e310c7791e730201ee6e9 ) This could be a community in “grooming” before subsequent more direct propaganda messaging. In other words, it follows the same pattern we’ve observed for other groups: first communities are built on the message of unification and shared values, exploiting real divisive issues and prejudices, then these “sleeper” groups turn into an active propaganda force before elections. This has been a model followed by actors who exploited patriotic, freedom and African-American themes during the community-building phase, then pivoted to electoral messages during the “active” phase. The “Brown Power” group appears to be repeating this pattern. Not counting the generic border security and alien invasion-themed posts, there were only 3 ads that referred to Latinos prior to the elections: 2 from “Being Patriotic” – referring to Latinos as criminals and Obama voters, and a post against the Million Man March; and one from “LGBT United” featuring a Latino gay soldier. Compared to the 3 ads over 16+ months, the post-election period saw 200+ ads mentioning Hispanic themes.

Compare this analysis with how these data have been reviewed in mass media – and don’t hesitate to examine the patterns on your own and share what you found. Selected stories in the media are listed below:

More about horse race polling post mortem

As pollsters continue to explore theories of what went wrong, an interesting picture is emerging. So far, little statistical support was found for the “shy Trump voters” theory (see this 538’s analysis) . Opinions seem to converge on polls underestimating the unity of the republican base and the eleventh hour splash in enthusiasm that the polls were slow  to capture. Strategies focused on fragmenting the republican base didn’t work as expected. This goes well with the conclusion we reached above: in case of republican women party identification was a much stronger factor of their opinions and voting behavior than gender identification. Similarly, post-election analysis shows that an attempt to influence larger numbers of Latino voters  – another target group – wasn’t too productive either, despite earlier expectations that GOP party, and its base, would splinter .

An additional issue that we also noted in the previous post is that horse race polls impose a potentially misleading identity framework, don’t sufficiently support micro-targeting and also are slow to reflect shifts in the electorate. Broad characteristics of the electorate measured by such polls (gender, party affiliation, ethnicity) cannot compete with detailed user profiles accumulated by social networks and search engines – which allow for a much more precise targeting. Neither can traditional polling compete with the speed of measuring feedback in social networks – in particular the effects of micro-targeted fake news, which appear to have been a factor in boosting or depressing enthusiasm. Analysis of larger surveys, such as ANES, can help with micro-targeting – but certainly won’t help with the need to measure responses at the Internet speed. The three types of analysis – large infrequent surveys, horse race polls, and social network analysis – should be done in combination.

Elections polling post mortem

While polling and forecasting of the 2016 presidential election may have been not too far off (within error margins), the outcome was the opposite to the widely perceived Clinton advantage. So many people were so surprised, the pundits and pollsters not less so than the public! Many explanations of this shocking miss have been offered: respondents too shy to say that they were planning to vote for Trump, media’s focus on scandals at the expense of analysis and the false sense of parity when discussing opposing positions, FBI’s intervention 10 days before Nov 8th – too late for polling to appreciate, demotivating effects of polls anointing Clinton too early, etc. Many more explanations are yet to be offered. But one undisputed issue is that horse race polls are not designed to uncover and explain fundamental patterns and shifts in the opinions of the electorate. Sufficient depth is provided by larger surveys, with many questions. In stock market terms, technical analysis should be supported by the analysis of fundamentals. Not only we need better data, but we also need better ways to query, visualize and understand it. Let’s try to find some explanations in the 2016 American National Election Survey (ANES) Pilot. With all of its drawbacks (it is an opt-in online panel, and we are looking at the initial release, from January 2016, it may contain data errors), it has a wealth of information on political preferences and stereotypes.

One of the key issues in this electoral season, from the Clinton’s side, was engaging women. However, republican women preferred Trump by a wide margin. Was trying to encourage them to vote for the first female presidential candidate based on issues that should be important to them as women, the right strategy? Reasonable questions would then be: do opinions of republican women align more with republicans or with women? Should they be considered a separate target group in a campaign?  That would be natural based on horse race polls where questions about gender and party affiliation are normally asked – but is that right or would it skew campaign planning? What are attitudes to women-related issues among republican women?

The 2016 ANES Pilot provides some interesting answers. Open this survey in SuAVE and explore along.

  1. Let’s switch to bucket view (second icon at top-right), and zero in on republican women. Type “Gender” in the “Search variables…” search box, expand it, and check “Female”. Then search for “repub” by typing it in the same search box, expand “PartyID Republican First”, and check “Republican”. Now you see icons that represent rep
    87 percent of republican women voted in 2012

    87 percent of republican women voted in 2012

    ublican women, and can explore how they responded to survey questions. Just click on any icon to see answers of that respondent, or select a variable from the drop-down list at top-right, to examine distributions. For example, here is how this group voted in 2012: 87% definitely participated in the elections – so this is a solid group to follow and try to influence.

  2. Let’s now see where this group stands with respect to common controversies/stereotypes. Type “Obama” in the search box, and then select “Barack Obama a Muslim is/is not a Muslim” from the drop-down. See it in SuAVE: 68% of republican women think he is a Muslim. Click “more info” next to the red box with the
    61 percent of republicans believe that Obama is a Muslim; 68 percent among republican women

    61 percent of republicans believed that Obama is a Muslim; 68 percent among republican women

    count and percentage of respondents. This will give you interesting numerical measures of the pattern you observe: both factors (being a republican and being a female) have positive contributions to the response “Obama is a Muslim.”  If we remove gender from this equation, the statement “Obama is a Muslim” will become less accurate (by 7 percentage points), because the percentage of republicans who support this stereotype is 61% (or 75 out of 123 republicans in the survey). See this result in SuAVE. In other words, republican women are to the right of republican men on this question. Only 54% of republican men think that Obama is a Muslim.

    Women's rights most important for choosing a political candidate only ranked fourth by one person among republican women

    “Women’s rights most important for choosing a political candidate” only ranked fourth by one person among republican women

     

  3. Now let’s see how this political-demographic group responds to women’s issues. Type “women” in the search variables box and select “Women’s rights most important for choosing” [a political candidate] from the drop-down box (you can see how all questions and responses were formulated in the survey codebook – click About Survey at the top right). There will be just one respondent who included this issue in her preferences (and gave it the 4th rank). See it in SuAVE. If you remove “republicans” as one of the filters to
    Overall, 92 women ranked women's rights among first four priorities

    Overall, 92 women ranked women’s rights among first four priorities

    see how women of any political persuasion define their preferences wrt women’s issues – you’ll get a very different picture: 92 women rank women’s issues among their top four preferences.  You can type “most important” in the search box to explore importance of other issues for these groups, from gun control and terrorism to health, national debt, environment, inequality, poverty and morality.

 

What do we have in the end? Republican women as a group are relatively less concerned about women’s rights, and may be to the right of republican men on some political questions. Another way to put it is that there are few gender differences in terms of how policy-related questions are answered by respondents who identify as republicans, and therefore treating republican women as a separate group of voters makes little sense.

As a side note: it reminds me of the first large public opinion survey during Soviet times, led by Boris Grushin in the city of Taganrog – that was during a brief period when sociology was acknowledged as a science. One of the interesting outcomes was that there were no discernible gender differences in how respondents answered questions.

Note: the above is a software demonstration of what one can do with a large survey dataset in SuAVE, and it focuses on only one issue where a more in-depth analysis might have resulted in a better strategy and forecasting. It is not an analysis of the reasons for election outcomes; for the latter see a list of 24 theories from CNN, and more detailed studies that will eventually follow. A discussion among pollsters at fiverthirtyeight.com about what went wrong is another excellent source.

– Ilya Zaslavsky