Featured post

SuAVE Data Insights Blog: Introduction

This blog will highlight SuAVE analysis capabilities and demonstrate how your insights from exploring a dataset in SuAVE can be stitched together into analysis pathways. Basically, we want to tell a data analysis story, and illustrate it with references to patterns you can examine on your own. We invite readers to click on the links and open SuAVE at the annotation points, to follow our analysis, or to take the analysis in other directions. I am not a sociologist, biologist, geologist, or art historian (as some of the datasets we are going to analyze might suggest) – I am just playing with the data. So if you take the analysis in other directions and share what you found – that will be great.

– Ilya Zaslavsky

Slicing and Dicing Facebook Ads from Russia

Many reviews of Facebook and Instagram ads purchased by the Internet Research Agency (IRA) in Saint-Petersburg, Russia, have been published by national news media since these ads were released by Democrats on the Permanent Select Committee on Intelligence of the US House of Representatives (see links below this post, and the context of the release at https://democrats-intelligence.house.gov/facebook-ads/). These reviews primarily emphasized that:

  1. The ads were meant to sow discord in the US political system by emphasizing key divisive points in the US political discourse;
  2. There is much contradictory messaging as ads focused on different target groups and presented different themes to them;
  3. Overall, the ads were favorable to the Trump campaign and negative to Clinton. At the same time, there were a number of pro-Clinton ads as well, which led some observers, on the Republican side, to suggest that the main focus was not to influence the elections in favor of Trump but to generally undermine public trust in the system.

While these points are generally valid, an additional analysis reveals a more interesting picture. With SuAVE one can analyze the ads in several ways. For example, we can now look at how various target groups were addressed over time, what types of ads were pushed stronger than others (based on their cost, number of impressions, clicks, etc.), which groups were targeted for different types of ads (based on communities built around several monikers, as well as on target interests and geographic location of the audience, and on audiences to be excluded), and the multitude of voices impersonated by the Russian actors for each group. All these factors are present in the SuAVE application that enables joint analysis of ad metadata, content and images. This approach lets us uncover a somewhat more systematic and well thought out set of strategies for influencing electoral opinions. Moreover, availability of the data on the web in an easy-to-use form lets the public to analyze the data by themselves and make their own conclusions, rather than rely on narratives as presented in competing reports from the Republican majority (https://docs.house.gov/meetings/IG/IG00/20180322/108023/HRPT-115-1_1-p1-U3.pdf ) and Democratic minority (https://docs.house.gov/meetings/IG/IG00/20180322/108023/HRPT-115-2.pdf) members of the Committee on Intelligence.

Here are some patterns one can easily see in SuAVE. Disclaimer: the information was parsed from PDF files. While every effort was made to keep the data free of errors (through a series of automatic checks and manual corrections), PDF parsing is far from fool-proof. SuAVE users can retrieve and examine the original PDF document for any ad by clicking the ad’s numeric label in the right (information) pane. Please email me (or add a comment to that ad) if you run into any discrepancies.

Let’s look at the most expensive ads first. The majority of the ads were paid for in rubles, with the most expensive one costing over 330,000 rubles (roughly over $5.5K). 113 ads, or about 3% of all ads, were over 10K rubles  Click to open in SuAVE (click the snapshot on the right, or the following link: http://suave2.sdsc.edu/main/file=suavedemos_Facebook_Ads_from_Russia.csv&view=bucket&id=602dfec57791e730201ee6e2 ). They were posted by various actors, from accounts referencing Black Matters, LGBT United, Save the 2nd, South United, Secured Borders, Army of Jesus, United Muslims, and others. In 2015, such relatively expensive ads were used to build and promote group identities. For example, they would focus on Texas identity and target Texan audience (“Time to secede”, “Best state in the US”, from actor “Heart of Texas”), promote themes of African-American identity and issues (with messages such as “Black Lives Matter!”, “I am black and I am proud!”, “Join us because we care”, from actors named “Black Matters” and “Blacktivist”), Muslim identity (from “United Muslims of America”), LGBT causes (from “LGBT United”: “We speak for all fellow members of LGBT community across the nation”). The key message of the most expensive Facebook ad, from an actor named “Being Patriotic” in June 2015, is simply “United we Stand!” (click image on the left, or open  Click to open in SuAVEhttp://suave2.sdsc.edu/main/file=suavedemos_Facebook_Ads_from_Russia.csv&view=bucket&id=602e2cca7791e730201ee6e4),  while the next most expensive ad from the same actor invites Facebook users to share their 9/11 stories under a #My911Story tag. The most expensive Instagram ads, at the same time, exploit predominantly black issues, such as black unity in the face of oppression (“People, our race is in danger!”) and police violence. It appears that these 2015 ads were mostly meant to build and expand a base of followers for each of the respective groups; for example, the ads of this period often exclude Facebook audiences who liked these groups already, focusing on potential new recruits. Many of these ads also mean to establish news centers where group members would continuously get information about the group’s activity and positions, and otherwise lay the groundwork for subsequent more targeted (and less expensive) messaging.

In 2016, and especially in the months leading to the November elections, the nature of the ads changes. Instead of community building, through the emphasis on unity around common cultural and social issues and very limited political action content, the ads from the ringleaders of the already established groups pivot to direct political messaging. For example, the “Being Patriotic” actor we saw earlier now shares a “March for Trump” event (with an additional #Hillaryforprison hashtag) or claims that “Clinton is the co-author of Obama’s anti-police and anti-Constitutional propaganda”, or shows Clinton in a prison cell. To see the evolution of “Being Patriotic” ads over time click the image on the right, Click to open in SuAVEor follow http://suave2.sdsc.edu/main/file=suavedemos_Facebook_Ads_from_Russia.csv&view=bucket&id=602e2d987791e730201ee6e5). At the same time, the actor “Donald Trump America” calls for the removal of Clinton from the presidential ballot. At this point in 2016, all of these actors have a significant following because of the earlier community building effort. At the same time, expensive ads promote anti-immigration and border security topics (e.g., from the “Stop A.I.” actor, where A.I. stands for “alien invaders”, and from the “Secured Borders” actor), and focus on freedom (the “Stand for Freedom” actor), values (“Angry Eagle”), “militarized” religion (“Army of Jesus”), gun ownership (“Defend the 2nd”). While the identity themes from the earlier period continue (e.g., “Proud to be black”, “South United”, “LGBT United”) such messages are now mixed with pro-Trump election themes and attempts to suppress or change the vote of certain traditionally pro-democratic groups (such as “Hillary Clinton Doesn’t Deserve Black Vote” from the “Black Matters” actor – seeClick to open in SuAVE http://suave2.sdsc.edu/main/file=suavedemos_Facebook_Ads_from_Russia.csv&view=bucket&id=602e2e847791e730201ee6e6 or click image on the left), or “We don’t have any other choice this time but boycott the election. This time we choose between two racists” by “Williams & Kalvin”, a YouTube group that combined anti-Clinton messages with more common issues of racism and police brutality. Click the image on the right or Click to see in SuAVEfollow  http://suave2.sdsc.edu/main/file=suavedemos_Facebook_Ads_from_Russia.csv&view=bucket&id=602e2f717791e730201ee6e7 to examine this pattern and similar boycott posts from “Woke Blacks”.

Another interesting observation is that the target communities changed after the elections. For example, the “Being Patriotic” actor disappeared for half a year after October 2016 (was that a well-deserved vacation for the operative?), and only reappeared in May 2017 with 8 ads, which were a mixture of generic “patriotism” wording with historical references and a message against US involvement in Syria (click image on the left or follow Click to open in SuAVEhttp://suave2.sdsc.edu/main/file=suavedemos_Facebook_Ads_from_Russia.csv&view=bucket&id=602e308d7791e730201ee6e8 to see these ads.) The same happened with the “United Muslims of America”: after not being seen for 6 months the actor published 9 ads during the same week as “Being Patriotic” – also against the US involvement in Syria. Several communities never reappeared after the elections (e.g., “LGBT United”, “Don’t Shoot”, “Stop A.I.”, “MusicFB”) – possibly because it wasn’t immediately clear how they could be turned into propaganda channels on issues related to Russian political interests of the day. At the same time, an entirely new and active group was created after the election, targeting Hispanics. The “Brown United Front”, posting as “Brown Power”, with 203 ads after the elections, mostly focused on community-uniting issues, such as Mexican culture and music, and Chicano history, with many ads, including the more expensive ones, Click to open in SuAVEemphasizing “Mexican Pride” (see image on the right or http://suave2.sdsc.edu/main/file=suavedemos_Facebook_Ads_from_Russia.csv&view=bucket&id=602e310c7791e730201ee6e9 ) This could be a community in “grooming” before subsequent more direct propaganda messaging. In other words, it follows the same pattern we’ve observed for other groups: first communities are built on the message of unification and shared values, exploiting real divisive issues and prejudices, then these “sleeper” groups turn into an active propaganda force before elections. This has been a model followed by actors who exploited patriotic, freedom and African-American themes during the community-building phase, then pivoted to electoral messages during the “active” phase. The “Brown Power” group appears to be repeating this pattern. Not counting the generic border security and alien invasion-themed posts, there were only 3 ads that referred to Latinos prior to the elections: 2 from “Being Patriotic” – referring to Latinos as criminals and Obama voters, and a post against the Million Man March; and one from “LGBT United” featuring a Latino gay soldier. Compared to the 3 ads over 16+ months, the post-election period saw 200+ ads mentioning Hispanic themes.

Compare this analysis with how these data have been reviewed in mass media – and don’t hesitate to examine the patterns on your own and share what you found. Selected stories in the media are listed below:

More about horse race polling post mortem

As pollsters continue to explore theories of what went wrong, an interesting picture is emerging. So far, little statistical support was found for the “shy Trump voters” theory (see this 538’s analysis) . Opinions seem to converge on polls underestimating the unity of the republican base and the eleventh hour splash in enthusiasm that the polls were slow  to capture. Strategies focused on fragmenting the republican base didn’t work as expected. This goes well with the conclusion we reached above: in case of republican women party identification was a much stronger factor of their opinions and voting behavior than gender identification. Similarly, post-election analysis shows that an attempt to influence larger numbers of Latino voters  – another target group – wasn’t too productive either, despite earlier expectations that GOP party, and its base, would splinter .

An additional issue that we also noted in the previous post is that horse race polls impose a potentially misleading identity framework, don’t sufficiently support micro-targeting and also are slow to reflect shifts in the electorate. Broad characteristics of the electorate measured by such polls (gender, party affiliation, ethnicity) cannot compete with detailed user profiles accumulated by social networks and search engines – which allow for a much more precise targeting. Neither can traditional polling compete with the speed of measuring feedback in social networks – in particular the effects of micro-targeted fake news, which appear to have been a factor in boosting or depressing enthusiasm. Analysis of larger surveys, such as ANES, can help with micro-targeting – but certainly won’t help with the need to measure responses at the Internet speed. The three types of analysis – large infrequent surveys, horse race polls, and social network analysis – should be done in combination.

Elections polling post mortem

While polling and forecasting of the 2016 presidential election may not have been far off (within error margins), the outcome defied the widely expected Clinton advantage, leaving many people  – pollsters and pundits included – shocked. Numerous explanations for this surprising miss have been proposed: some voters may have been reluctant to admit they were planning to vote for Trump, the media’s focus on scandals rather than policy analysis, the illusion of parity between opposing positions, the FBI’s intervention 10 days before November 8th (too late to affect polling), and the demotivating effect of polls that prematurely suggested a Clinton victory. More explanations may emerge, but one clear issue is that “horse race” polls are not designed to uncover or explain deeper patterns and shifts in voter sentiment. Such depth requires larger surveys with a broader set of questions. In stock market terms, technical analysis (polling) should be supported by a fundamental analysis of voter attitudes. We need not only better data but also better methods to query, visualize, and interpret it. 

To gain insight, let’s explore the 2016 American National Election Survey (ANES) Pilot. Although it has limitations (it’s an opt-in online panel, we’re using the initial release from January 2016, and it may contain data errors), it offers a wealth of data on political preferences and stereotypes.

One critical issue for the Clinton campaign was engaging women. Yet Republican women strongly preferred Trump. Was it effective to try to attract them by emphasizing issues assumed to be important to women? This raises a reasonable question: do Republican women’s views align more with Republicans in general or with women in general? Should they have been treated as a distinct target group? That might be a natural conclusion from “horse race” polling, where gender and party affiliation are commonly noted—but does it reflect reality, or would it skew campaign planning? What are Republican women’s attitudes on women’s issues?

The 2016 ANES Pilot provides some interesting answers. Open this survey in SuAVE and explore along.

  1. Open the survey in SuAVE and switch to bucket view (the second icon at the top right). To filter for Republican women, type “Gender” into the “Search variables…” box, expand the “Gender” variable in the left frame, and check “Female.” Then type “repub” in the same search box, expand “PartyID Republican First,” and check “Republican.” Now you’ll see icons representing Republican women, which you can explore by clicking each icon to view individual responses or by selecting variables from the drop-down menu. 
    87 percent of republican women voted in 2012

    87 percent of Republican women voted in 2012

     

    For example, here is how this group voted in 2012: 87% of Republican women in this sample definitely participated in the 2012 election, indicating they are a politically engaged group worth following.

     

  2. To examine how this group aligns with common stereotypes, type “Obama” in the search box, and select “Barack Obama is/is not a Muslim” from the drop-down. The result is striking: 68% of Republican women think Obama is a Muslim (See it in SuAVE). By clicking “more info” next to the red count box, you’ll see numeric in
    61 percent of Republicans believe that Obama is a Muslim; 68 percent among Republican women

    61 percent of Republicans believe that Obama is a Muslim; 68 percent among Republican women

    sights: being both Republican and female increases the likelihood of this belief.

    Removing the gender filter reduces this percentage to 61% among Republicans in general (or 75 out of 123 Republicans in the survey.) See this result in SuAVE. In other words, Republican women are even more likely than Republican men (54%) to believe this statement.

    Women's rights most important for choosing a political candidate only ranked fourth by one person among republican women

    “Women’s rights most important for choosing a political candidate” only ranked fourth by one person among Republican women

     

  3. Next, type “women” in the search box and select “Women’s rights most important for choosing” [a political candidate]. Only one Republican woman in this group ranked this issue among her top priorities, and she placed it fourth (See it in SuAVE.)
    Overall, 92 women ranked women's rights among first four priorities

    Overall, 92 women ranked women’s rights among the first four priorities

     

    Removing the “Republican” filter gives a very different picture: across all political affiliations, 92 women ranked women’s rights among their top four issues. Typing “most important” in the search box allows you to explore other prioritized issues, including gun control, health, national debt, environment, and morality.

 

 

In summary: Republican women, as a group, are less concerned with women’s rights and are sometimes more conservative than Republican men on certain issues. This suggests that treating Republican women as a distinct group of voters may not be necessary or may even mislead campaign planning, as they tend to align much more closely with Republican views than with those of women generally.

Note: This text demonstrates how SuAVE can be used to explore a large survey dataset, focusing on one example where deeper analysis might have improved strategy and predictions. It does not attempt to fully analyze the reasons for the election outcome; for that, see  CNN’s list of 24 theories, or the detailed discussions on FiveThirtyEight about polling missteps.

– Ilya Zaslavsky