Tag: analyzing news headlines

Categories
APIs NLP

Ask NLP: The Media’s Portrayal of Obama Part 1

Recently, I’ve been playing with the NY Times Archived Headlines. As part of a series on exploring news headlines, I am also exploring how the news portrays presidents. While Joe Biden may be president now, I thought it would be apt to start with the internet’s favorite president, Barack Obama. In the recent post about how to actually get all of the headlines about Obama, we found out that there were 1000s of headlines about Obama during his presidency, including 62 in January of 2017.

Click here to go to the results.

In this post, we’ll be doing an analysis of the media portrayal of Obama during his presidency via the most common phrases used in NY Times headlines. To follow along with this tutorial you’ll need to get your API key from the NY Times and The Text API. You’ll also need to use your package manager to install the requests module. You can install it by using the line below in your terminal.

pip install requests

Setting Up the HTTP Request

As we do with every program, we’ll start by handling our imports. For this program, we’ll need to import the requests library to handle the API request and the json library to parse it. With my code structure, seen here on Github, you’ll need to use sys to pull the system path up two levels in order to import the config file containing the API key from the Text API and the base _url to direct our HTTP request to. Alternatively, you could also just save your API key in this file. The base URL we’ll be using is “https://app.thetextapi.com/text”, you can read more on The Text API Docs.

After handling our imports, we’ll set up the headers we need and finish out the URL. The headers we need will tell the server that we’re sending JSON data and also pass the API key. For this program’s endpoint, we’ll append most_common_phrases to the base URL.

# import libraries
import requests
import json
import sys
sys.path.append("../..")
from nyt.config import thetextapikey, _url
 
# set up request headers and URL
headers = {
    "Content-Type": "application/json",
    "apikey": thetextapikey
}
mcp_url = _url + "most_common_phrases"

Get the Most Common Phrases from the Headlines

We have all the Obama based headlines saved in txt files from the post about how to get all the Obama headlines. The logic for getting the most common phrases from each file is the same so we’ll just loop through all the years from 2008 to 2017. Note from the last time that we only have the last two months of 2008 and January of 2017.

The first thing we’re going to do in this function is open the document we want. Note that we use an f-string in our with open() function. This is the new syntax for the “format” string and lets us directly inject variables into our string via brackets. With our opened files, we’ll read in each line to headlines. This will return a list of the headlines. After reading these in, we’ll join them into one string. Then we’ll set up a request body using this string. The num_phrases keyword defaults to 3, for this program, let’s ask for 10.

Now that we’ve built our request body and have the headers and endpoint from earlier, we can send our HTTP request to the API. We use the requests library to send a POST request to the endpoint with the body as a JSON. When we get the response back, we’ll use the json library to parse the string into a dictionary. Next, we’ll open up another file, this time in the obama folder, and write to it. Note that you’ll have to create the obama folder before you can write to this file. Finally, we loop through each entry in the most common phrases (no underscores) entry of the dictionary and write each entry to the file, ending each entry with a new line.

# loop through each year
for i in list(range(2008, 2018)):
    with open(f"obama_{i}.txt", "r") as f:
        headlines = f.readlines()
    # combine list of headlines into one text
    text = "".join(headlines)
   
    # set up request bodies
    body = {
        "text": text,
        "num_phrases": 10
    }
    # parse responses
    response = requests.post(url=mcp_url, headers=headers, json=body)
    _dict = json.loads(response.text)
    # save results to a txt
    with open(f"obama/{i}_mcp.txt", "w") as f:
        for mcp in _dict["most common phrases"]:
            f.write(mcp + '\n')

Most Common Phrases About Obama in Headlines from 2008 to 2017

Once we run our program, we’ll get our results of the 10 most common phrases used in Obama headlines during his presidency. Here they are.

Most Common News Phrases about Obama: 2008

  1. Obama Cabinet
  2. Obama Win
  3. Obama Choice
  4. Obama Deal
  5. Obama Education
  6. Obama Hopes
  7. Obama Report
  8. Obama Breach
  9. Obama Pledge
  10. Obama Slur

Most Common News Phrases about Obama: 2009

  1. Obama Bill
  2. Obama Policies
  3. Obama Effort
  4. Obama Trip
  5. Obama Budget
  6. Obama Inauguration
  7. Team Obama
  8. Obama Video
  9. Obama Lunch
  10. Obama Nominees

Most Common News Phrases about Obama: 2010

  1. Obama Speech
  2. Obama Agenda
  3. Obama Team
  4. Obama Visit
  5. Obama Diplomacy
  6. Obama Nominee
  7. Obama Family
  8. Obama Adviser
  9. Obama Advisers
  10. Obama Aides

Most Common News Phrases about Obama: 2011

  1. Obama Policies
  2. Obama Visit
  3. Obama Meeting
  4. Obama Ad
  5. Obama Aides
  6. Obama Jobs Plan
  7. Obama Re
  8. Obama Party
  9. Obama Turf
  10. Obama Vacation

Most Common News Phrases about Obama: 2012

  1. Obama Policy
  2. Obama Voters
  3. Obama Role
  4. Obama Supporters
  5. Obama Shift
  6. Obama Remark
  7. Obama Visit
  8. Obama Edge
  9. Obama Event
  10. Obama Allies

Most Common News Phrases about Obama: 2013

  1. Obama Plan
  2. Obama Agenda
  3. Obama Nominees
  4. Obama Advisers
  5. Obama Meeting
  6. Obama Trip
  7. Obama Call
  8. Obama Aide
  9. Obama Message
  10. Obama Panel

Most Common News Phrases about Obama: 2014

  1. Obama Trip
  2. Obama Plan
  3. Obama Effort
  4. Obama Allies
  5. Obama Appointment
  6. Obama Aide
  7. Obama Aides
  8. Obama Suit
  9. Obama Initiatives
  10. Obama Critics

Most Common News Phrases about Obama: 2015

  1. Obama Talk
  2. Obama Rules
  3. Obama Library
  4. Obama Victory
  5. Obama Programs
  6. Senator Obama
  7. Obama Order
  8. Obama Aide
  9. Obama Veteran
  10. Obama Operative

Most Common News Phrases about Obama: 2016

  1. Obama Speech
  2. Obama Effort
  3. Obama Trip
  4. Obama Nominees
  5. Obama Presidency
  6. Obama Veto
  7. Obama Agenda
  8. Obama Aides
  9. Obama Access
  10. Obama Strategy

Most Common News Phrases about Obama: 2017

  1. Obama Legacy
  2. Obama Policies
  3. Obama Regulations
  4. Obama Envoys
  5. Obama Watchdog
  6. Obama Health Plan
  7. Obama Health Law
  8. Obama Staff Members
  9. Trump Siege
  10. Barack Obamas Musical Legacy

I run this site to help you and others like you find cool projects and practice software skills. If this is helpful for you and you enjoy your ad free site, please help fund this site by donating below! If you can’t donate right now, please think of us next time.