Updated 6:19pm PST 11/29/2021 – Our sentiment prediction was right! Next step is to predict how much it’ll go up.

Recently I’ve been playing around with sentiment analysis on Tweets a lot. I discovered the Twitter API over the Thanksgiving holidays and it’s like Christmas came early. Sort of like how Christmas comes earlier to malls every year. I have applied for upgraded access already because it’s been so fun and I’m hoping they grant it to me soon. My sentiment analysis of Black Friday on Twitter was quite popular, getting over 400 views this last weekend! Other than just holidays though, I wanted to see if analyzing Twitter sentiment could be used for a real life application. THIS IS NOT FINANCIAL ADVICE AND SHOULD NOT BE TAKEN AS SUCH.

I decided that hey, I like to play with Twitter’s API, Natural Language Processing, and stocks, why not see if I can combine all of them? Thus, the idea for using Twitter sentiment to predict stocks was born for me. I’ve always been a big fan of Starbucks, both the actual store and the stock. #SBUX has made me a lot of money over the past couple years. So this post will be about the Starbucks stock and seeing how Twitter does in predicting its performance for just one day. Click here to skip directly to the results.

This project will be built with two files. You’ll need access to not only the Twitter API linked above, but also to get a free API key from The Text API to do the sentiment analysis part. Make sure you save the Bearer Token from Twitter and your API Key from The Text API in a safe place. I stored them in a config file. You’ll also need to install the requests library for this. You can install that in the command line with the command below:

pip install requests

Using the Twitter API to Get Tweets About Starbucks

As we always do, we’ll get started by importing the libraries we need. We’ll need the requests library we installed earlier to send off HTTP requests to Twitter. We will also need the json library to parse the response. I have also imported my Twitter Bearer Token from my config here. As I said above, you may choose to store and access this token however you’d like.

import requests
import json
from config import bearertoken

Once we’ve imported the libraries and the Bearer Token, we’ll set up the endpoint and header for the request. You can find these in the Twitter API documentation. We need to use the recent search endpoint (only goes back last 7 days). The only header we need is the Authorization header passing in the Bearer token.

search_recent_endpoint = "https://api.twitter.com/2/tweets/search/recent"
headers = {
    "Authorization": f"Bearer {bearertoken}"
}

Creating Our Twitter Search Function

Everything to search the Twitter API is set up and ready to go. The reason we declared the headers and URL outside of the function is because they may be able to be used in a context outside of the function. Now let’s define our search function. Our search function will take one parameter – a search term in the form of a string. We will use our search term to create a set of search parameters. In this example, we will create a query looking for English Tweets with no links and are not retweets that contain our term. We are also going to set the maximum number of returned results to 100. 

With our parameters, headers, and URL set up, we can now send our request. We use the requests module to send a request and use the json module to parse the text of the returned response. Then we open up a file and save the JSON to that file.

# automatically builds a search query from the requested term
# looks for english tweets with no links that are not retweets
# saves the latest 100 tweets into a json
def search(term: str):
    params = {
        "query": f'{term} lang:en -has:links -is:retweet',
        'max_results': 100
    }
    response = requests.get(url=search_recent_endpoint, headers=headers, params=params)
    res = json.loads(response.text)
    with open(f"{term}.json", "w") as f:
        json.dump(res, f)

Once we’ve set up the search function, we simply prompt the user for what term they’d like to search for and then call our search function on that term.

term = input("What would you like to search Twitter for? ")
search(term)

When we run our program it will look like this:

The saved JSON file should look something like this:

Analyzing the Tweets for Average Sentiment

Now let’s get into the second part, the fun part, the part you’re all here for, the sentiment analysis. As I said earlier, we’ll be using The Text API for this. If you don’t already have a free API key, go over to the site and grab one. We won’t be needing any Python libraries we didn’t already install for using the Twitter API so we can dive right into the code. I created a second file for this part to follow the rule of modularity in code. You can opt to do this in the same file and you’ll only need the third import – The Text API key.

Setting Up and Building the Request

As always, we’ll want to get started by importing our libraries. Just like before we’ll need the requests and json library and we will do the same thing with them as above – sending the HTTP request and parsing the response. For this file we’ll import our API key from The Text API instead of the Twitter Bearer Token. 

We have a couple other differences as well. The URL endpoint we’ll be hitting is going to be the polarity_by_sentence URL. The headers we need to send will tell the server that we’re sending JSON content and also pass in the API key through an apikey keyword.

import requests
import json
from config import text_apikey
 
text_url = "https://app.thetextapi.com/text/"
polarity_by_sentence_url = text_url+"polarity_by_sentence"
headers = {
    "Content-Type": "application/json",
    "apikey": text_apikey
}

Just like we did with the Twitter API, we’ll need to build a request for The Text API. Our build_request function will take in a term in the form of a string. We’ll use this term to open the corresponding JSON file. Then we’ll combine all the text from the tweets to form a final text string that we will send to The Text API to be analyzed. Finally, we’ll create a body in the form of a JSON that we will send to the endpoint and return that JSON body.

# build request
def build_request(term: str):
    with open(f"{term}.json", "r") as f:
        entries = json.load(f)
 
    text = ""
    for entry in entries["data"]:
        text += entry["text"] + " "
 
    body = {
        "text": text
    }
    return body

Getting the Average Sentiment

Okay so here we’re actually going to get the average Text Polarity, but that’s about synonymous with sentiment. Text polarity tells us how positive or negative a piece of text was, sentiment is usually used in the same way. Let’s create a polarity_analysis function that will take in a dictionary as a parameter. The dictionary input will be the JSON that we send as the body of the request to the polarity_by_sentence endpoint. Once we get our response back and parse it to get the list of polarities and sentences, we can calculate the average polarity. For an idea of what the response looks like, check out the documentation.

Once we have the response, all we have to do is calculate the average polarity. The way the response is structured, the first element of an entry is the polarity and the second is the sentence text. For our use case, we just care about the polarity. We are also going to ignore neutral sentences because they’re entirely useless to and don’t affect whether the overall outcome will be positive or negative. They could affect the absolute value of the outcome, and maybe we could take that into account, but as long as we approach these problems with the same method each time, it won’t matter.

# get average sentence polarity
def polarity_analysis(body: dict):
    response = requests.post(url=polarity_by_sentence_url, headers=headers, json=body)
    results = json.loads(response.text)["polarity by sentence"]
    # initialize average polarity score and count
    avg_polarity = 0.0
    count = 0
    # loop through all the results
    for res in results:
        # ignore the neutral ones
        if res[0] == 0.0:
            continue
        avg_polarity += res[0]
        count += 1
    # average em out
    avg_polarity = avg_polarity/count
    print(avg_polarity)

Twitter Sentiment on Starbucks Over Thanksgiving Weekend 2021

When we run this program we’ll get something that looks like the following. I’ve pulled Tweets about Starbucks from Saturday, Sunday, and Monday morning (today) and renamed the files from their original names (starbucks.json).

Looks like Sunday was a little less positive than Saturday or Monday, but overall the Twitter sentiment towards Starbucks is positive. I predict the stock price will go up today. Let’s see how much.

I run this site to help you and others like you find cool projects and practice software skills. If this is helpful for you and you enjoy your ad free site, please help fund this site by donating below! If you can’t donate right now, please think of us next time.