Twitter is such a treasure trove of interesting information. Recently, I’ve been using Twitter sentiment to predict NFL games. We can also see if we can use Twitter sentiment to predict the stock market. In this post, we’re going to go over how you can get all of the text from the last 100 Tweets for a search term using Python. 

In this post we’ll cover:

• Creating Twitter Request Headers
• Creating a Function to Search Twitter
  • Configuring the Twitter Search Parameters
  • Parsing the Response
• Testing the Twitter Search Function

To get started you’re going to need a Twitter API key by signing up for a developer account on Twitter. You’ll also need to download the `requests` module with the line below in your terminal.

pip install requests

Create Twitter Request Headers

The first thing we need to do is establish how we’re going to send our request to Twitter. Twitter provides some Python libraries, but none that have good documentation or are well maintained. Let’s just do it the old fashioned way.

First, we’ll import `requests` to send HTTP requests and `json` to parse the response. We also need the Bearer Token, which we should have gotten via the Twitter API earlier. We’ll set up the search endpoint, which will be the “recent” search endpoint. Then, we’ll set up the headers. The only headers we need is the `Authorization` header which will pass in the Bearer Token.

import requests
import json
 
from twitter_config import bearertoken
 
search_recent_endpoint = "https://api.twitter.com/2/tweets/search/recent"
headers = {
    "Authorization": f"Bearer {bearertoken}"
}

Create Twitter Search Function

After we’ve set up the headers, we need to create a function to search Twitter. In our example, we’ll create a function that searches Twitter for the last 100 tweets that are a) in English, b) don’t have links, and c) are not retweets.

Configure Twitter Search Parameters

Our Twitter `search` function will take one parameter, the search term. We expect the search term to be a string. The first thing we’ll do is configure our params for the search request. The params will construct a query for the search `term` and specify that we want only English results that don’t have links nor are retweets. After setting up the query, we will also specify that we want a maximum of 100 results.

# automatically builds a search query from the requested term
# looks for english tweets with no links that are not retweets
# returns the tweets
def search(term: str):
    params = {
        "query": f'{term} lang:en -has:links -is:retweet',
        'max_results': 100
    }

Parse Response

Next, we’ll use the headers and the params we just set up to send a GET response to the endpoint. We’ll use the `json` module to parse the response we get. Remember that we want to return all the Tweets as a string. The next thing we’ll do is get all the Tweets from the `data` key. Then we’ll join all of the `text` from each of the Tweets with a period and a new line. Finally, we’ll return that string.

    response = requests.get(url=search_recent_endpoint, headers=headers, params=params)
    res = json.loads(response.text)
    tweets = res["data"]
    text = ".\n".join( for tweet in tweets])
    return text

Full Code

Here’s the full code for the function to search Twitter and return the text from the Tweets.

# automatically builds a search query from the requested term
# looks for english tweets with no links that are not retweets
# returns the tweets
def search(term: str):
    params = {
        "query": f'{term} lang:en -has:links -is:retweet',
        'max_results': 100
    }
    response = requests.get(url=search_recent_endpoint, headers=headers, params=params)
    res = json.loads(response.text)
    tweets = res["data"]
    text = ".\n".join( for tweet in tweets])
    return text

Test Twitter Search Functionality

Since it’s December 26, which is Boxing Day in Canada, but the day after Christmas in the US, we’ll search Twitter for Christmas.

print(search("christmas"))

The results will show us the 100 most recent tweets people are making that contain the word “Christmas”.

I run this site to help you and others like you find cool projects and practice software skills. If this is helpful for you and you enjoy your ad free site, please help fund this site by donating below! If you can’t donate right now, please think of us next time.