Scrape the Text from All Tweets for a Search Term

Twitter is such a treasure trove of interesting information. Recently, I’ve been using Twitter sentiment to predict NFL games. We can also see if we can use Twitter sentiment to predict the stock market. In this post, we’re going to go over how you can get all of the text from the last 100 Tweets for a search term using Python. 

In this post we’ll cover:

  • Creating Twitter Request Headers
  • Creating a Function to Search Twitter
    • Configuring the Twitter Search Parameters
    • Parsing the Response
  • Testing the Twitter Search Function

To get started you’re going to need a Twitter API key by signing up for a developer account on Twitter. You’ll also need to download the `requests` module with the line below in your terminal.

pip install requests

Create Twitter Request Headers

The first thing we need to do is establish how we’re going to send our request to Twitter. Twitter provides some Python libraries, but none that have good documentation or are well maintained. Let’s just do it the old fashioned way.

First, we’ll import `requests` to send HTTP requests and `json` to parse the response. We also need the Bearer Token, which we should have gotten via the Twitter API earlier. We’ll set up the search endpoint, which will be the “recent” search endpoint. Then, we’ll set up the headers. The only headers we need is the `Authorization` header which will pass in the Bearer Token.

import requests
import json
 
from twitter_config import bearertoken
 
search_recent_endpoint = "https://api.twitter.com/2/tweets/search/recent"
headers = {
    "Authorization": f"Bearer {bearertoken}"
}

Create Twitter Search Function

After we’ve set up the headers, we need to create a function to search Twitter. In our example, we’ll create a function that searches Twitter for the last 100 tweets that are a) in English, b) don’t have links, and c) are not retweets.

Configure Twitter Search Parameters

Our Twitter `search` function will take one parameter, the search term. We expect the search term to be a string. The first thing we’ll do is configure our params for the search request. The params will construct a query for the search `term` and specify that we want only English results that don’t have links nor are retweets. After setting up the query, we will also specify that we want a maximum of 100 results.

# automatically builds a search query from the requested term
# looks for english tweets with no links that are not retweets
# returns the tweets
def search(term: str):
    params = {
        "query": f'{term} lang:en -has:links -is:retweet',
        'max_results': 100
    }

Parse Response

Next, we’ll use the headers and the params we just set up to send a GET response to the endpoint. We’ll use the `json` module to parse the response we get. Remember that we want to return all the Tweets as a string. The next thing we’ll do is get all the Tweets from the `data` key. Then we’ll join all of the `text` from each of the Tweets with a period and a new line. Finally, we’ll return that string.

    response = requests.get(url=search_recent_endpoint, headers=headers, params=params)
    res = json.loads(response.text)
    tweets = res["data"]
    text = ".\n".join( for tweet in tweets])
    return text

Full Code

Here’s the full code for the function to search Twitter and return the text from the Tweets.

# automatically builds a search query from the requested term
# looks for english tweets with no links that are not retweets
# returns the tweets
def search(term: str):
    params = {
        "query": f'{term} lang:en -has:links -is:retweet',
        'max_results': 100
    }
    response = requests.get(url=search_recent_endpoint, headers=headers, params=params)
    res = json.loads(response.text)
    tweets = res["data"]
    text = ".\n".join( for tweet in tweets])
    return text

Test Twitter Search Functionality

Since it’s December 26, which is Boxing Day in Canada, but the day after Christmas in the US, we’ll search Twitter for Christmas.

print(search("christmas"))

The results will show us the 100 most recent tweets people are making that contain the word “Christmas”.

Learn More

To learn more, feel free to reach out to me @yujian_tang on Twitter, connect with me on LinkedIn, and join our Discord. Remember to follow the blog to stay updated with cool Python projects and ways to level up your Software and Python skills! If you liked this article, please Tweet it, share it on LinkedIn, or tell your friends!

I run this site to help you and others like you find cool projects and practice software skills. If this is helpful for you and you enjoy your ad free site, please help fund this site by donating below! If you can’t donate right now, please think of us next time.

Yujian Tang
Yujian Tang

I started my professional software career interning for IBM in high school after winning ACSL two years in a row. I got into AI/ML in college where I published a first author paper to IEEE Big Data. After college I worked on the AutoML infrastructure at Amazon before leaving to work in startups. I believe I create the highest quality software content so that’s what I’m doing now. Drop a comment to let me know!

One-Time
Monthly
Yearly

Make a one-time donation

Make a monthly donation

Make a yearly donation

Choose an amount

$5.00
$15.00
$100.00
$5.00
$15.00
$100.00
$5.00
$15.00
$100.00

Or enter a custom amount

$

Your contribution is appreciated.

Your contribution is appreciated.

Your contribution is appreciated.

DonateDonate monthlyDonate yearly

3 thoughts on “Scrape the Text from All Tweets for a Search Term

Leave a Reply

%d bloggers like this: