Building a twitter bot

 , ,

Andy Herd, 15th June 2020

A flurry of really upsetting stuff aimed at transgender and non-binary people on social media got me thinking: I’m tired of this. How can I contribute some love and solidarity to this debate?

So I set out to build a twitter bot.

TL;DR – I built a thing! You can follow my little bot on twitter @transnb. The code is here github/herdingdata/transnb. This page talks about the process of building it and some stuff I learned along the way. If you’re interested in building something similar, I hope this helps you to get started.

Side note: I won’t go into the details of the awfulness of this social media “event” which prompted the whole idea except for to mention this response by the charity Mermaids.

Getting set up

Coincidentally, on the same day as I was thinking of ideas of nice things to tweet, the very talented @smolrobots drew a nice robot. They were kind enough to let me use it as my profile pic. If you like it as much as I do, you can purchase their merchandise here.

With my brand new twitter account @transnb set up and ready to tweet, the next task was to request a developer account on twitter. This is necessary to allow you to generate the api keys to be able to authenticate with your app later. It’s worth starting this application before writing any code as it can take a day or so for twitter to look over your application and perhaps ask follow up questions. https://developer.twitter.com/en/apply-for-access

The code

I used Python. It’s what I know.

I recommend using a virtual environment to contain your installation. That way you can have different versions of python and versions of libraries for different projects. To get started:

  1. Download python (latest version is 3.8.3 at the time of writing) https://www.python.org/downloads/. On my macOS machine it’s installed to /usr/local/bin/python3.8. Your mileage may vary.
  2. Download virtualenv and virtualenvwrapper. The use of virtualenvwrapper (which gives us the mkvirtualenv command) is not necessary. There are loads of ways to create virtual envs. This is just the one I find easiest to get up and running with. https://virtualenvwrapper.readthedocs.io/en/latest/install.html

Once you have that installed, you can do the following steps from your favourite terminal to create your development environment. I use macOS & bash so the commands may vary a little if you have a different setup, but hopefully this gives you an idea of what the steps are that we’re trying to achieve.

Note: these steps are in the readme. If ever the readme deviates from the steps below, follow the readme because the walkthrough below probably got a little stale.

# clone the git repo
cd /wherever/you/want/to/save/it
git clone git@github.com:herdingdata/transnb.git

# change directory to the new project folder
cd transnb

# make a virtual environment, specifying its name as "transnb" and also the python version
mkvirtualenv transnb -p /usr/local/bin/python3.8

# install all the libraries and dependencies
# `make` commands are GNU Make, see the file named Makefile for more
make setup

# I'm using black & isort so let's install a commit hook for the project
# to make sure that we don't accidentally commit anything which breaks the formatting rules
make install-git-hooks

There are a few different elements to my code.

  • commands.py does the tweeting
  • time.py does some statistical stuff to get random intervals
  • messages.py has all the magic behind constructing variants on the messages

Commands

src/transnb/commands.py

The library click allows us to link up python scripts to become actual commands. Without click, running python might look something like:

/usr/local/bin/python3.8 src/transnb/commands.py

With click, and our virtualenv activated, we can type any one of the following commands:

  • transnb-tweet will send a tweet
  • transnb-all shows the output of all the messages
  • transnb-analyse some analysis of how long it takes to go through all the messages and how random the choices are

An important requirement for click to work is declaring entry_points in setup.py. This is how click joins the dots between the command you type and which code to run:

    entry_points="""
        [console_scripts]
        transnb-tweet=transnb.commands:tweet
        transnb-all=transnb.commands:all_messages
        transnb-analyse=transnb.commands:analyse
    """

The library tweepy lets us interact with twitter.

Messages

src/transnb/messages.py
(Spoiler alert: all the messages my bot will tweet are in here. If you’d rather enjoy them as they trickle onto twitter, don’t click this link)

With tweepy, sending tweets is actually pretty easy, but now to think of things to say! Turns out this is the most time consuming part of building a twitter bot.

In the messages module I set up some templates of things which can be iterated through. For example, take the phrase “Trans men are men”. One way to do this is through listing out lots of statements individually. 

MESSAGES = (
    "Trans men are men",
    "Trans women are women",
    "etc...",
)

However I want an element of randomness so I don’t have to write everything explicitly. I created a template dataclass and some string substitution. This allowed me to build up lots of templates with variations to mix it up a little. Extracts from messages.py:

from dataclasses import dataclass


@dataclass
class Msg:
    """Template for messages"""
    text: str
    substitution: tuple = None


GENDER_PLURALS = ("men", "women", "people")
MESSAGES = (
    Msg("Trans {s} are {s}.", GENDER_PLURALS),
)


def get_all_messages() -> tuple:
    messages = []
    for msg_template in MESSAGES:
        if msg_template.substitution is None:
            messages.append(msg_template.text)
        else:
            substitutions = msg_template.substitution
            for sub in substitutions:
                messages.append(msg_template.text.format(s=sub))
    return tuple(messages)

The built in python library secrets lets us choose a tweet at random.

import secrets

def get_random_message() -> str:
    return secrets.choice(get_all_messages())

Time

src/transnb/time.py

I want my bot to post stuff at intervals which appear random. That happens here.

  • The function do_i_post_a_tweet uses a random number generator with a set probability to sometimes return True, sometimes False.
  • If run on a cron job every seven minutes */7 * * * * then it will, on average, tweet 7 times a day. The tweets per day rate can be easily tweaked.

Tests

src/transnb/tests

I can’t have a python library without tests! Also test driven development is fun. I was constantly running these tests as I was writing the code.

  • make messages exports all the possible messages to tests/expected_message_list.txt 
  • In test_all_messages.py:
    • test__all_messages__match_expected goes through all the messages which are returned from messages.py
    • test_all_messages__none_more_than_200chars does what it says on the tin. I know I can stretch to 240 chars if I want but 200 feels right.
  • test_integration.py runs a command to check that we can see twitter. It won’t work until we give python some credentials to be able to speak to twitter.

Credentials and sending my first tweet

It’s really important not to save the credentials in github. This could expose them to the world and allow anyone to interact with my account. So I decided to keep them in a file called settings.py.

To make sure I don’t expose the credentials accidentally, I added a line to the .gitignore to make sure this file cannot be affected in any git commit. That way I know that on any new computer it won’t be able to speak to twitter until I set up these credentials.

# Create credentials file
cp src/settings_local.py src/settings.py

By this point I had received a developer account on twitter. I created the “app” by following the steps here https://developer.twitter.com/en/apps. Then I was able to access credentials from twitter which I copied into the new settings.py file. It looks something like this:

CONSUMER_KEY = "REPLACEME"
CONSUMER_SECRET_KEY = "REPLACEME"
ACCESS_TOKEN = "REPLACEME"
ACCESS_TOKEN_SECRET = "REPLACEME"

With that set up, I manually triggered the post tweet function… and it worked!

transnb-tweet --message "hello, world"

Alright it works, let’s get it posting regularly

For the task of running my bot regularly I need to host it on a server somewhere. A t2.micro or t3.micro AWS EC2 instance would probably just about stay inside the free AWS tier. Even better would be a cronjob triggering a lambda function or something so that it’s only running for the duration it’s needed and not idling the rest of the time.

However I already have a raspberry pi 4 which runs pi-hole on my home network. It seemed like a good choice considering that it’s always running anyway.

The full steps are listed in the readme. Generally the things we’re accomplishing here are:

  1. Get python3.8.
  2. Clone our github repo onto the pi.
  3. Repeat the credentials step here, too.
  4. make test will run the tests and also make sure that that our server can speak to twitter.
  5. Create a cron job which will trigger our bot every 7 minutes

There’s one thing worth highlighting on my cron job. I received some annoying messages in /var/log/syslog which coincided with the times the cron job triggered. No MTA installed, discarding output. This message is because the pi was trying to email the output, but it didn’t have an email server installed. One approach would have been to install postfix. However after a bit of digging I discovered that I can pipe the output into | logger and my syslog shows me the messages. My crontab looks like this:

*/7 * * * * /path/to/.virtualenv/transnb/bin/transnb-tweet | logger

Edit April 2023 – Adding mastodon support to my bot

We all know that a man who hasn’t been told to f*ck off enough during his time on this green earth purchased twitter and promptly set about vandalising the community resource relied upon by millions. Turns out he is competent at one thing: destroying twitter.

Anyway an email announcing that my API access was about to get turned off was the nudge I needed to repoint my bot to post to mastodon instead. So I created an account in the mastodon instance botsin.space. https://botsin.space/@transnb

It was quite easy to change the code as some very nice open source people had already ported the library I was using (tweepy) to tweepy_mastodon. https://pypi.org/project/tweepy-mastodon/

If you’re curious what I changed, you can see the pull request here. https://github.com/herdingdata/transnb/pull/3

Conclusion

Building a twitter mastodon bot is a fun little project. I highly recommend giving it a go! If you do, I hope you find this walkthrough helpful.