SHiFTBot is an umbrella term that encompasses the collection of scripts that power the SHiFT & VIP Code archive as well as a variety of Twitter accounts. It began as a single PHP Script in 2012 and has evolved into a mix of Python, Bash, PHP and Webhook scripts. None of these scripts are public as they’re highly customized for their operating environments but below is a general outline of how it all works.

SHiFTBot.py

This is the primary script that automates a lot of the behavior of the website and Twitter accounts. I refer to it as a “He” because it can be pretty tempermental and really stupid at times but tries really hard. SHiFTBot has gone through a lot of interations over the years but as of 2019 he exists as a Python 3 script running on Debian.

Every 5 minutes he is asked to go out and check the timeline of some pre-configured accounts stored in an SQLite database. Using regular expressions he looks through the last 10 Tweets for each account searching for SHiFT Codes, Rewards and Expirations. It’s a pretty mundane operation with boolean matches but he can also perform partial matches to increase his search result confidence. Once a code is found he must then determine if it is new or not. To do this he checks his own internal database of codes and if it has not been cached he then takes it a step further and checks dgSHiFTCodes to see if maybe I, the human, have already tweeted it out.

If a code is new, uncached and not manually tweeted then he has to check what configuration mode he is running in. If he is 100% sure this is a new code and he’s in retweet mode then he’ll create a Website Archive, publish it and then share the tweet on Twitter. However, if he’s running in notification mode or only has a partial match then a Pushover Notification will be sent for manual review. This helps reduce the amount of unnecessary timeline spam and arises directly from an issue in 2017 where an older version of the bot decided that anything about ducks was worth a retweet. True story, it was really weird.

SHiFTBot-VIP.py

This auxillary script performs almost all the same operations as the parent script however it only checks for VIP (Vault Insider Program) codes. The matching is a lot more looser as generally these codes are published in long english sentences meaning it’s harder for the bot to detect. When a new VIP Code is found, much like its parent script, it checks for tweet/notification mode and operates as requested by either sending a new Tweet or just sending a notification for human interaction.

SHiFTBot-VIP runs every 5 minutes as well.

SHiFTBot-BL3.py

In October of 2019 it became increasingly obvious that a large portion of new followers were only interested in Borderlands 3 codes. During release these same SHiFT Codes would often expire in a matter of hours making Twitter’s curated notification system impossible to work with. To further complicate matters if you enabled account notifications you would get notifications for every game and VIP Code even if you only wanted Borderlands3 codes.

This is what SHiFTBot’s little brother SHiFTBotBL3 was created for. His job is to run exactly 1 minute after his big brother and then check the primary timeline specifically for Borderlands 3 SHiFT Codes. If a code is found, retweet it to his own account dgSHiFTCodesBL3 . That’s it, that’s all.

Now followers can enable account notifications specifically for this account and ONLY get alerted when Borderlands 3 SHiFT Codes are published. It is the best of both worlds for everyone involved. Followers get specifically what they want, and I get a lot less complaining when I publish codes for older games, VIP points or community giveaways.

SHiFTBot-FAQ.py

In January of 2020 the SHiFTBot-FAQ script was added to the suite. This script monitors the Twitter timeline in real-time and looks for Keyword Triggers sent from the human. When a trigger is detect it parses the configured reply and sends it back. This is useful for some Frequently Asked Questions as well as for dynamically pulling back data about the most recent codes or links to the site.

SHiFTBot-Giveaway.py

When a giveaway is set up on Twitter, this script is what is used to monitor submissions and randomly pick a winner. Prior the script’s creation I was paying a 3rd-party service betwee $10 and $20 everytime a giveaway was run. This script is not only saving money but also demystifies the process of picking a winner. Previously I had to trust the service to function while now I know exactly how winners are picked and how the Twitter API is utilized.

SHiFTBot WebUI (Private)

Another portion of the SHiFTBot Suite is the WebUI. This password protected, unlisted, private site allows for the remote management of everything SHiFTBot Related and untethers all the management operations from an in-house laptop. With the mobile-first WebUI SHiFTBot can toggle notification/retweet modes as well as manually create SHiFT and VIP code archives on the public website.

The WebUI consists of a series of PHP 7 scripts and classes running in Apache alongside a small BASH script to push commits to the remote Git repository. It is designed to run independantly of the public facing site utilizing Github as the deployment middleman.

Public Site

The public website (where you’re reading this) has gone through 3 rewrites since being released in 2012. Originally it was a place to sign-up for email notifications, included just 3 RSS Feeds and was written in PHP 5.6. However the email service became cost prohibitive very rapidly and new sign-ups were disabled within the first few months. The site remained as is for the next 5 years with only a Twitter cache updating hourly.

In 2017 the site was rewritten in PHP 7 for security purposes and it was greatly simplified. For all intents and purposes it was really just a mirror of the Twitter account. This all changed in 2019 when I decided to spend some time building out the site to be a proper SHiFT & VIP Code archive.

My primary concern was keeping the site fast and lightweight. By this time the Twitter account’s followers had exploded and I was worried about how expensive it might get to have a dynamic PHP enabled site running. I figured that since there are rarely more than a code or two a day it would be smarter to create a static site that updates on demand rather than dynamically generate content on every page load.

This could reduce my server costs and allow for aggressive cacheing policies. I opted to go with Hugo which is a static site generate written in Go. With a bit of a learning curve I was able to customize it suit the websites needs and integrate near seamlessly with the existing SHiFTBot infrastructure.

Keeping inline with the “speed-is-king” mentality I set up a seperate virtual machine with nothing more than NGinX and Git installed to keep the site lightning fast and scalable. As demand for the site grows load balances and additional cloud VMs can be added with relative ease.

GitHub & Webhooks

To marry the public and private websites together I utilize a private Github repository along with Webhooks to setup a continuous deployment path. When a new commit is pushed to Github a webhook is triggered that reaches out to the public server and informs it that new content is available. If the signatures pass authentication then a localized BASH script runs to synchronize the local repository and request that Hugo recompile the website. Once Hugo has recompiled the site then RSync is activated and new or changed content is copied to the public path that powers this website.

Local Development

My local development environment for managing SHiFTBot is a KUbuntu 18.04 desktop with PyCharm, PHPStorm and Insomnia Rest. I have an inhouse hypervisor running ProxMox that powers a series of virtual machines and containers containing Apache, NGinX, Gitea and Dokuwiki amungst other non-SHiFTBot related resources.

The website is written with a mobile-first / responsive approach so testing is done primarily though resizing Chromium and Firefox browsers then loading an iPhone and android device to ensure functioning mobile displays.

Technology Breakdown

  • Languages: Python 3, PHP 7, Bash 5
  • Platforms: Debian, Proxmox, KUbuntu
  • Servers: Apache, NGinx, Webhooks
  • Utilities: Hugo, Git, RSync, SQLite3, Certbot
  • Code Hosting: Gitea, Github
  • Libraries: Tweepy, DateParser, GitPython, DateUtil
  • IDE: PyCharm, PHPStorm, Insomnia
  • Other: TweetDeck, Pushover, Dokuwiki
  • Hosting: Digital Ocean

Useless Statistics

  • SHiFTBot.py : 881 lines (+143 Config)
  • SHiFTBot-VIP.py : 651 line (+72 Config)
  • SHiFTBot-BL3.py : 351 lines (+11 Config)
  • SHiFTBot-FAQ.py : 386 lines (+182 Config)
  • SHiFTBot-Giveaway.py : 401 lines (+18 Config)
  • SHiFTBot WebUI : 1479 lines
  • Public Website : 781 files
    • Avg hits/month ~3 Million
    • Avg page paint 1.82 seconds
    • Avg page load 0.52 seconds
    • Avg Hugo compile 1.097 seconds
    • 77.2% of all vistors are mobile
    • 63.6% of all vistors are american
  • Twitter @dgSHiFTCodes
    • Avg replies/day: 23.8
    • Avg top-level tweets/day: 1.4
    • Avg top-level engagement: ~32,000/24h
    • Avg top-level impression: ~350,000/24h
  • Orcicorn
    • Anxiety pills per day: 1
    • Disdain for Justin: Infinite