Harnessing the Power of Redis for Efficient Trading Operations: A Detailed Look at Redis Pub/Sub and Stream #1

Introduction

While developing a trading robot that processes webhook messages from TradingView and places orders through the Interactive Brokers (IB) Insync API, I opted to use a message broker - Redis Pub/Sub. This decision stems from the convenience it offers in handling asynchronous operations. The two operations, i.e., receiving webhook messages and placing orders, can function independently, thus increasing efficiency.

Although it's feasible to place orders instantly upon receipt of a webhook, there's a risk of receiving multiple webhook messages from TradingView with the same timestamp based on PineScript. In addition, network or software issues may lead to the loss of webhook messages.

By integrating message brokers and their databases into the process, these potential issues can be effectively managed. In this scenario, Python Redis was chosen due to its compatibility with the IB Insync API and its dual functionality as a message broker and a NoSQL database.

Redis as a Message Broker and Time Series Store

Redis provides two key features as a message broker - Redis Pub/Sub and Redis Stream. My initial exploration was directed towards Redis Stream, an append-only data structure apt for real-time messaging and acting as a message broker. It can also serve as a time series store, aligning well with the time series structure that PineScript utilizes.

Redis Stream, in a unique way, can be seen not just as a messaging system, but as a time series store. This feature dovetails with the core functionality of Pine Script, which is designed to process time series efficiently. Time series are not merely a form or a type; they're the fundamental structure Pine Script uses to store successive variable values over time.

Redis Stream automatically timestamps data in milliseconds, a feature I leveraged to measure delay. This allowed for accurate timing when the Flask server received messages from TradingView, pushed them into Redis Stream, and when the stream listeners fetched the data from Redis.

Redis Pub/Sub vs. Redis Stream

In addition to storing time information, Redis Stream offers a significant advantage over Redis Pub/Sub - data retrieval in case of disruptions. For instance, if the trading bot application (TBOT) experiences hang-ups or network interruptions, Redis Stream can still provide unhandled data as soon as the stream is opened. This ensures no data is lost, a crucial feature when running TradingView indicators or strategies over longer timeframes.

Conversely, with Redis Pub/Sub, if the message listener doesn't participate in the channel when alert messages arrive, it may result in lost webhook messages. From the perspective of reliable and robust webhook messaging, Redis Stream undoubtedly has an edge.

The Trade-Off of Using Redis Stream

Despite its benefits, Redis Stream does involve some overhead in message passing. In my testing on an Amazon EC2 T2.Small instance, I observed a delay of approximately 100 milliseconds more between the sender and listener when compared to Redis Pub/Sub. It's important to note that this latency might vary depending on various factors, including the specifics of your setup and network conditions. If this delay is crucial, Redis Pub/Sub might be a more suitable choice.

To accommodate this, I've made TBOT on TradingBoat configurable through environment variables. Users can choose between Redis Pub/Sub or Redis Stream by setting TBOT_USES_REDIS_STREAM to True or False.

Configuration Variables

The full source of the TBOT application can be found on https://github.com/PlusGenie/tbot-tradingboat

For detailed instructions on setting up a simple and fast trading robot using Docker and TradingView, please refer to the following Udemy course: Link to Udemy Course