Promptpal
A microservices and an event-driven architecture for a financial application, as a final project for Software Architecture.
Published onUpdated on
6.645
Since this is a project done as an assignment/project for university as the result of the course, there are parts I did not do, but there are parts that I did.
I’m definitely not well-versed in the finance, stocking fields so a lot of what I do is entirely up to my interpretation, and not the correct implementation. The main idea of this project is to provide a method for services to communicate with each other synchronously or asynchronously.
Overview
This project provides a set of backend services and a frontend service that gathers candles data from Binance through web sockets, store them in a reusable format in a database, as fodder for other services to consume.
The project uses RabbitMQ as the message broker for publishing and subscribing concepts and messages, as an event-driven mechanism.
Infrastructure
A list of services are setup in Docker Compose, mainly because of the following reasons:
- Ease of development: a lot of services can be rebuilt, redeployed locally for live testing purposes, and mounting sources for instant reloads.
- Easy deployment: in pair with development-based Dockerfile, there are originally planned for production-ready Dockerfiles and Compose (since production does not need code to reload every time, and minimizing images and storage usage).
- Consistent environment: all developers in the project can install Docker and get a similar development environment instantly.
Also, it’s because Compose is generally for a single-host multi-service system, which is a lot lot cheaper than deploying on an IaC set with Kubernetes Clusters or across mutliple VPSes with Docker Swarm.
Services
There are 4 contributors of this project, but 2 of them dropped out or got kicked off for actively sabotaging the project by not respecting Git workflows, pushing breaking changes and fully unchecked API-generated code that does not compile or run, requiring a lot of time to fix. So the work is split between two key members: me and Mr. Hoang Van Khai.
Thanks to the microservices setup, each service can use its own best language and/or framework to carry out the job the best, instead of locking into one vendor for the entire project.
Frontend Service
Technology Used: Vite + React, TypeScript, TailwindCSS.
Contributors:
- Me: everything.
Missions:
- Provide a general health check page that shows the health of all other services.
- Provide a chart of real-time candlesticks data, taken from data service.
- Provide a view into what was done by Crawlers and Analysis services.
- Provide users with a login/register page that interacts with the authentication service.
I did not like how the webpage turned out, as the two other contributors dropped out of the project on the last stretch week where a lot of things don’t work in integration yet. I had to design this with basic components as I did it, as well as debugging all CORS-related issues and React’s double-render shenanigans.
API Gateway
Technology Used: Golang, Go HTTP.
Contributors:
- Me: everything.
Missions:
- Provide the CORS settings for frontend to connect to. Every single request from the frontend must go through the API Gateway. This API Gateway also acts as a reverse proxy for all other services.
- Provide a mean for frontend to check health of all other services without access to them.
One thing I extremely love about this service using Go is that, primarily using Node with NPM or Python with PIP incur crazy storage usage (average 300MB for backend services, average 1GB for frontend services with SSR).
This Go image with the reverse proxy, is entirely distributed in a 20MB image.
Authentication Service
Technology Used: Node, JavaScript, ExpressJS.
Contributors:
- Khai: everything
- Me: SQL scripts & integrations
Missions:
- Provide a mean to access the database for user accounts, while also using a floating Redis cache for faster queries.
- Provide logging in and registering endpoints.
This is probably going to be repeated a lot. But I really dislike how Khải over-rely on AIs like GPT for his work. The commits reek of unchecked GPT code, and everything is in untyped and undocumented JavaScript, which made it extremely hard to scan over to know which parameters I need to pass for the reverse proxy.
Even the README documentation was generated by GPT without full context, so I could tell that what the GPT suggested won’t work because of outdated training data, and mainly the human not doing any due diligence.
Data Transformation Service
Technology Used: Node, JavaScript, Express.
Contributors:
- Khai: everything
- Me: SQL scripts & integrations
Missions:
- Provide a websocket for clients to connect to (after reverse proxy by API Gateway) to retrieve subscribed real-time candlesticks data.
- Save all retrieved data in a database.
Same thing as the Authentication Service. Everywhere is riddled with AI’s convoluated work. And from reading the code, I can tell that it does not adhere to Binance’s standards, but I couldn’t really test and prove that error.
Again, same with the Authentication Service, the README was also generated by
GPT fully. A lot of the code examples won’t work, since a lot of preconditions
have to be met, yet they are not documented anywhere. Parameters are full any,
I had to open my IDE to track invocations and still had a very hard time.
Crawler Service
Technology Used: Puppeteer, Node, TypeScript, Express.
Contributors:
- Me: everything
Missions:
- Periodically, or on command, run a crawler to crawl posts on certain forums or message boards related to financial topics. Only implemented crawlers for two websites: Reddit and StockTwits.
- Pipe the messages through a message broker to the Sentiment Analysis Service.
This was originally started by one of the dropouts, but she did so badly that it actively sabotaged the progress of the project. Khai used AI heavily, but it still ran, this one didn’t even.
At first, I wanted to do it with Python, but a lot of problems came up with trying to crawl DOM trees with Python since it’s untyped most of the times, so I swapped to Node with TypeScript.
Sentiment Analysis Service
Technology Used: Python, FastAPI, HuggingFace.
Contributors:
- Me: everything
Missions:
- Runs a consumer that takes in messages from the crawler service, manages it and runs it through a sentiment analysis pipeline, provided by HuggingFace.
- Saves the analyzed data into the database for querying later, as well as providing an endpoint to get back the analyzed data.
This was originally started by one of the dropouts, but she did so badly that it actively sabotaged the progress of the project. Khai used AI heavily, but it still ran, this one didn’t even.
At first, I wanted to do it with Python, but a lot of problems came up with trying to crawl DOM trees with Python since it’s untyped most of the times, so I swapped to Node with TypeScript.
Backtesting Service
Technology Used: JavaScript, Node, Express.
Contributors:
- Khai: everything
- Me: integrations
Missions:
- Used data saved in a database to test certain strategies such as moving average.
- Returned the tested data as a REST endpoint.
I think this is one of the least researched and least properly setup service one, and due to time constraints, I did not have enough time to research properly on backtesting strategies, and what they mean in Candles Data.
This service was created by patching together what GPT generated without checking, and he could not come up with any explanation about the input parameters, what to expect for a return value. Complete basic mistakes such as importing the wrong file with a typo were there.
Other Services
Technology Used:
- MongoDB: NoSQL database
- Mongo Express: Web-based Mongo management
- PostgreSQL: Main database of the project
- RabbitMQ: Message broker for all backend services
I originally wanted to also deploy this project up on my hosted cloud platform, but due to the quality of this project, I decided not to platform it as it’s a disgrace to what I stand for.
Conclusion
In conclusion, I was not that proud of the final result, and I sure as well earned the 70, and not the 90 I was expecting from the start of the course. I think it was a huge miss as this could be one of the projects you would be proudly hosting and displaying it in CVs.
This might have been a bad idea to cut ties, but I now know who to not team up with next time. I was immensely disappointed at the final result, but I’m proud of what I have done and could have done to salvage the project, which is why I graded it a Tier 4 (highest pride) on my projects list.