High throughput data gathering API
High throughput data gathering API
Python | FastAPI | Kafka | Postgresql | Angular
Problem:
Our client had a need for collection of regulatory data, in order to validate the data from their customers. System needed to be designed as fail-safe, resilient and robust web application that can withstand high volume of incoming data. Taking into account that the data came from multiple customers and was in high frequency, we also needed to have inconsistency checks and reporting capabilities.
Solution:
After thorough analysis of the client’s business model, needs and processes, our team came up with the solution where we used highly scalable, idempotent, asynchronous processes in order to insure zero to none data loss and high availability.
In practical terms, through a unified and standardized API, client’s customers were required to develop new mechanisms in accordance to specific standards set by the client. Through our custom made UI, client had customer data reports and was aware of any possible data inconsistencies, based by set of rules defined by the client.
Technology stack:
Python | FastAPI | Kafka | Postgresql | Angular