Challenge
The main objective of the project was the real-time processing, analysis, and presentation of a large volume of sports betting data from various source systems through a predefined report.
For data storage and management, the choice was made to use Delta Lakehouse technology based on the client’s requirements. This modern solution combines the structured data management of traditional data warehouses with the flexibility and scalability of data lakes. Delta Lakehouse allows for the simultaneous storage and analysis of structured and unstructured data, simplifying the handling of complex sports betting data.
Despite the billions of records of sports betting data, the system must be capable of providing extremely low response times during report execution.
The project required close collaboration with international partners and service providers to ensure its success. As part of this collaboration, we developed common data formats and communication protocols that enable seamless data exchange among partners operating in different countries.
How we helped?
- Data Source Identification
- The system can ingest data from various sources, including sports betting providers' APIs, historical databases, and external data providers.
- Data Collection
- Using RabbitMQ, we receive real-time data streams, ensuring continuous and reliable data flow from sports betting providers.
- Processing
- The Apache Spark cluster enables parallel processing of large volumes of data, ensuring the system’s scalability and performance.
- Storage
- Delta Lake technology is suitable for storing both structured and unstructured data, providing version control and ensuring ACID transaction properties.
- Analysis and Reporting
- Through the Azure SQL layer, complex queries and analysis can be performed, generating accurate and up-to-date reports for regulatory authorities.