OVERVIEW
A Cloud infrastructure for a data-driven business.
BANOR SIM is one of the most important independent companies specialized in capital management and consultancy on large assets. The brokerage firm has launched a project to generate real-time buying and selling signals based on the collection of financial data through Data Ingestion services hosted on internal Data Centres.
BANOR SIM has entrusted beSharp with the creation of an Amazon Web Services Cloud infrastructure for Data Ingestion and predictive analysis. With the new infrastructure BANOR SIM aimed to obtain a Data Lake for analysis purposes based on archiving, historicization and high-performance processing of the huge amount of data coming from the feeder machines.
Managing, storing, and processing vast amounts of data requires a robust, high-performance, and secure system, capable of dynamically adapting to workload changes, regardless of the incoming data volume. beSharp collaborated with BANOR SIM to develop a bespoke Data Ingestion, Data Processing, and Data Analysis solution, adhering to DevOps best practices and fully compliant with the AWS Well-Architected Framework. beSharp’s Cloud Experts supported BANOR SIM’s technical team through a training-on-the-job approach to build awareness within the company and provide them with all the necessary tools to operate autonomously and leverage the full potential of the new solution.
The Challenge
Processing Ever-Increasing Volumes of Data.
- Improve the performance of Data Ingestion, storage, and analysis by exploiting the potential of the Cloud and process ever-increasing amounts of data thanks to a scalable infrastructure capable of handling any workload.
- Store and historicize all raw data coming from feeder applications, reducing storage costs at full capacity and archive them on databases with high read/write capacity.
- Effectively process archived data, internalize data analysis operations and master Machine Learning tools and notions in order to build efficient systems for predicting the behavior of market players.
The Solution
Data-Driven Predictive Analytics.
- Creation of an infrastructure based on fully-managed services such as AWS Lambda to obtain computing power, Amazon Simple Queue Service (SQS) for optimal management of message queues and Amazon Kinesis Data Stream, together with SQS, allowing data ingestion on Amazon S3.
- Design and implementation of a Data Lake based on different classes of storage (storage tiering) using Amazon S3 and Amazon Glacier services and use of an hot-warm-cold model for data categorization, in relation to the frequency of access required.
- Implementation of services such as AWS Athena, AWS Select and AWS Glue for the rapid development of custom queries on Data Lake and use of Amazon DynamoDB as a database for the real-time consultation of the latest data through a dashboard.
