Background
Our customers are from logistic domain dealing with sensitive data covers customs and other information. This required higher level of security standards. On the other hand business demands on getting faster insights to data and a scalable data platform. This required the architecture team to build a cloud based data hub which enabled them to bring the data of huge volume as quick as possible. Ability to scale quickly, horizontally as well as vertically also one of the main objective. Compliance required the customer to process and archive huge volume of data with secured access
Solution Implemented
- Parameter driven ETL framework
- S3 based Data lake, Glue and PySpark for scalable and serverless solution
- Fine grained security (glue database level as well as for S3 folders) using AWS IAM
- AWS KMS for Encryption (S3 and Glue)
- EDI Integration (IBM MQ)
Outcomes
With the ETL framework and other implemented agile methodologies, a robust and scalable cloud data hub was built. It helped the end-user with seamless data handling and effective data processing. AWS KMS encryption approach gave top-notch security standards. The product totally impressed the user with its best-in-class performance.
Domain
Logistics
Methodology
Agile with sprint every 2 weeks
Duration
7 Months
Resources
8
Project Status
Live
Maintenance support
Technology Used
