A Fortune 50 energy company, serving over 90% of utilities worldwide with its grid solutions, wanted to reduce the costs associated with their data and analytics platforms while simultaneously improving their agility and scalability. To do this, they needed a partner to redesign and migrate the existing architectures to AWS to take advantage of cloud-native technologies. HCLTech has been a trusted partner of the client for multiple projects, including the implementation of several technologies, and was once again selected to perform the migration and the accompanying tasks.
The Challenge
Managing large and complex data
The client generates a massive amount of data daily because of key operations, resulting in large and complex data sets serving mission-critical processes. This presented several unique challenges for the migration project:
- Migrating 3,000+ Talend ETL jobs, thousands of stored procedures and hundreds of Tableau reports from on-premises databases
- Simultaneously altering relevant architecture to take advantage of AWS-native technologies
- Maintain high-quality data during the migration to minimize the maintenance required after go-live
The Objective
Optimizing data and analytics platform performance — quickly
In addition to helping the client’s data and analytics perform better at any scale via cloud-native services, the migration also needed to ensure quick time to value with a robust and cost-effective system that would reliably serve key dashboard-based analytics functions immediately after deployment.
The Solution
Re-architecting and migrating to leverage cloud-native technologies
After a preliminary assessment of critical factors such as workloads, peak times and potential issues, the HCLTech team’s senior AWS architects designed the new architecture and led a “bottom-up” team focusing on the platforms, data integrity and ETL/UDF conversion, and a “top-down” team handling report conversion and outlying data quality issues.
- Used lift-and-shift approach to migrate from Oracle and Greenplum to AWS
- Deployed custom-developed toolset to automate and optimize repeatable tasks to increase productivity when re-architecting the client’s on-premises data lake and its component
- Re-architected solutions to fully leverage cloud-native technologies such as S3, Spectrum, Glue and Redshift
- Optimized platform performance and cost-efficiency for given use cases and functional contents
- Developed a web-based application to enable the client to monitor and test migration results
- Enabled continuous report delivery for each of the client’s business units during the migration
- Leveraged close relationship with AWS to recommend new features and introduce them on-the-fly during the migration
- Provided hyper care for 30 days, using state-of-the-art operations monitoring systems to enable predictive maintenance, reduce the reliance on manual labor and future-proof the client’s new, cloud-based environment
The Impact
Lower total cost of ownership with increased agility and scalability
The client realized excellent time-to-value by using automation, saving hundreds of man hours during the migration process, and was able to rely on the new architecture during the end-of-year closing shortly after go-live. This was also possible because the migrated dashboards required very little maintenance thanks to the high data quality retention during the automated processes. The client reduced the total cost of ownership by using end-to-end AWS services to achieve savings on hardware, license and maintenance costs, and the eventual fine-tuning of the architecture will help realize further cost optimization.
