There were a few challenges that CleanSlate initially faced when determining the best strategy for KSM. These include:

• Siloed data sources and inconsistent reporting.

KSM’s information was stored in separate departments and systems and was not accessible to the rest of the organization. This made it difficult for KSM to share information, collaborate effectively, and make informed decisions. With different teams using incomplete data this led to inaccuracies and confusion.

• Lack of a formalized, centralized data foundation.

KSM’s underlying infrastructure, strategies, and processes for managing data was not formalized or centralized which led to inconsistent data and inefficient operations. From how data is collected and stored to ensuring its quality and accessibility, there was lots of room for improvement by moving to an AWS infrastructure.

There also wasn’t a standard policy or clear governance for how data is handled, which led to unreliable information.

• Desire to move toward AI/ML-enabled insights.

KSM wanted to shift its business strategy to rely on data-driven recommendations generated by artificial intelligence (AI) and machine learning (ML). Instead of relying on traditional analytics, spreadsheets, or human intuition, they wanted to take advantage of advanced technology to uncover patterns, predict outcomes, and automate decision-making from large sets of data.

By doing this, KSM can become more proactive and competitive by turning complex, raw data into clear, actionable intelligence. CleanSlate needed a clear plan to help them accomplish this.

• Complex data transformation and ingestion needs.

KSM was in need of a system to collect, clean, and convert data from multiple, diverse sources into a usable format for analysis. This was a challenge due to the volume, variety, and velocity of the data. This required handling issues like integrating different data formats, ensuring data quality, and setting up robust processes to handle potentially large volumes of information efficiently.

CleanSlate delivered a complete system for generating time entry reports using two key cloud services: Amazon Web Services (AWS) and Databricks. This solution not only established repeatable data processing patterns required to begin migrating Time Entry reporting and future data to the new Lakehouse, but it also enabled new analytic and AI/ML capabilities. CleanSlate handled the entire process, from collecting data to creating the final reports. Here is an overview of the solutions CleanSlate provided:

• Replaced legacy use of Microsoft SSIS/PowerBI with Databricks functionality.

By migrating to Databricks, KSM could now have a single, collaborative workspace for all data teams—engineers, analysts, and scientists. They no longer needed separate tools like SSIS for data movement and Power BI for reporting, which streamlined their entire workflow. Through Databricks’ AI/BI Genie, KSM could generate booking data reports automatically. This process bypassed the traditional, time-consuming method of hiring or relying on Power BI developers to create custom reports manually.

KSM’s legacy process needed to continue to exist as they moved forward with modernization. CleanSlate provided a hybrid approach that allowed them to do just that, bringing on new sources and analytics while maintaining the existing BI solution.

• Developed scalable data pipelines for ingestion and transformation.

Through AWS and Databricks, CleanSlate built an automated system to efficiently move and process large amounts of data, which can automatically handle increased data volume without performance issues. This process involves ingestion (collecting raw data) and transformation (cleaning and preparing it for analysis), all designed to scale up as needed.

• Trained internal team to manage Lakehouse.

CleanSlate wanted the KSM team to be able to manage the Lakehouse on its own long after the project is complete. To make this possible, CleanSlate invested time in training the internal team on how to manage the Lakehouse so they would be the experts on their own data. The training was customized to focus on the KSM’s unique priorities and use cases for the lakehouse.

• Created reusable patterns using Infrastructure-as-Code.

CleanSlate used Infrastructure as Code (IaC) principles to create reusable, standardized templates for common infrastructure setups. By defining infrastructure in code, KSM can reuse these patterns across different projects or environments, which saves time, reduces errors, and ensures consistency. CleanSlate delivered a data lakehouse platform with two functional use cases. This included object hierarchy design, role based access, ingestion patterns, data quality enforcement, a dimensional model for consumption, and connectivity with Power BI.

Technologies & Services Used

• Databricks Lakehouse
• AWS Glue
• Amazon S3
• Amazon EC2
• Terraform
• Data & AI Consulting
• Data Ingestion & Transformation Pipeline

CleanSlate delivered the solution KSM was looking for: a modernized data lakehouse that combines AWS and Databricks Lakehouse to improve scalability and security. This innovative platform positions KSM for success both now and in the future. Here are a few results that were delivered:

• Centralized and scalable data foundation

  • Pre-Built Accelerator for rapid deployment
  • Automation built in
  • Batch and streaming use cases supported
  • Automated stand up of Lakehouse
  • Bronze & Silver Medallion layers in AWS or Azure
  • Seamless data integration pipelines

• Ready for experimentation in AI/ML.

With this new infrastructure, data pipeline, and workflow in place, KSM can rapidly test, train, and manage different artificial intelligence and machine learning models. CleanSlate equipped KSM with the tools they needed to explore new ideas and move promising models into production quickly and efficiently.

KSM wanted a genAI foundation for natural language analysis, using their own business terminology. CleanSlate helped put the foundation in place to do just that.

• Increased reporting efficiency and governance.

  • Full monitoring and observability
  • Dashboards for cost job execution and data quality
  • Databricks Unity Catalog with Role Based Access
  • Data Quality with ML Models

• Established organizational structures and roles, enabling governed growth and observability.

Through this new foundation CleanSlate helped implement, KSM could now expand in a controlled, deliberate, and secure way through clear processes, decision-making rules, and risk management strategies.

• Project completed on-time in 10 weeks.

By completing the project on-time and quickly, KSM can achieve improved performance and cost savings through one platform with lower storage and compute costs.