The solution was to replace the legacy, home-built personnel management system with a streamlined, modern platform for all HR Information within one system.
SMX migrated the FBI’s architecture to Amazon Redshift, a cloud-based data warehouse that can analyze exabytes of data simultaneously. By migrating to Amazon Redshift, the speed of data warehouse queries increased by 3x or more and offered a seamless integration with S3 allowing the customer to parse storage from compute. This Extract-Load-Transform (ELT) platform, branded as Datashift, increased the number of complex queries answered using the FBI’s HR data.
Datashift uses Amazon Database Migration Service (DMS) for real-time change-data-capture updates, Apache Airflow for job orchestration, and the high-powered Amazon Redshift database allowing for the ELT process to run continuously throughout the business day. This end-to-end process gave the customer’s Human Resources division access to near real-time data compared to the data that was only available to users from the nightly batch loads on the previous platform.
Our team leveraged agile methods using 2-week sprints; priorities were set during weekly collaborative meetings with the client. As a result, end users experienced improved ease of use and enhanced interaction, gaining the ability to produce robust reports that are more specific to business needs.