Uplift Education’s mission is to create and sustain public schools of excellence by establishing and empowering a results’-oriented culture. This means managing and improving both student and teacher performance with accurate, timely, and consistent data that is available on-demand. Uplift had made a significant investment in developing a data warehouse that had a number of key challenges
No user confidence in that data warehouse – the data extraction and integration process was complex and the data collection and transformation logic (“ETL”) was very difficult for business users to diagnose/verify. Lack of meta-data transparency between DW (Data Warehouse) and operational SIS (Student Information System) made it very difficult to find and correct data errors. This lead to report data consistency errors which resulted in the majority of business users having no confidence in the data warehouse reports and KPIs (Key Priority Indicators).
Integration of new data sources was cumbersome and slow – data resides in multiple systems and flat files including PowerSchool SIS (Student Information System), external assessments (SAP, ACT, AP, ITBS etc.), interim assessments (3rd party web based tools), teacher evaluation forms (.NET forms) and other tools such as Naviance (college and alumni tracking), Study Island (online assessments and assignments), Finance and HR systems.
LogicMatter used its best practices based framework as a reference architecture for an ODS (Operational DataStore) that collected data from various sources and made the transactional meta-data and data from various silos of sources available to analysts in a single staging environment.
We used an agile engagement model, VISPER to first assess the current Datawarehouse and ETL (Extract Transform and Load) state, identified alternative EL (Extract-Load) based strategy to make all OLTP (online transaction processing) data coming from various source systems replicated in the ODS.
The solution was customized iteratively, leveraged the infrastructure investment made earlier and included:
- a unified operational data store (ODS) to house all the transactional data
- Data collector integration of internal transactional systems and applications such as PowerSchool and Kickboard
- Data collector integration of 13 external assessment data source including MAP, ACT, AP, PSAT, STAAR, TAKS, TELPAS, SAT, EOC, PLAN, EXPLORE, and CFA.
- incremental data loading combined with indexing to improve performance
- detection of data errors and feedback to enable fixing them in the source systems
- integration of Tableau, the visualization tool, directly with the ODS for ad-hoc analysis and discovery
Using the best-practices framework LogicMatter helped Uplift achieve several key benefits:
- ability to change data sources without breaking reports from the warehoouse or the ODS i.e. flexible data environment
- continuous monitoring and fixing cycles to improve data quality
- separation of transformation from extract-load into ODS increased the transparency of data, which inturn helped the analysts fix data entry errors at the source systems that fed invalid data.
- fine grained row level security was a side benefit of the ODS, enabling precise access control over data from various sources
- improve usability due to consistent terminology between source systems and reports
- ability for business users to easily author reports (eliminating IT dependency)
- Uplift team liked our subscription based fixed monthly devOps service model
- The framework and ODS implementation was highly appreciated and widely used for distributed reports and adhoc-analysis
- Flexibility of collector integration and ability to add/remove data sources was well received(e.g. add data sources, state reporting)