Управління якістю даних в ETL процесі в умовах обмежених ресурсів

Abstract

Currently, access to data is necessary for many companies, particularly those engaged in marketing, to make decisions that will improve the quality of their services and businesses. They frequently find the knowledge they need from several sources in a variety of formats. Following a dedication to the quality of information offered to data consumers, a system will be implemented to consolidate all these data sources for analysis and decision-making. This study addresses the evaluation of data quality (DQ) in an ETL process developed to support a marketing data management platform. More specifically, this study addressed the problem of evaluating the quality of data with a high-volume trait. Addressing the problem of DQ assessment at high ingestion rates is beyond the scope of this study, which focuses on data quality assessment with limitations to vertical or horizontal scaling of the ingestion system. We also analyze the use of the model developed on real data to assume an improvement in the quality of the data in the ETL. The methodology used consisted of studying each feature related to the characterization of high-quality data and analyzing the impact of those in an ETL concerned with voluminous data. We propose algorithms for improving a more generalizable integration DQ assessment. We conducted a practical implementation study of the different criteria and characteristics proposed to evaluate the impact of the data collected throughout the process of data Extraction, Transformation, and Loading.

We highlight a quality assessment framework that models the different necessary parts of the process, including data sources, metrics characterizing data quality, data destination, and the analysis and performance of the algorithms used in the assessment process. The ETL practical implementation in this research is based on a Direct Acyclic Graph (DAG) model, with the main purpose of extracting, transforming, and transmitting data from this first service to the rest of the Marketing Data Management Platform infrastructure, which is considered as the end user. The evaluation and quality are based on the development of algorithms that take source data as input in combination with predefined properties encompassing the expected result of the ETL transformation to produce the evaluation result. The evaluation findings may be used to support or contradict the standards for quality. Decisions are made in the event of a DQ failure to improve and enhance the data. We suggest including data checks at the very end of the ETL data manipulation process as well as a model for data volume reduction using algorithms that are intended to make the procedure more generic to enable quick review. The quality of the data evaluated during the test is a statistical representation of the ingested dataset, which provides an accurate profile that enables user applications to retrieve high-quality data without delay. The main contributions of this thesis are: i) the development of an ETL service in a Marketing Data Management Platform and ii) an examination of data reduction models with a view to assessing data quality. Chapter 1 presents a literature review of this research and describes the basic concepts and their definitions in other research, including sampling, ETL, Data Quality and Big Data. Chapter 2 We present the manner in which the ETL system fits into the framework of the data-management platform and how the entire architecture is modelled. Chapter 3 presents the outcomes of the experiment. The experimental findings, which were obtained using various types of actual data, are presented in this chapter. The performance over time and the effect of the startified sample are depicted in graphs. The closing part presents the conclusions of this thesis and discusses the prospective research.