SWEN-440

You need to

• understand the piece of the system you have
• design a data architecture from the given elements in a way that it meets all the requirements and constraints
• draw a diagram of the data architecture

The system you need to design the data architecture for is a distributed web application details of which do not matter. What matters is the part of the system - there is a microservice that accepts the data and saves it to its database, and along with this database there is a folder on a file system containing additional data. For, example, it can be a service responsible for processing one of the steps of mortgage applications. In this case, the database stores the core data, while the folder contains user documents, photos, data catalogs, tabular data from other organizations, etc.

You need to design a data architecture where

• data is becoming immediately available for analytics and visible on the dashboard after saving (less than 1 minute delay)
• there is a guarantee that even if there is a glitch and some data is lost in the process where we make the data available for analytics immediately, eventually we fix it and make that data available there anyway (less than 1 hour of data inconsistency)
• ML model is periodically retrained
• ML model is retrained only on consistent data
• analytics dashboard has access only to a subset of data
• analytics dashboard doesn't recalculate all aggregates every time it's opened
• information on the connections between files and pieces of data in the service database is stored in the very same database (e.g., if we process mortgage applications, every mortgage application stores a list of relevant files and paths to them)
• data in the service database and in the folder is not used for analytics and ML training directly as it can affect the web application performance

• service - any service that receives some data and saves it to a database
• database - an OLTP database storing data for a service, the volume of updates varies from 10 records / second to 1 record per minute
• folder - a folder on a cloud file system that contains data supplemental to data stored in the database, this data have to be used for any analytics or ML-training where data from the database is used. The rate of updates is not the same as for the data in the database - some database updates will be accompanied by updates to the data in the folder, and some will not
• message broker - a system that facilitates communication between different services by routing messages between them (such as Kafka, RabbitMQ, etc.)
• data warehouse - a structured repository of data that will use for analytics and ML training
• data mart (one or more) - a structured subset(-s) of data from the data warehouse that is organized for a specific purpose or audience
• analytics dashboard - a frontend that shows reports and visualizations of the data
• ML model - a model that is trained on the data and used to make predictions or classifications
• data pipelines - any number of data pipelines for data transformation and movement (you need to understand what every pipeline you use does)
• secret element A - something we discussed
• secret element B - something that we didn't discuss yet, but is logical to have to make handling the architecture you design easier