🎓 Data Mining — Midterm

Definition: Data Mining is the process of selection, exploration, and modelling of large quantities of data to discover regularities or relations that are at first unknown, with the aim of obtaining clear and useful results for the owner of the database. It is also known as Knowledge Discovery in Databases (KDD).

The 5 KDD steps:

1. Data Selection — identify relevant data sources and extract the target dataset.
   What can go wrong: picking the wrong date range, e.g. using only 2020 COVID-year data to predict "normal" customer behavior.
2. Preprocessing — handle missing values, noise, inconsistencies.
   What can go wrong: failing to standardize "M"/"Male"/"male" → the algorithm treats them as 3 different categories.
3. Transformation — feature engineering, normalization, dimensionality reduction.
   What can go wrong: not scaling variables so one feature (e.g. income in thousands) dominates another (age 0–100).
4. Data Mining — apply algorithms to discover patterns.
   What can go wrong: choosing a neural network for a small dataset when a decision tree would give an interpretable answer.
5. Interpretation / Evaluation — evaluate, visualize, communicate findings.
   What can go wrong: presenting a correlation as causation, or reporting an accuracy that hides poor performance on rare classes.

Key point: each stage feeds the next — data quality at early stages directly affects the value of the final knowledge. (Garbage in, garbage out.)

Step 1 — Assign each data type to the right architecture:

• Account/transaction data → Keep the Data Warehouse. Structured, governance-critical for regulatory reporting (Basel III, GDPR). Don't throw away 10 years of clean data.
• 8M customer support emails → Data Lake. Unstructured text, schema-on-read, needs NLP and exploratory analysis (sentiment, topic modelling, fraud language detection).
• Real-time ATM sensor logs → Data Lake with streaming ingestion (e.g. Apache Kafka). Real-time patterns require stream processing, not batch SQL. Supports fraud detection and predictive maintenance.
• Bottleneck of 4 teams fighting 1 data team → Move toward Data Mesh principles: each team (retail, wealth, fraud, credit risk) becomes a domain owning its own data product, with a shared self-serve platform.

Step 2 — Add Data Marts for the 4 teams: dependent marts built from the warehouse give each team focused, fast access without exposing the whole warehouse.

Step 3 — The governance caveat: Banks are heavily regulated. Data Mesh principle 4 (federated governance) is non-negotiable here. Central standards on privacy, access control, and regulatory compliance must be enforced even while domains own their pipelines.

Final recommendation: A hybrid architecture using all 4 patterns simultaneously — warehouse for structured governed data, lake for unstructured/streaming, marts for team-level analytics, mesh principles to remove the bottleneck under strict federated governance.

Why colored pixel panels beat a simple line chart — 5 reasons:

1. Data density: The dataset has 16,350 daily values. A line chart becomes unreadable beyond ~1,000 points; a pixel-oriented view shows every value — one pixel per observation.
2. Temporal locality preserved: The Peano-Hilbert curve places nearby-in-time pixels nearby in space, so stable periods appear as colored blocks that eye can instantly see.
3. Pattern visibility: Prolonged low/high periods become large single-color patches (purple = low, green = high). Sudden color shifts highlight volatility and shocks.
4. Cross-variable comparison: 4 panels side-by-side let you spot simultaneous color shifts — something a line chart with 4 overlapping lines hides in clutter.
5. Global-event detection: When Stock + Index + Currency + Commodity all change color at the same region, it signals a systemic shock (financial crisis, oil shock, policy change, war).

What you could learn from it:

• Periods of market stability (large same-color patches)
• Bull markets (green clusters in Dow Jones)
• Bear markets or low-price periods (purple clusters in IBM or Dow)
• Correlations between markets — do IBM and Dow move together? Does Gold rise when Dollar falls?
• Global shocks — 1987 Black Monday or the early 1990s recession could appear as synchronized color transitions across all 4 panels

One-line summary: Simple line charts cannot show 16,000+ points or compare 4 variables visually at once; pixel panels with the Hilbert curve compress this into a visual form the human eye can read in seconds.

Core differences:

Why complementary:

• Statistics provides the theoretical grounding and rigor for methods data mining uses.
• Machine learning provides many of the algorithms (neural nets, decision trees) that data mining applies.
• Data mining expands scope and application — turning them toward business advantage.
• Explorative bottom-up mining can generate hypotheses; confirmative top-down statistics can then test them. Each enhances the other.

Modern data mining explicitly borrows from both: it uses ML algorithms but applies statistical generalization principles (penalizing overly complex models) to avoid the historical critique of "data fishing".

Definition (Inmon, 1996): A data warehouse is an integrated collection of data about a collection of subjects, which is not volatile in time and can support decisions taken by management.

The 4 SIVT characteristics:

1. Subject-Oriented (S)
   Organised around business subjects (Customer, Product, Transaction), NOT around the applications that created the data.
   Bank example: The billing application is organised around invoices. The warehouse reorganises this around Customers — because that's what analysts study.
2. Integrated (I)
   Unifies data from many sources using consistent naming, encoding, and measurement conventions.
   Bank example: The retail system codes gender as M/F, the mortgage system uses 1/0, legacy systems use "Male"/"Female". The warehouse unifies all three into one consistent encoding.
3. Non-Volatile (V)
   Data is loaded once and never modified. Changes are recorded as new rows with timestamps.
   Bank example: When a customer moves from London to Manchester, the operational CRM updates the record. The warehouse adds a new row with the new address, preserving the old address with its timestamp — so you can always see where the customer lived on any past date.
4. Time-Variant (T)
   Stores 5–10 years of history (operational DBs only keep 60–90 days). Every record has a time dimension.
   Bank example: The warehouse lets analysts compare Q3 2023 mortgage performance with Q3 2015 — trend analysis, seasonality, long-range forecasting. Operational systems have thrown that data away.

Why these properties matter: Together SIVT makes a warehouse trustworthy, consistent, and historically complete — the three prerequisites for reliable analytics and regulatory reporting.

The 3 types:

• Dependent mart — derived directly from a central data warehouse. The warehouse is the single source of truth. Changes in business rules propagate automatically. This is the gold standard.
• Independent mart — built directly from source systems, bypassing the warehouse. Faster to set up but creates its own silo with its own ETL, its own definitions, and its own quality problems.
• Hybrid mart — pulls from both warehouse and source systems. Adds flexibility (e.g., clean history + real-time streaming) but increases complexity and inconsistency risk.

Why the board arguments happen:

With independent marts, each team built its own extraction from source systems with its own interpretation of the business rules. So:

• The Claims mart calculates "total claims paid" one way (e.g., gross of reinsurance).
• The Underwriting mart calculates it differently (net of reinsurance recoveries).
• The Finance mart calculates it yet a third way (including loss-adjustment expenses).

When the CFO asks "What were our claims costs last quarter?" she gets 3 different numbers from 3 marts, all "correct" by their own definitions. The boardroom debates whose number to believe instead of making decisions.

The fix: Convert to dependent marts derived from a single enterprise data warehouse. Define "claims cost" once in the warehouse; let all marts inherit that definition. The numbers then agree by construction.

Data Lake: a centralized repository that stores all structured, semi-structured, unstructured, and streaming data at any scale in its raw, unprocessed format.

Schema-on-Write (Warehouse) — the data's structure is defined before it is loaded. ETL cleans and standardizes. Data quality is enforced at write time. Rigid but trustworthy.

Schema-on-Read (Lake) — data is stored raw. Structure is applied at query time by whoever reads it. The same raw data can be interpreted many ways for different use cases. Flexible but quality varies.

The Data Swamp Problem: A data swamp is a lake that has ingested vast amounts of data with no metadata, no cataloguing, no access controls, no governance. The symptoms:

• Analysts cannot find what they need
• Nobody knows which version of a dataset is current
• Sensitive personal data sits unprotected
• The lake becomes an expensive, unusable mess

Prevention — 4 things:

1. Metadata & data catalog — every dataset must be documented (what it is, where it came from, who owns it). Tools: Apache Atlas, AWS Glue Data Catalog, Unity Catalog.
2. Access controls — role-based permissions, especially for personal/sensitive data.
3. Quality monitoring — automated checks for freshness, completeness, schema drift.
4. Data governance policies — lifecycle rules, retention, privacy compliance (GDPR).

Key insight: A lake needs as much governance as a warehouse. The governance is just applied at read time instead of write time. Skipping governance is the root cause of every data swamp.

Short answer: Yes, Data Mesh directly addresses this bottleneck — provided the company invests in a shared platform and governance.

The 4 Principles of Data Mesh (Zhamak Dehghani, 2019):

1. Domain Ownership — each business domain (payments, logistics, customers) owns, manages, and is accountable for its own data. The people who create the data (and understand it best) should own it. No more central gatekeeper.
2. Data as a Product — each domain's data is treated like a software product: clear owners, SLAs for uptime and freshness, documentation, quality standards, versioning. Designed with consumers (other teams) in mind.
3. Self-Serve Data Infrastructure — a shared platform abstracts complexity away. Templates for pipelines, cataloguing tools, access control, quality monitoring, compute infrastructure. Enables autonomy without chaos.
4. Federated Computational Governance — global standards (privacy, security, GDPR, compliance) defined and enforced at enterprise level. Domains choose HOW to implement but cannot opt out. Autonomy WITH shared rules.

Why it fits this case: 40 units queuing for a 15-person team is a textbook bottleneck. With mesh, each unit publishes its own data products through a shared catalogue. Any team can discover and consume any other team's data — like a marketplace, no central hub.

The main risk — governance complexity: Decentralization without standards creates anarchy. If each of 40 units defines "active customer" differently, the company ends up with the same "multiple versions of truth" problem that independent marts create. Principle 4 (federated governance) is non-negotiable. For regulated industries (banking, healthcare), this is especially critical — you cannot outsource GDPR or HIPAA compliance to domain teams.

Other risks: cultural change (domain teams need new skills), upfront platform investment, possible duplication of pipelines across domains.

The hierarchy (lowest to highest information capacity):

Why Data Mining is at the top: The other tools answer questions you already know to ask. Data Mining discovers patterns you didn't know to look for. It brings together all variables in different ways, finding unknown relations. That's the highest information capacity — but also the hardest to implement, requiring skilled teams and high-quality data.

The trade-off: there is an inverse relationship between information capacity and ease of implementation. A simple query is quick but tells you only "what happened"; data mining takes months/years but reveals "what patterns exist that we never suspected".

OLAP vs Data Mining — complementary, not competing:

• OLAP tests user-driven hypotheses ("let me check sales by region by quarter"). The user asks the questions.
• Data Mining uncovers unknown relations automatically. The data reveals the questions.
• OLAP is often used in the preprocessing stages of data mining — to understand the data, identify special cases, spot principal interrelations before applying mining algorithms.
• Used together they create useful synergies — OLAP scopes what to mine, mining reveals what OLAP should investigate further.

Important detail: OLAP fails with tens or hundreds of variables — the hypothesis space becomes too complex for a human to navigate. That's where data mining's automation becomes essential.

Predictive (Asymmetrical / Supervised / Direct):
Describes one or more target variables in relation to others. There is a known answer to predict.

• Classification — categorize into known classes. Example: Gmail classifying emails as spam or inbox.
• Regression — predict a continuous value. Example: Zillow estimating house prices from features.
• Time-series forecasting — predict future from past. Example: forecasting next month's electricity demand.

Descriptive (Symmetrical / Unsupervised / Indirect):
Describes groups of data more briefly. Observations classified into groups not known beforehand; variables connected through association or graphical models.

• Clustering — group similar items without labels. Example: a marketing team clustering customers into personas (bargain hunters, loyalists, premium buyers).
• Association rules — find "if A, then B" co-occurrence patterns. Example: Netflix viewers who watched Stranger Things also watched Dark.
• Anomaly detection — identify unusual observations. Example: credit card fraud detection.

Where "association rules" fit: They are a local method — a descriptive technique that identifies characteristics in subsets of the database (not the whole dataset). Giudici classifies them as local, alongside outlier detection.

Why the beer-and-nappies story is famous:

• Walmart mined billions of transactions from their 2.5 petabyte warehouse.
• Discovered: beer + nappies bought together on Friday evenings.
• Investigation revealed new fathers on "emergency supply runs".
• Walmart moved beer displays next to nappies → sales of both rose.

The lesson: No analyst would have thought to test the beer-nappy hypothesis. The unsupervised association-rule mining found a pattern nobody was looking for. This is the essence of data mining: finding what you were not looking for. It's the canonical example of association-rule value and defines descriptive mining in action.

Key lesson: Just because something LOOKS like a number doesn't mean it's numeric. Postcodes, Zip codes, Student IDs are stored as numbers but are nominal — no order, no arithmetic. Asking a mining algorithm to "average postcodes" is nonsense. Always ask: does order matter? does distance make sense? If not, it's nominal (or ordinal at best).

Sorted data: 20, 22, 24, 25, 26, 28, 30, 1000 (all in $1000s).

Mean: (20 + 22 + 24 + 25 + 26 + 28 + 30 + 1000) / 8 = 1175 / 8 ≈ $146.9k

Median: With 8 values, the median is the average of the 4th and 5th sorted values: (25 + 26) / 2 = $25.5k

Mode: No value repeats, so technically no mode (or every value is a mode, depending on convention).

Quartiles and IQR:

• Q1 (25th percentile) = median of lower half {20, 22, 24, 25} = (22 + 24) / 2 = 23
• Q3 (75th percentile) = median of upper half {26, 28, 30, 1000} = (28 + 30) / 2 = 29
• IQR = Q3 − Q1 = 29 − 23 = 6 (i.e., $6k)

Best measure of "typical income" — MEDIAN ($25.5k):

• The mean ($146.9k) is badly distorted by the $1M outlier — no one in this group actually earns anywhere near $146.9k.
• The median is robust to outliers: removing or changing that $1M value doesn't move the median at all.
• The IQR (6) also shows that the middle 50% of the data is tightly clustered between $23k and $29k — the outlier is a single extreme value, not representative.

General rule: for skewed data (income, house prices, company revenues), always prefer the median over the mean. It's why governments report "median household income", not "mean".

Differences per dimension:

• |25 − 30| = 5 (Age)
• |40 − 55| = 15 (Income)
• |12 − 18| = 6 (Purchase Frequency)

Manhattan distance (h = 1):
D = 5 + 15 + 6 = 26

Euclidean distance (h = 2):
D = √(5² + 15² + 6²) = √(25 + 225 + 36) = √286 ≈ 16.91

Why Cosine similarity is inappropriate here:

• Cosine measures the angle between vectors, ignoring magnitude. Two customers with identical purchasing patterns at different scales (e.g., one is 2× the other on every feature) would have cosine similarity 1 — deemed "identical".
• But in customer analytics, magnitude matters a lot: a 25-year-old earning $40k with 12 purchases is genuinely different from a 50-year-old earning $80k with 24 purchases, even though they have the same "shape".
• Cosine is designed for sparse, high-dimensional data where direction matters more than magnitude — most typically text documents, where doc length varies wildly but topic/direction is what counts.
• For dense, low-dimensional numeric data like customer attributes, use Euclidean or Manhattan.

One more caveat — scaling: Notice that Income (difference of 15) dominates the Euclidean distance because it has a larger numeric range than Age (difference of 5). In practice, before computing distance, normalize or standardize the variables so no single attribute dominates. This is part of KDD step 3 (Transformation).

1. A — Define Objectives: Aim = reduce monthly churn by 15% in 6 months. Specifically, identify customers at high risk of churning 30–60 days before they leave, so marketing can intervene. This is the MOST critical phase — all downstream work depends on it.
2. B — Select & Organise Data: Pull from internal sources (cheaper, more reliable). Needed: transaction history, loyalty-card data, support call logs, demographics, marketing interactions. Build a Customer data mart from the warehouse. Data cleansing: handle missing emails, standardize phone formats, remove deceased customers.
3. C — Exploratory Analysis & Transformation: Use OLAP-style analysis: what % churn per store region? per age group? per tenure? Create derived features (recency of last visit, frequency per month, monetary value — the RFM variables). Detect anomalies: customers with $0 spend but 20 visits (probably card error).
4. D — Specify Methods: This is a predictive task (target = "will churn in next 60 days?"). Candidate methods: decision tree (interpretable, easy to explain), logistic regression (classic baseline), random forest (higher accuracy), neural net (if lots of data). Decision tree might be chosen first for interpretability with marketing team.
5. E — Data Analysis: Train multiple models on historical data (e.g., 2022–2023). Use cross-validation. Track precision, recall, AUC, and lift at different score thresholds.
6. F — Evaluate & Compare: Compare models on held-out data. Consider also training time, interpretability, ease of deployment, data quality sensitivity. If two models tie on accuracy, choose the more interpretable one for business adoption.
7. G — Interpretation & Implementation: Integrate the chosen model into the CRM: each night, score every customer's churn probability. High-scoring customers get a retention offer (discount, loyalty bonus, personal call). Track actual churn reduction versus baseline over the 6-month period. The virtuous circle of knowledge kicks in — new churn data flows in, model is retrained.

Implementation phases (from Section 6.1): Strategic (where does mining add most benefit?) → Training (pilot on 3 stores) → Creation (scale to all stores) → Migration (train users, evaluate continuously).

1. Pixel-Oriented Visualization
   Maps each data value to one colored pixel. Uses space-filling curves (e.g., Peano-Hilbert) to order pixels meaningfully. Maximum data density on a screen.
   Best scenario: visualizing 16,000+ daily financial observations across multiple markets — exactly like the IBM/Dollar/Dow/Gold example.
2. Geometric Projection
   Projects high-dimensional data into lower-dimensional space. Examples: scatter-plot matrix (pairwise scatterplots of all variables), parallel coordinates (each variable = vertical axis, each point = a line crossing them).
   Best scenario: exploring relationships among 6–12 numeric variables in a customer dataset to find clusters or correlations.
3. Icon-Based Visualization
   Encodes multiple attributes as features of small icons. Examples: Chernoff faces (face features encode variables), stick figures.
   Best scenario: showing multi-dimensional health patient profiles where humans instinctively read "facial" patterns — a clinician can spot anomalies faster via Chernoff faces than a table of numbers.
4. Hierarchical Visualization
   Partitions dimensions into subspaces and displays them in nested structures. Examples: tree-maps (nested rectangles sized by value), Worlds-within-Worlds (n-Vision).
   Best scenario: visualizing budget allocations across departments, sub-departments, and line items — each level is a rectangle whose area = spend.

Choosing rule: pixel-oriented for massive time series, geometric projection for relationships among moderate-dimensional numeric data, icon-based for human-pattern-matching, hierarchical for nested/tree-structured data.

Metadata = data ABOUT data. It describes the structure, meaning, history, and ownership of the data itself.

Typical metadata answers:

• How was this variable calculated? (e.g., "monthly_revenue" = sum of completed transactions, excluding refunds, in local currency)
• When did a field definition change? (e.g., "region" was redefined in 2021 when sales territories were restructured)
• Which source system did each record come from?
• Who last modified the record, when, and why?

Why a warehouse without metadata fails:

• The Library analogy: A warehouse without metadata is like a library without a catalogue. The books exist; nobody can find what they need.
• The Definition trap: Two teams compute "active customer" — one uses "bought in last 30 days", the other uses "logged in last 30 days". Both put numbers in the warehouse. Without metadata, the CFO sees two conflicting counts with no way to know which is correct. Meetings descend into definition arguments.
• Historical changes get lost: If the "region" field was redefined in 2021, any trend analysis crossing 2021 is meaningless unless the change is documented in metadata.
• Compliance failure: GDPR and other regulations require organizations to know where personal data came from and where it's used. Without metadata, you can't answer "the right to be forgotten" requests.

Metadata increases the VALUE of the warehouse because it makes the data reliable, trustworthy, and traceable. Many organizations invest heavily in the warehouse infrastructure but skimp on metadata — and their warehouse becomes a source of constant arguments rather than clear answers.

When to choose each — one-sentence rule:

• Warehouse: when you need reliable, governed, consistent reporting for structured data — typically finance, compliance, and executive dashboards.
• Mart: when a specific team needs fast, focused access to a subset of warehouse data without navigating the full complexity.
• Lake: when you have large unstructured or semi-structured data that data scientists need raw for ML training, or when you want to store first and decide use later.
• Mesh: when your organization has many autonomous business units, a central data team has become a bottleneck, and domain teams need to move independently.

The Big Insight: These 4 architectures are not mutually exclusive. Every mature large organization (Netflix, Spotify, big banks) runs ALL 4 simultaneously, assigning each data type to the architecture that best fits its characteristics.

(a) Classify email as spam
Task: Predictive — classification (known label: spam / not-spam).
Algorithm: Naive Bayes (classic baseline) or Logistic Regression. For higher accuracy: Random Forest or a Neural Network on text features.
Why: supervised classification with labeled training data; email text gives high-dimensional features; Naive Bayes is famously effective here.

(b) Group supermarket customers into segments
Task: Descriptive — clustering (no predefined labels).
Algorithm: K-Means (most popular).
Why: unsupervised clustering of numeric customer features (RFM — recency, frequency, monetary). Caveat: K-Means is sensitive to the choice of k (number of clusters); use the elbow method or silhouette score to choose.

(c) Predict tomorrow's electricity demand
Task: Predictive — time series forecasting (continuous target, ordered in time).
Algorithm: classical ARIMA, Exponential Smoothing, or modern LSTM neural networks.
Why: time-ordered data with seasonality (daily, weekly, yearly cycles). Needs a time-series-aware method, not standard regression.

(d) Find frequently co-purchased product pairs
Task: Descriptive — association rule mining (local method).
Algorithm: Apriori (classic) or FP-Growth (faster on big data).
Why: classic market basket analysis — the beer-and-nappies problem. Apriori finds rules of the form {A, B} → {C} with support and confidence measures.

(e) Detect fraudulent credit-card transactions
Task: Can be framed two ways:

• If you have labeled historical fraud examples → Predictive classification. Use Random Forest, Gradient Boosting, or Neural Networks. Imbalanced classes require resampling (SMOTE) or cost-sensitive learning.
• If fraud patterns are novel and unlabeled → Anomaly / outlier detection (descriptive, local method). Use Isolation Forest or One-Class SVM.

Key takeaway — map task type to algorithm family:

• Predictive + known categories → Classification (trees, forests, neural nets, logistic regression)
• Predictive + numeric target → Regression (linear, trees, neural nets)
• Predictive + temporal → Time-series (ARIMA, LSTM)
• Descriptive + grouping → Clustering (K-Means, hierarchical)
• Descriptive + rules → Association (Apriori)
• Descriptive + outliers → Anomaly detection (Isolation Forest)