DS Unit 2: Complete Concept Guide

Why Data Preparation MattersCopy Anchor Link

• A model is only as good as the data fed into it.
• 70–80% of a data scientist’s time is spent finding, cleaning, and preparing data — not modelling.
• Consequences of dirty data:
  • Wrong decisions (e.g., duplicated customer inflates sales counts)
  • Broken models (missing values crash training or bias predictions)
  • Wasted money (acting on dirty data costs real revenue)
  • Lost trust (one bad dashboard and stakeholders stop believing the data team)

The Data Preparation Journey (8 stages)

1. Sources of Data
2. Types of Data
3. Collection Methods
4. Data Quality Issues (Missing & Noisy data)
5. Data Cleaning
6. Integration
7. Transformation
8. Preprocessing (ready for modelling)

---

1. Sources of DataCopy Anchor Link

Data scientists group origins into three broad families, each with cost–speed–reliability trade-offs.

Primary Sources (collected by you for the exact problem)

• Surveys & questionnaires
• Experiments & sensors
• Interviews & observation

Secondary Sources (already collected by someone else)

• Government & census records
• Kaggle & research datasets
• Company databases (sales, HR, ERP)

Web & Digital (continuously generated online)

• Social media & clickstreams
• APIs & web scraping
• IoT device streams

Primary vs Secondary Comparison

Scenario: GCU wants to know why students skip 8 AM classes.

• Primary → run a fresh survey
• Secondary → reuse existing attendance data already in the ERP

---

2. Types of DataCopy Anchor Link

Classified by internal structure — this decides storage, querying, and preprocessing effort.

1. Structured

• Fixed rows & columns with defined data types
• Examples: Relational tables (SQL), spreadsheets, CSV, ATM transactions
• Easy to store, query, sort, filter, aggregate
• Schema is predefined

2. Semi-structured

• No rigid table, but tags/keys make it self-describing
• Examples: JSON, XML, YAML, emails (header + free body), NoSQL documents
• Flexible schema (records can differ)
• Hierarchical and common in APIs

3. Unstructured

• No predefined model — ~80% of real-world data
• Examples: Text (reviews, PDFs, chats), images, audio, video, social media posts, sensor blobs
• Richest in meaning but hardest to analyse directly (needs NLP / Computer Vision / ML first)

Side-by-side Comparison

---

3. Data Collection MethodsCopy Anchor Link

Each method suits different situations, budgets, and data-change speeds.

• Surveys & Questionnaires — fixed questions to a sample; cheap & scalable but self-report bias
• Sensors & IoT — continuous automatic readings; high-volume real-time but needs hardware
• Web Scraping — extract from web pages when no API exists; powerful but must respect legal terms
• APIs — clean structured data on request (weather, maps, Twitter/X); sanctioned & reliable
• Existing Databases — internal systems already log everything; usually the fastest starting point
• Observation & Logs — server logs, clickstreams, or direct field observation

Which method fits?

• Student satisfaction → Survey
• Campus power usage → IoT sensors
• Live cricket scores → Public API
• Competitor pricing → Web scraping
• Past admissions → Existing database

---

4. Data Quality IssuesCopy Anchor Link

“Garbage in, garbage out.”

Six quality dimensions:

1. Accuracy — values truly match the real world (no typos/errors)
2. Completeness — no required values are blank or lost
3. Consistency — same fact never contradicts itself across sources
4. Timeliness — data is recent enough to reflect current reality
5. Validity — values obey expected rules, formats, ranges (e.g., age > 0)
6. Uniqueness — each real-world entity appears only once (no accidental duplicates)

Dirty Dataset Example (customer table from two systems)

• Duplicate rows (same person with different casing/city spelling)
• Inconsistent city names (Bengaluru vs Bangalore)
• Invalid ages (−5, 210)
• Missing phone & age values

---

4A. Missing ValuesCopy Anchor Link

Why data goes missing

• Data entry errors
• Non-response
• Sensor / system failure
• Merging mismatches

How to handle them

• Delete rows/columns — only if few & random
• Mean / Median / Mode imputation
• Forward / Backward fill (time series)
• Predict (regression / KNN) from other columns
• Flag & keep as “Unknown” category

Worked Example (marks: 72, 68, 90, ?, 85, 76)

• Mean ≈ 78.2 (good for roughly symmetric data)
• Median = 76 (robust to outliers)
• Mode — none (not useful for continuous marks)

---

4B. Noisy DataCopy Anchor Link

Noise = random error or variance that hides the true signal (outliers, wrong entries, fluctuations).

Smoothing techniques

1. Binning — sort into bins, smooth by bin mean / median / boundary
2. Regression — fit a function and replace with fitted values
3. Clustering — group similar points; values outside clusters = outliers
4. Outlier removal — detect with IQR / Z-score, then drop or cap

Binning Example (prices: 4, 8, 15, 21, 21, 24, 25, 28, 34 → 3 equal-frequency bins)

• Smooth by mean replaces every value with its bin average
• Smooth by boundary snaps each value to the nearer bin edge

IQR Outlier Detection

• Q1 = 25th percentile, Q3 = 75th percentile
• IQR = Q3 − Q1
• Lower bound = Q1 − 1.5 × IQR
• Upper bound = Q3 + 1.5 × IQR
• Any value outside [Lower, Upper] is an outlier

Example (salaries in ₹k: 20, 22, 24, 25, 27, 90) → 90 is an outlier.

---

5. Data Cleaning TechniquesCopy Anchor Link

A repeatable pipeline that turns raw dirty data into analysis-ready data.

Golden rule: Clean a copy, log every change, keep it reproducible — never overwrite raw data.

Pipeline steps

1. Handle Missing (impute or drop)
2. Remove Duplicates
3. Fix Errors (typos, wrong types)
4. Smooth Noise (binning / regression)
5. Handle Outliers (IQR / Z-score)
6. Standardise Formats (dates, units, case)

Before vs After Cleaning Example

• Duplicate removed
• City names standardised (Bangalore → Bengaluru)
• Invalid ages replaced by median
• Missing age imputed

---

6. Data IntegrationCopy Anchor Link

Combining data from multiple sources into one unified, consistent view (usually via ETL / ELT).

Key challenges

• Schema mismatch
• Entity resolution (same entity, different names)
• Data redundancy
• Conflicting values
• Different units / formats

Problems & Fixes

1. Schema mismatch (DOB vs BirthDate) → map columns to common schema
2. Entity resolution (A. Sharma vs Anil Sharma) → key matching / fuzzy matching
3. Redundancy → detect & drop duplicates
4. Value conflicts → rule: trust newer / verified source
5. Unit mismatch (₹ vs $, kg vs lbs, date formats) → convert to one standard

---

7. Data TransformationCopy Anchor Link

Convert data into a form better suited for mining and modelling. Six core operations:

1. Normalisation (Min-Max) — rescale to [0, 1]

   [ x' = \frac{x - \min}{\max - \min} ]

   Prevents large-scale features from dominating.

2. Standardisation (Z-score) — mean 0, std 1

   [ z = \frac{x - \text{mean}}{\text{std}} ]

   Required by distance-based models (KNN, clustering).

3. Aggregation — summarise detail (daily → monthly)

4. Generalisation — replace low-level with high-level (city → state)

5. Encoding — turn categories into numbers

   • Label Encoding — one integer per category (use only for ordinal data)
   • One-Hot Encoding — one 0/1 column per category (safe default for nominal data)

6. Discretisation — split continuous range into bands (age → age groups)

Why rescale? Age (0–100) vs Salary (0–20,00,000) → salary dominates distance calculations unless both are scaled.

---

8. Data Reduction & PreprocessingCopy Anchor Link

Data Reduction — shrink the dataset without losing important signal

• Dimensionality reduction (fewer columns, e.g., PCA)
• Numerosity reduction (fewer rows via sampling)
• Feature selection (keep only useful columns)
• Aggregation (roll up to coarser level)

Data Preprocessing (the umbrella workflow)

1. Cleaning — fix missing, noisy, inconsistent data
2. Integration — merge multiple sources
3. Transformation — normalise, encode, aggregate
4. Reduction — fewer rows/features while preserving signal

This is where the majority of a data scientist’s time is spent.

---

End-to-End Scenario: Swiggy Late DeliveriesCopy Anchor Link

Goal: Predict which orders will be delivered late.

1. Collect — Order logs (DB) + GPS (IoT) + weather API + reviews (text)
2. Assess quality — missing GPS pings, duplicate orders, impossible times
3. Clean — impute missing times, drop duplicates, cap outlier distances
4. Integrate — join on order_id & timestamp
5. Transform — normalise distance, encode weather, extract review sentiment
6. Reduce — keep only the 8 features that actually predict delay

Only after these steps is the data ready for a machine-learning model.

---

Practical ToolsCopy Anchor Link

• Pandas — workhorse for cleaning, merging, transforming tables
• NumPy — fast numeric arrays
• Scikit-learn — scalers, encoders, imputers (StandardScaler, OneHotEncoder, etc.)
• SQL — query & join at the source
• OpenRefine — point-and-click cleaning of messy data
• Google Colab — free cloud notebooks

Simple Pandas example:

df.drop_duplicates() and df.fillna(df.median())

---

Key TakeawaysCopy Anchor Link

1. Know your source & type — Primary vs Secondary; Structured / Semi / Unstructured decides everything downstream.
2. Quality is non-negotiable — Accuracy, Completeness, Consistency… garbage in → garbage out.
3. Cleaning is a pipeline — Missing values, noise, duplicates, formats — always on a copy, logged, reproducible.
4. Integrate + Transform — Unify sources, then normalise/encode so models can actually use the data.
5. Preprocessing ≈ 70% of the job — Master this module and the rest of Data Science becomes far easier.

---