DS Unit 3: Complete Concept Guide

1. Introduction to StatisticsCopy Anchor Link

1.1 What is Statistics?Copy Anchor Link

Statistics is the science of collecting, organizing, summarizing, analyzing, and interpreting data to make informed decisions. In Data Science, statistics provides the mathematical foundation for understanding data patterns, quantifying uncertainty, and validating hypotheses.

1.2 Branches of StatisticsCopy Anchor Link

1.3 Key TerminologyCopy Anchor Link

• Population: The entire group of interest (e.g., all customers of a company).
• Sample: A subset of the population used for analysis.
• Parameter: A numerical summary of a population (e.g., population mean μ).
• Statistic: A numerical summary of a sample (e.g., sample mean x̄).
• Variable: A characteristic being measured (e.g., age, income).
• Observation: A single data point or record.

1.4 Types of DataCopy Anchor Link

2. Descriptive StatisticsCopy Anchor Link

Descriptive statistics summarize data using measures of central tendency and measures of dispersion.

2.1 Measures of Central TendencyCopy Anchor Link

These measures identify the “center” or “typical value” of a dataset.

(a) Mean (Arithmetic Average)Copy Anchor Link

• Formula (Population): μ = (Σxᵢ) / N
• Formula (Sample): x̄ = (Σxᵢ) / n
• Use: Best for symmetric, interval/ratio data without outliers.
• Limitation: Highly sensitive to outliers.

Example: Data: 10, 20, 30, 40, 50 Mean = (10+20+30+40+50)/5 = 30

(b) MedianCopy Anchor Link

• The middle value when data is sorted.
• If n is odd: middle value.
• If n is even: average of the two middle values.
• Use: Best for skewed data or data with outliers.

Example: Data: 10, 20, 30, 40, 1000 Median = 30 (robust to the outlier 1000)

(c) ModeCopy Anchor Link

• The most frequently occurring value.
• A dataset can have no mode, one mode (unimodal), two modes (bimodal), or multiple modes.
• Use: Best for categorical/nominal data.

Example: Data: 2, 3, 3, 5, 7, 7, 7, 9 Mode = 7

Comparison of Central Tendency MeasuresCopy Anchor Link

Skewness and Central TendencyCopy Anchor Link

• Symmetric distribution: Mean = Median = Mode
• Right-skewed (positive): Mode < Median < Mean
• Left-skewed (negative): Mean < Median < Mode

2.2 Measures of DispersionCopy Anchor Link

Dispersion measures how spread out the data is around the center.

(a) RangeCopy Anchor Link

• Formula: Range = Maximum − Minimum
• Pros: Simple to compute.
• Cons: Only uses two values; sensitive to outliers.

Example: Data: 5, 10, 15, 20 → Range = 20 − 5 = 15

(b) VarianceCopy Anchor Link

• Measures the average squared deviation from the mean.
• Population Variance: σ² = Σ(xᵢ − μ)² / N
• Sample Variance: s² = Σ(xᵢ − x̄)² / (n − 1)
• Note: Dividing by (n−1) gives an unbiased estimator (Bessel’s correction).

(c) Standard Deviation (SD)Copy Anchor Link

• The square root of variance; expressed in the same units as the data.
• Population SD: σ = √σ²
• Sample SD: s = √s²
• Interpretation: Larger SD → more spread; smaller SD → data clustered near mean.

Example: Data: 2, 4, 4, 4, 5, 5, 7, 9 Mean = 5 Variance (population) = [(−3)²+(−1)²+(−1)²+(−1)²+0²+0²+2²+4²]/8 = (9+1+1+1+0+0+4+16)/8 = 32/8 = 4 SD = √4 = 2

(d) Interquartile Range (IQR)Copy Anchor Link

• Formula: IQR = Q3 − Q1
• Q1 = 25th percentile, Q3 = 75th percentile.
• Use: Robust measure of spread; used in box plots and outlier detection.
• Outlier rule: Value < Q1 − 1.5×IQR or > Q3 + 1.5×IQR → potential outlier.

(e) Coefficient of Variation (CV)Copy Anchor Link

• Formula: CV = (SD / Mean) × 100%
• Use: Compares variability between datasets with different units or means.

Summary Table of Dispersion MeasuresCopy Anchor Link

2.3 Five-Number SummaryCopy Anchor Link

A compact description of a dataset:

1. Minimum
2. Q1 (25th percentile)
3. Median (Q2)
4. Q3 (75th percentile)
5. Maximum

Use: Foundation of the box plot.

3. Basics of ProbabilityCopy Anchor Link

3.1 Core ConceptsCopy Anchor Link

• Experiment: A process with uncertain outcomes (e.g., tossing a coin).
• Sample Space (S): All possible outcomes (e.g., {H, T}).
• Event (E): A subset of the sample space (e.g., getting Heads).
• Probability: P(E) = Number of favorable outcomes / Total outcomes
  • 0 ≤ P(E) ≤ 1
  • P(S) = 1, P(∅) = 0

3.2 Rules of ProbabilityCopy Anchor Link

3.3 Random VariablesCopy Anchor Link

• Discrete: Countable outcomes (e.g., number of heads in 10 tosses).
• Continuous: Infinite outcomes in a range (e.g., height, time).

3.4 Probability DistributionsCopy Anchor Link

3.5 Normal Distribution (Gaussian)Copy Anchor Link

• Bell-shaped, symmetric around mean μ.
• Empirical Rule (68–95–99.7):
  • ~68% of data within μ ± 1σ
  • ~95% within μ ± 2σ
  • ~99.7% within μ ± 3σ
• Standard Normal (Z): μ = 0, σ = 1; Z = (x − μ) / σ
• Central Limit Theorem (CLT): The sampling distribution of the mean approaches normality as sample size increases (n ≥ 30), regardless of population distribution.

4. Correlation and CovarianceCopy Anchor Link

4.1 CovarianceCopy Anchor Link

• Measures the direction of the linear relationship between two variables.
• Formula (Sample): Cov(X, Y) = Σ[(xᵢ − x̄)(yᵢ − ȳ)] / (n − 1)
• Interpretation:
  • Cov > 0 → X and Y tend to move together.
  • Cov < 0 → X and Y tend to move in opposite directions.
  • Cov ≈ 0 → No linear relationship.
• Limitation: Magnitude depends on units → hard to interpret.

4.2 Correlation (Pearson’s r)Copy Anchor Link

• Standardized measure of linear relationship; unit-free.
• Formula: r = Cov(X, Y) / (σₓ × σᵧ)
• Range: −1 ≤ r ≤ +1

4.3 Spearman’s Rank CorrelationCopy Anchor Link

• Measures monotonic (not necessarily linear) relationships.
• Based on ranks rather than raw values.
• Use: Non-linear relationships, ordinal data, or when outliers exist.

4.4 Important NotesCopy Anchor Link

• Correlation ≠ Causation.
• Correlation only captures linear relationships (for Pearson).
• Always visualize data (scatter plot) alongside correlation.

5. Exploratory Data Analysis (EDA)Copy Anchor Link

5.1 What is EDA?Copy Anchor Link

EDA is the process of exploring data visually and statistically to:

• Understand data structure, patterns, and anomalies.
• Form hypotheses for further analysis.
• Guide feature selection and model building.

Coined by: John Tukey (1977)

5.2 Goals of EDACopy Anchor Link

1. Maximize insight into a dataset.
2. Uncover underlying structure.
3. Extract important variables.
4. Detect outliers and anomalies.
5. Test underlying assumptions.
6. Develop parsimonious models.

5.3 Steps in EDACopy Anchor Link

Step 1: Data UnderstandingCopy Anchor Link

• Check dimensions (rows × columns).
• Inspect data types (numeric, categorical, datetime).
• View first/last rows, summary statistics.

Step 2: Data CleaningCopy Anchor Link

• Handle missing values (imputation or removal).
• Fix inconsistent formats.
• Remove duplicates.

Step 3: Univariate AnalysisCopy Anchor Link

• Analyze one variable at a time.
• Numeric: Histogram, box plot, mean, median, SD.
• Categorical: Bar chart, frequency table, mode.

Step 4: Bivariate AnalysisCopy Anchor Link

• Analyze relationships between two variables.
• Numeric vs Numeric: Scatter plot, correlation.
• Categorical vs Numeric: Box plot, group means.
• Categorical vs Categorical: Contingency table, stacked bar chart.

Step 5: Multivariate AnalysisCopy Anchor Link

• Analyze three or more variables simultaneously.
• Techniques: Pair plots, correlation heatmaps, 3D scatter plots, PCA.

Step 6: Pattern Identification and Trend AnalysisCopy Anchor Link

• Identify seasonality, cycles, trends in time-series data.
• Detect clusters, correlations, and associations.
• Use moving averages, decomposition, and aggregation.

5.4 Common EDA TechniquesCopy Anchor Link

5.5 Outlier DetectionCopy Anchor Link

Methods:

1. IQR Method: Outliers < Q1 − 1.5×IQR or > Q3 + 1.5×IQR
2. Z-score Method: |Z| > 3 → outlier (Z = (x − μ)/σ)
3. Visual: Box plots, scatter plots
4. Isolation Forest / DBSCAN: Machine learning-based methods

Handling Outliers:

• Remove (if erroneous)
• Cap/Winsorize (replace with percentile values)
• Transform (log, square root)
• Keep (if genuine and important)

5.6 Missing Data HandlingCopy Anchor Link

5.7 Pattern Identification and Trend AnalysisCopy Anchor Link

• Trend: Long-term increase or decrease.
• Seasonality: Regular periodic fluctuations.
• Cyclical: Irregular long-term oscillations.
• Noise: Random variation.

Tools:

• Moving averages
• Exponential smoothing
• Time-series decomposition (trend + seasonal + residual)
• Autocorrelation (ACF/PACF)

6. EDA in Practice — Python ExampleCopy Anchor Link

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

# Load data
df = pd.read_csv('data.csv')

# 1. Basic info
print(df.shape)
print(df.info())
print(df.describe())

# 2. Missing values
print(df.isnull().sum())

# 3. Univariate - histogram
df['age'].hist(bins=20)
plt.title('Age Distribution')
plt.show()

# 4. Box plot
sns.boxplot(x=df['income'])
plt.show()

# 5. Bivariate - scatter
sns.scatterplot(x='age', y='income', data=df)
plt.show()

# 6. Correlation matrix
corr = df.corr(numeric_only=True)
sns.heatmap(corr, annot=True, cmap='coolwarm')
plt.show()

# 7. Pair plot
sns.pairplot(df[['age', 'income', 'score']])
plt.show()

# 8. Outlier detection via IQR
Q1 = df['income'].quantile(0.25)
Q3 = df['income'].quantile(0.75)
IQR = Q3 - Q1
outliers = df[(df['income'] < Q1 - 1.5*IQR) | (df['income'] > Q3 + 1.5*IQR)]
print(f"Outliers:{len(outliers)}")

7. Key Formulas — Quick ReferenceCopy Anchor Link

8. Summary — Unit III at a GlanceCopy Anchor Link

STATISTICAL FOUNDATIONS & EDA
│
├── Introduction to Statistics
│   ├── Descriptive vs Inferential
│   ├── Population vs Sample
│   └── Types of Data (Nominal, Ordinal, Interval, Ratio)
│
├── Descriptive Statistics
│   ├── Central Tendency: Mean, Median, Mode
│   └── Dispersion: Range, Variance, SD, IQR, CV
│
├── Probability Basics
│   ├── Rules & Conditional Probability
│   ├── Bayes' Theorem
│   └── Distributions (Normal, Binomial, Poisson...)
│
├── Correlation & Covariance
│   ├── Covariance (direction)
│   ├── Pearson's r (strength + direction)
│   └── Spearman's rank (monotonic)
│
└── Exploratory Data Analysis
    ├── Univariate → Bivariate → Multivariate
    ├── Outlier detection (IQR, Z-score)
    ├── Missing data handling
    └── Pattern & trend analysis

9. Exam-Focused PointsCopy Anchor Link

1. Difference between mean, median, mode — when to use each.
2. Variance vs Standard Deviation — SD is in original units.
3. Covariance vs Correlation — correlation is standardized.
4. Correlation ≠ Causation — classic exam question.
5. IQR method for outliers — memorize the 1.5×IQR rule.
6. Empirical rule (68-95-99.7) — for normal distributions.
7. Central Limit Theorem — n ≥ 30 rule of thumb.
8. EDA steps — univariate → bivariate → multivariate.
9. Skewness and mean/median relationship.
10. When to use Spearman vs Pearson correlation.