Machine Learning with Python:
Scikit-learn, NumPy & Deep Learning

01 — Foundations

ML Fundamentals & the Python Stack

the ecosystem · core concepts · ML taxonomy

Machine learning is a subset of artificial intelligence where systems learn patterns from data rather than following explicitly programmed rules. At its core, an ML model is a mathematical function that maps inputs to outputs — the "learning" is the process of finding the parameters of that function that minimize error on training data.

Python became the dominant language for ML due to its clean syntax, a world-class library ecosystem, and the research community adopting it as a standard. The essential Python ML stack consists of four foundational libraries that work seamlessly together.

The ML taxonomy divides algorithms into three major learning paradigms. Supervised learning trains on labeled (input, output) pairs to predict outputs for new inputs. Unsupervised learning discovers hidden structure in unlabeled data. Reinforcement learning trains agents that learn by interacting with an environment and receiving rewards or penalties — the approach behind AlphaGo, ChatGPT's RLHF, and robotics.

02 — Data Foundations

NumPy & Pandas for ML

arrays · dataframes · EDA · data cleaning

Before any model can be trained, raw data must be loaded, explored, cleaned, and transformed into numerical arrays. NumPy provides the core ndarray object — an efficient, typed, multi-dimensional array that supports vectorized operations (no Python loops needed). Pandas wraps NumPy arrays in labeled DataFrames that make real-world tabular data management intuitive.

Exploratory Data Analysis (EDA) with Pandas is the first mandatory step: understanding distributions, identifying missing values, detecting outliers, and discovering relationships between features. Skipping EDA almost always leads to poor model performance or subtle data leakage bugs.

import numpy as np
import pandas as pd
from sklearn.datasets import load_breast_cancer

data = load_breast_cancer()
df = pd.DataFrame(data.data, columns=data.feature_names)
df['target'] = data.target

print("Shape:", df.shape)
print("\nClass distribution:")
print(df['target'].value_counts())
print("\nMissing values:", df.isnull().sum().sum())
print("\nFeature stats:")
print(df.describe().round(2).iloc[:, :4])

X = df.drop('target', axis=1).values
y = df['target'].values
print(f"\nX shape: {X.shape}, y shape: {y.shape}")
print(f"Data type: {X.dtype}, Range: [{X.min():.3f}, {X.max():.3f}]")

Key Pandas operations every ML engineer must know include groupby for aggregation, merge/join for combining datasets, fillna for imputing missing values, get_dummies for one-hot encoding, and apply for custom transformations. NumPy's np.where, np.log1p, and broadcasting enable fast feature transforms without loops.

03 — Supervised Learning

Supervised Learning — Classification & Regression

logistic regression · SVM · KNN · linear regression · decision trees

Supervised learning is the most widely deployed ML paradigm. A model learns from a training set of (X, y) pairs — inputs paired with known outputs — and generalizes to predict outputs for unseen inputs. The two main categories are classification (predicting discrete labels) and regression (predicting continuous values).

Scikit-learn's consistent API makes algorithm comparison straightforward: every estimator implements .fit(X_train, y_train) and .predict(X_test), allowing you to swap algorithms with minimal code changes. This is by design — the right algorithm choice depends on data size, feature type, linearity, and interpretability requirements.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import accuracy_score, classification_report

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

scaler = StandardScaler()
X_train_sc = scaler.fit_transform(X_train)
X_test_sc  = scaler.transform(X_test)

models = {
    'Logistic Regression': LogisticRegression(max_iter=1000),
    'SVM (RBF)':          SVC(kernel='rbf', C=1.0, probability=True),
    'KNN (k=5)':          KNeighborsClassifier(n_neighbors=5),
}

for name, model in models.items():
    model.fit(X_train_sc, y_train)
    acc = accuracy_score(y_test, model.predict(X_test_sc))
    print(f"{name:22s}  Accuracy: {acc:.4f}")

Classification vs Regression: Key Algorithms

04 — Unsupervised Learning

Unsupervised Learning — Clustering & Dimensionality Reduction

k-means · DBSCAN · PCA · t-SNE · UMAP

Unsupervised learning operates on data without labels. Its goals include discovering natural groupings (clustering), compressing high-dimensional data while preserving structure (dimensionality reduction), detecting anomalies, and learning generative models of data distributions. These techniques are essential for data exploration, pre-processing high-dimensional data before supervised learning, and recommendation systems.

import numpy as np
from sklearn.datasets import load_iris
from sklearn.cluster import KMeans, DBSCAN
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import silhouette_score

X, y_true = load_iris(return_X_y=True)
X_sc = StandardScaler().fit_transform(X)

pca = PCA(n_components=2)
X_2d = pca.fit_transform(X_sc)
print(f"Explained variance: {pca.explained_variance_ratio_.sum():.3f}")

kmeans = KMeans(n_clusters=3, random_state=42, n_init=10)
labels = kmeans.fit_predict(X_sc)
sil = silhouette_score(X_sc, labels)
print(f"KMeans silhouette score: {sil:.4f}")
print(f"Cluster sizes: {np.bincount(labels)}")

dbscan = DBSCAN(eps=0.5, min_samples=5)
db_labels = dbscan.fit_predict(X_sc)
n_clusters = len(set(db_labels)) - (1 if -1 in db_labels else 0)
n_noise = (db_labels == -1).sum()
print(f"DBSCAN: {n_clusters} clusters, {n_noise} noise points")

Choosing between clustering algorithms: K-Means assumes spherical clusters of similar size and requires specifying k in advance — use the elbow method or silhouette analysis to select it. DBSCAN discovers arbitrarily-shaped clusters and automatically identifies noise points, but requires tuning eps and min_samples. Hierarchical clustering builds a dendrogram that shows cluster relationships at every scale, excellent for biological and text data.

Dimensionality reduction serves two distinct purposes. PCA (Principal Component Analysis) is a linear method that maximizes variance retention — ideal for preprocessing before supervised learning, as it removes correlated features and reduces overfitting. t-SNE and UMAP are non-linear methods designed for 2D/3D visualization of high-dimensional data, revealing cluster structure that PCA cannot show.

05 — Feature Engineering

Feature Engineering & Selection

encoding · scaling · interaction features · selection · imputation

Feature engineering is the craft of transforming raw data into representations that capture the signal a model needs. It is consistently the highest-leverage activity in applied ML — a domain expert who understands the data can create features that a model could never learn on its own. Andrew Ng famously described feature engineering as the key differentiator between junior and senior ML practitioners.

import numpy as np
import pandas as pd
from sklearn.preprocessing import (
    StandardScaler, MinMaxScaler, RobustScaler,
    LabelEncoder, OrdinalEncoder
)
from sklearn.impute import SimpleImputer, KNNImputer
from sklearn.feature_selection import SelectKBest, f_classif, RFE
from sklearn.linear_model import LogisticRegression

np.random.seed(42)
df = pd.DataFrame({
    'age':      np.random.randint(18, 70, 200).astype(float),
    'income':   np.random.exponential(50000, 200),
    'category': np.random.choice(['A','B','C'], 200),
    'score':    np.random.normal(0, 1, 200),
})
df.loc[np.random.choice(200, 20, replace=False), 'age'] = np.nan
df['target'] = (df['income'] > 50000).astype(int)

df['age'].fillna(df['age'].median(), inplace=True)
df['income_log'] = np.log1p(df['income'])
df['age_group'] = pd.cut(df['age'], bins=[18,35,50,70], labels=['young','mid','senior'])
df['age_income_interact'] = df['age'] * df['income_log']
df = pd.get_dummies(df, columns=['category', 'age_group'], drop_first=True)

print("Engineered features:", list(df.columns))
print(f"Shape after engineering: {df.shape}")

Scaling Methods: When to Use Each

• StandardScaler (Z-score): subtracts mean, divides by std. Best for SVM, Logistic Regression, PCA, and neural networks. Assumes roughly Gaussian distribution.
• MinMaxScaler: scales to [0, 1]. Good for algorithms sensitive to magnitude (KNN, neural networks). Sensitive to outliers.
• RobustScaler: uses median and IQR. Best when data has significant outliers — the scaler itself ignores them.
• Log / sqrt transform: apply to right-skewed features (income, transaction amounts) before any scaling. Reduces the effect of extreme values.

Feature selection removes irrelevant or redundant features, improving model generalization and training speed. SelectKBest uses statistical tests (F-score, mutual information) to rank features. Recursive Feature Elimination (RFE) trains models iteratively, removing the least important feature each round. LASSO regression (L1 regularization) automatically sets irrelevant feature coefficients to zero, acting as a built-in selector.

06 — Model Evaluation

Model Evaluation, Cross-Validation & Metrics

accuracy · precision · recall · F1 · AUC-ROC · k-fold · stratified CV

Evaluating a model on the same data it was trained on gives a wildly optimistic estimate of real-world performance. Proper evaluation requires held-out data the model has never seen. Cross-validation systematically rotates this held-out set across the full dataset, giving a statistically robust performance estimate without wasting data.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import (
    cross_val_score, StratifiedKFold, cross_validate
)
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import (
    confusion_matrix, classification_report,
    roc_auc_score, average_precision_score
)
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
import numpy as np

X, y = load_breast_cancer(return_X_y=True)

pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('clf',    RandomForestClassifier(n_estimators=100, random_state=42))
])

skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
cv_results = cross_validate(
    pipe, X, y, cv=skf,
    scoring=['accuracy', 'f1', 'roc_auc'],
    return_train_score=True
)
for metric in ['accuracy', 'f1', 'roc_auc']:
    test_scores = cv_results[f'test_{metric}']
    print(f"{metric:12s}  {test_scores.mean():.4f} ± {test_scores.std():.4f}")

Choosing the Right Metric

Accuracy is misleading on imbalanced datasets — a model that always predicts the majority class achieves 99% accuracy on a dataset with 1% positive cases, while being completely useless. Use these guidelines instead:

• Precision: use when false positives are costly (spam detection — don't block legitimate emails).
• Recall: use when false negatives are costly (cancer diagnosis — don't miss a positive case).
• F1 Score: harmonic mean of precision and recall — balanced metric for imbalanced classification.
• AUC-ROC: measures discriminative ability across all classification thresholds. Best single metric for binary classification with class imbalance.
• RMSE / MAE: for regression. RMSE penalizes large errors heavily; MAE is more robust to outliers.

07 — Ensemble Methods

Ensemble Methods — Random Forest & Gradient Boosting

bagging · boosting · random forest · XGBoost · LightGBM · stacking

Ensemble methods combine multiple weak learners into a single strong learner. The two dominant paradigms are bagging (parallel training on random subsets, predictions averaged — reduces variance) and boosting (sequential training where each model corrects the previous one's errors — reduces bias). Both consistently outperform single models on structured/tabular data.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.ensemble import (
    RandomForestClassifier, GradientBoostingClassifier,
    VotingClassifier, StackingClassifier
)
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
import numpy as np

X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)

rf = RandomForestClassifier(
    n_estimators=200, max_depth=10,
    min_samples_leaf=2, random_state=42
)
rf.fit(X_tr, y_tr)
rf_auc = roc_auc_score(y_te, rf.predict_proba(X_te)[:, 1])
print(f"Random Forest AUC: {rf_auc:.4f}")

feat_imp = pd.Series(rf.feature_importances_,
    index=load_breast_cancer().feature_names).nlargest(5)
print("\nTop 5 features:")
for feat, imp in feat_imp.items():
    print(f"  {feat:30s} {imp:.4f}")

gb = GradientBoostingClassifier(
    n_estimators=200, learning_rate=0.05,
    max_depth=4, subsample=0.8, random_state=42
)
gb.fit(X_tr, y_tr)
gb_auc = roc_auc_score(y_te, gb.predict_proba(X_te)[:, 1])
print(f"\nGradient Boosting AUC: {gb_auc:.4f}")

Random Forest vs Gradient Boosting

08 — Deep Learning

Neural Networks & Deep Learning Fundamentals

perceptron · backpropagation · activation functions · CNN · RNN · transformers

Neural networks are universal function approximators — given enough capacity and data, they can learn any input-output mapping. A feedforward neural network (multilayer perceptron) consists of layers of neurons where each neuron computes a weighted sum of its inputs, applies an activation function (introducing non-linearity), and passes the result forward.

Backpropagation is the algorithm that makes learning possible: it computes the gradient of the loss function with respect to every weight in the network using the chain rule of calculus, then an optimizer (SGD, Adam, AdamW) uses those gradients to update weights. Modern deep learning is primarily about architectures that scale well: CNNs for vision, RNNs/LSTMs for sequences, and Transformers for NLP and increasingly everything else.

from sklearn.neural_network import MLPClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import accuracy_score, roc_auc_score
import numpy as np

X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)

sc = StandardScaler()
X_tr_sc = sc.fit_transform(X_tr)
X_te_sc  = sc.transform(X_te)

mlp = MLPClassifier(
    hidden_layer_sizes=(128, 64, 32),
    activation='relu',
    solver='adam',
    alpha=0.001,
    learning_rate_init=0.001,
    max_iter=500,
    early_stopping=True,
    validation_fraction=0.15,
    random_state=42
)
mlp.fit(X_tr_sc, y_tr)

acc = accuracy_score(y_te, mlp.predict(X_te_sc))
auc = roc_auc_score(y_te, mlp.predict_proba(X_te_sc)[:, 1])
print(f"MLP Accuracy: {acc:.4f}")
print(f"MLP AUC-ROC:  {auc:.4f}")
print(f"Best val loss: {mlp.best_loss_:.6f}")
print(f"Training epochs: {mlp.n_iter_}")

Activation Functions

• ReLU (Rectified Linear Unit): max(0, x) — the default choice. Fast, avoids vanishing gradients in hidden layers, but can "die" (neurons stuck at 0).
• Sigmoid: maps to (0, 1) — only for binary classification output neurons. Suffers from vanishing gradients in deep networks.
• Softmax: multi-class classification output — converts logits to a probability distribution summing to 1.
• GELU / Swish: modern activations used in Transformers (BERT, GPT). Smooth approximations of ReLU with better gradient flow.
• Tanh: maps to (-1, 1) — historically used in RNNs; zero-centered, which helps gradient flow but still has vanishing gradient issues.

09 — Production ML

ML Pipelines, Hyperparameter Tuning & Deployment

Pipeline · GridSearchCV · RandomizedSearchCV · MLflow · ONNX · FastAPI serving

A scikit-learn Pipeline chains preprocessing steps and a model into a single object that behaves like an estimator. This prevents data leakage in cross-validation (the scaler is fit only on training folds), makes deployment trivial (one object to serialize), and enables clean hyperparameter tuning across both preprocessing and model parameters simultaneously.

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import RandomizedSearchCV, StratifiedKFold
from sklearn.datasets import load_breast_cancer
import numpy as np
import joblib

X, y = load_breast_cancer(return_X_y=True)

pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('clf', RandomForestClassifier(random_state=42))
])

param_dist = {
    'clf__n_estimators':      [100, 200, 300],
    'clf__max_depth':         [None, 5, 10, 15],
    'clf__min_samples_split': [2, 5, 10],
    'clf__max_features':      ['sqrt', 'log2', 0.5],
    'clf__min_samples_leaf':  [1, 2, 4],
}

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
search = RandomizedSearchCV(
    pipe, param_dist, n_iter=30, cv=cv,
    scoring='roc_auc', n_jobs=-1, random_state=42
)
search.fit(X, y)
print(f"Best AUC: {search.best_score_:.4f}")
print(f"Best params: {search.best_params_}")

joblib.dump(search.best_estimator_, 'model.pkl')
loaded_model = joblib.load('model.pkl')
print(f"Loaded model prediction: {loaded_model.predict(X[:3])}")

ML Deployment Strategies

• Batch prediction: run model on stored data at scheduled intervals (daily/hourly). Simple, scalable, no latency requirements. Use joblib or pickle for model serialization.
• Real-time REST API: serve predictions via FastAPI or Flask. Load the model at startup, expose a /predict endpoint. Containerize with Docker for reproducibility.
• MLflow: open-source MLOps platform for experiment tracking, model versioning, and deployment. Track every training run's metrics, parameters, and artifacts automatically.
• ONNX: export models to the Open Neural Network Exchange format for cross-platform inference — run PyTorch/TensorFlow models in C++, Java, or edge devices.
• Model monitoring: production models degrade over time as data distributions shift (concept drift). Monitor prediction distributions, feature statistics, and business KPIs continuously.

FAQ

Common ML Questions Answered