⬡ Hub
Skip to content

Phase 3: Deep Learning & Neural Networks

Deep learning utilizes artificial neural networks with multiple internal ("hidden") layers to model highly complex, non-linear relationships. While traditional ML requires humans to manually extract features from data, deep learning models learn the features automatically natively from the raw data.

1. Artificial Neural Networks (ANNs)

The foundation of deep learning. Data is passed through multiple layers of "neurons", multiplied by weights, passed through activation functions, and outputted.

Key Concepts: - Activation Functions: The mathematical equations that determine the output of a neural network. ReLU (Rectified Linear Unit) is standard for hidden layers to prevent the "vanishing gradient" problem. Sigmoid (0 to 1) is used for binary output. Softmax is used for multi-class probability output. - Optimization Strategy: Neural Networks train by making predictions, observing the error (loss), and using "Backpropagation" combined with algorithms like Adam (Adaptive Moment Estimation) or SGD (Stochastic Gradient Descent) to mathematically tweak the weights to make the error smaller next time. - Overfitting & Regularization: Neural networks are highly prone to overfitting (memorizing the training data). We combat this using Dropout (randomly turning off neurons during training) and Early Stopping (halting training when the test performance stops improving).

Example 1: Creating a Multi-Layer Perceptron (MLP) with TensorFlow/Keras

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout, BatchNormalization
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.callbacks import EarlyStopping

def build_robust_mlp_model(input_dim):
    """
    Builds an industry-standard MLP with Dropout for regularization
    and Batch Normalization for training stability.
    """
    model = Sequential([
        # 1st Hidden Layer (128 neurons)
        Dense(128, input_dim=input_dim, activation='relu'),
        BatchNormalization(),
        Dropout(0.3), # Drops 30% of neurons randomly to prevent overfitting

        # 2nd Hidden Layer (64 neurons)
        Dense(64, activation='relu'),
        BatchNormalization(),
        Dropout(0.2),

        # Output Layer (Binary Classification: values 0 or 1)
        Dense(1, activation='sigmoid')
    ])

    # Compile the model specifying the optimizer and loss function
    # Binary Crossentropy is the standard loss for Binary Classification
    model.compile(optimizer=Adam(learning_rate=0.001),
                  loss='binary_crossentropy',
                  metrics=['accuracy'])
    return model

# Example Usage:
# Assume X_train represents 50 features per sample
# model = build_robust_mlp_model(input_dim=50)
# early_stopping = EarlyStopping(patience=5, restore_best_weights=True)
# history = model.fit(X_train, y_train, epochs=100, validation_split=0.2, callbacks=[early_stopping])

2. Convolutional Neural Networks (CNNs)

Designed specifically for grid-like data, such as images. Standard ANNs flatten an image into a 1D line, losing spatial information (like the fact that two pixels are physically next to each other). CNNs preserve this 2D structure.

Key Concepts: - Convolutional Layers: Slide mathematical "filters" (kernels) across the image to detect features like edges, curves, and eventually complex objects like eyes or wheels. - Pooling Layers: Downscale the image (e.g., MaxPooling takes the largest value in a 2x2 grid), reducing computation while preserving the most prominent features. - Flatten & Dense: Once the CNN has shrunk the image into a tiny map of prominent features, it flattens it and passes it to a standard ANN to make the final prediction.

Example 2: CNN Architecture for Image Classification (PyTorch)

import torch
import torch.nn as nn
import torch.nn.functional as F

class SimpleCNN(nn.Module):
    """
    A foundational CNN architecture implemented in PyTorch.
    """
    def __init__(self, num_classes):
        super(SimpleCNN, self).__init__()

        # Convolutional Block 1 (Input channels: 3 (RGB), Output channels: 32)
        self.conv1 = nn.Conv2d(in_channels=3, out_channels=32, kernel_size=3, padding=1)
        self.pool1 = nn.MaxPool2d(kernel_size=2, stride=2)

        # Convolutional Block 2 (Input channels: 32, Output channels: 64)
        self.conv2 = nn.Conv2d(in_channels=32, out_channels=64, kernel_size=3, padding=1)
        self.pool2 = nn.MaxPool2d(kernel_size=2, stride=2)

        # Fully Connected Block
        # Assuming input images are 32x32. After two 2x2 pools, spatial dimension is 8x8.
        # So we flatten 64 channels * 8 * 8 = 4096
        self.fc1 = nn.Linear(64 * 8 * 8, 512)
        self.dropout = nn.Dropout(0.5)
        self.fc2 = nn.Linear(512, num_classes)

    def forward(self, x):
        # Apply Conv -> ReLU -> Pool
        x = self.pool1(F.relu(self.conv1(x)))
        x = self.pool2(F.relu(self.conv2(x)))

        # Flatten for Dense layer
        x = torch.flatten(x, 1) 

        # Apply FC -> ReLU -> Dropout -> Output
        x = F.relu(self.fc1(x))
        x = self.dropout(x)
        x = self.fc2(x)
        return x

# Example Usage:
# model = SimpleCNN(num_classes=10) # Classifying 10 types of objects
# print(model)

Example 3: Transfer Learning (Industry Standard for CNNs)

In the real world, you almost never train a CNN from scratch. You import a model trained by Google or Microsoft on 14 million images, freeze its core, and only train the final layer on your specific medical/manufacturing images.

import torch
import torch.nn as nn
from torchvision import models

def setup_transfer_learning(num_classes: int):
    """
    Downloads pre-trained ResNet-50 and adapts it for a custom use-case.
    """
    # 1. Load the pre-trained ResNet-50 model
    # Using ResNet50_Weights.DEFAULT loads the best available weights trained on ImageNet
    model = models.resnet50(weights=models.ResNet50_Weights.DEFAULT)

    # 2. Freeze the complex convolutional feature-extraction layers
    for param in model.parameters():
        param.requires_grad = False

    # 3. Replace the final classification layer
    # The new layer automatically has requires_grad=True
    num_ftrs = model.fc.in_features
    model.fc = nn.Linear(num_ftrs, num_classes)

    return model

# Example Usage
# custom_resnet = setup_transfer_learning(num_classes=2) # e.g., Dog vs Cat
# To train this, you only need to pass the parameters of the final layer to the optimizer
# optimizer = torch.optim.Adam(custom_resnet.fc.parameters(), lr=0.001)

3. Recurrent Neural Networks (RNNs)

Designed for sequential data where order matters (e.g., time-series stock data, text sentences, audio waves). Standard ANNs treat inputs independently; RNNs have an internal "memory" state that passes information from previous steps to the current step.

Key Concepts: - The Vanishing Gradient Problem: Standard RNNs "forget" information from early in the sequence rapidly because gradients approach zero during backpropagation. - LSTMs (Long Short-Term Memory): The industry solution to vanishing gradients. They introduce complex "gates" (Forget Gate, Input Gate, Output Gate) that mathematically decide what information to keep in long-term memory and what to throw away.

Example 4: LSTM for Time-Series Forecasting

This example demonstrates predicting future values based on past sequential values.

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dense
import numpy as np

def build_lstm_forecaster(sequence_length, feature_count):
    """
    Builds an LSTM architecture for predicting the next value in a continuous time-series.
    """
    model = Sequential([
        # The input shape requires (number_of_timesteps, number_of_features)
        LSTM(50, activation='relu', input_shape=(sequence_length, feature_count), return_sequences=True),
        # A second LSTM layer. Notice return_sequences=False because we only want a single output now.
        LSTM(50, activation='relu', return_sequences=False),

        # Final output dense layer (Predicting a single continuous value)
        Dense(1)
    ])

    # Compile with Mean Squared Error for regression/forecasting
    model.compile(optimizer='adam', loss='mse')
    return model

# Example Usage:
# Imagine predicting tomorrow's temperature using the prior 10 days of data (temp, humidity, pressure)
# sequence_length = 10 days
# feature_count = 3 (temp, hum, press)
# lstm_model = build_lstm_forecaster(sequence_length=10, feature_count=3)

# Data shape must be [samples, timesteps, features]
# history = lstm_model.fit(X_train_3D, y_train, epochs=20, batch_size=32)