📝 Gradient Descent Pseudocode

We include the full loop to show how gradient descent is used in real training: multiple epochs, loss tracking, and a stopping condition. This bridges the math update rule and practical implementation.

ALGORITHM: Gradient Descent Training
─────────────────────────────────────────────────────────────────

INPUT:
    X: Training data of shape (n_samples, n_features)
    y: Target values of shape (n_samples,)
    α: Learning rate (e.g., 0.01)
    max_epochs: Maximum number of epochs
    tolerance: Convergence threshold (e.g., 1e-6)
    loss_function: Function to compute loss
    gradient_function: Function to compute gradient

OUTPUT:
    θ: Optimized parameter vector
    history: List of loss values per epoch

PROCEDURE:
    1. Initialize parameters θ randomly or to zeros
    
    2. history ← empty list
    
    3. FOR epoch = 1 TO max_epochs:
        
        a. // Forward pass: compute predictions
           ŷ ← predict(X, θ)
        
        b. // Compute and record loss
           L ← loss_function(y, ŷ)
           APPEND L to history
        
        c. // Compute gradient
           ∇L ← gradient_function(X, y, ŷ, θ)
        
        d. // Update parameters (gradient step)
           θ ← θ - α · ∇L
        
        e. // Check for convergence
           IF ||∇L|| < tolerance:
               PRINT "Converged at epoch", epoch
               BREAK
    
    4. RETURN θ, history

We use SGD when datasets are large or we need very frequent, cheap updates. It introduces useful noise during training, which can help escape local minima and speed up early learning.

ALGORITHM: Stochastic Gradient Descent (One Epoch)
─────────────────────────────────────────────────────────────────

INPUT:
    θ: Current parameter vector
    X: Training data of shape (n_samples, n_features)
    y: Target values of shape (n_samples,)
    α: Learning rate

OUTPUT:
    θ: Updated parameter vector

PROCEDURE:
    1. Shuffle the dataset indices
    
    2. FOR each sample i in shuffled order:
        
        a. Get single sample: xᵢ, yᵢ
        
        b. Compute prediction: ŷᵢ ← predict(xᵢ, θ)
        
        c. Compute gradient for single sample:
           ∇Lᵢ ← gradient(xᵢ, yᵢ, ŷᵢ, θ)
        
        d. Update parameters:
           θ ← θ - α · ∇Lᵢ
    
    3. RETURN θ

We use Mini-Batch GD to balance stability and efficiency: each update sees more than one sample, but not the whole dataset. That is why it is the standard choice in modern deep learning training.

ALGORITHM: Mini-Batch Gradient Descent (One Epoch)
─────────────────────────────────────────────────────────────────

INPUT:
    θ: Current parameter vector
    X: Training data of shape (n_samples, n_features)
    y: Target values of shape (n_samples,)
    α: Learning rate
    batch_size: Number of samples per batch (e.g., 32)

OUTPUT:
    θ: Updated parameter vector

PROCEDURE:
    1. n ← number of samples
    
    2. Shuffle the dataset
    
    3. n_batches ← ceil(n / batch_size)
    
    4. FOR batch_idx = 0 TO n_batches - 1:
        
        a. // Extract mini-batch
           start ← batch_idx × batch_size
           end ← min(start + batch_size, n)
           X_batch ← X[start:end]
           y_batch ← y[start:end]
        
        b. // Forward pass
           ŷ_batch ← predict(X_batch, θ)
        
        c. // Compute gradient on mini-batch
           ∇L ← gradient(X_batch, y_batch, ŷ_batch, θ)
        
        d. // Update parameters
           θ ← θ - α · ∇L
    
    5. RETURN θ