neural networks from scratch pdf

Astor Nivec

4 min read

·

01-10-2024

72

0

Share

Neural networks have revolutionized the field of artificial intelligence, enabling breakthroughs in various applications like image recognition, natural language processing, and more. In this article, we will explore how to create neural networks from scratch, providing a foundational understanding along with practical examples.

Table of Contents

• What Are Neural Networks?
• Key Components of Neural Networks
• Creating a Simple Neural Network from Scratch
• Practical Example: Building a Neural Network to Classify Handwritten Digits
• Conclusion

What Are Neural Networks?

Neural networks are computational models inspired by the human brain's interconnected neuron structure. They consist of layers of nodes (neurons) that process input data, enabling the network to learn and make predictions. The primary goal of neural networks is to approximate complex functions through training on labeled data.

Key Components of Neural Networks

Before we jump into building a neural network, let's review some key components:

• Neurons: The basic units of a neural network that receive inputs, apply a weight, and pass the output through an activation function.
• Layers: Neural networks consist of input layers, hidden layers, and output layers. Each layer serves a unique purpose in processing data.
• Activation Functions: Functions such as sigmoid, ReLU, and tanh that determine the output of a neuron based on its input.
• Weights and Biases: Parameters adjusted during training to minimize the difference between the predicted output and the actual output.
• Loss Function: A method to evaluate how well the network's predictions match the actual outcomes, guiding the training process.
• Optimizer: Algorithms like SGD (Stochastic Gradient Descent) that update weights and biases to minimize the loss function.

Creating a Simple Neural Network from Scratch

To create a basic neural network, we will use Python and its libraries. Here’s a high-level overview of the steps involved:

1. Initialize Weights and Biases: Randomly initialize weights and biases for the neurons.
2. Forward Pass: Compute the predicted output by passing inputs through the network.
3. Compute Loss: Use a loss function to quantify the error in predictions.
4. Backpropagation: Adjust weights and biases to minimize loss.
5. Repeat: Iterate the forward pass and backpropagation steps until convergence.

Here's a simplified code example:

import numpy as np

# Sigmoid activation function
def sigmoid(x):
    return 1 / (1 + np.exp(-x))

# Derivative of sigmoid function
def sigmoid_derivative(x):
    return x * (1 - x)

# Neural Network Class
class SimpleNN:
    def __init__(self, input_size, hidden_size, output_size):
        self.weights_input_hidden = np.random.uniform(size=(input_size, hidden_size))
        self.weights_hidden_output = np.random.uniform(size=(hidden_size, output_size))
        self.bias_hidden = np.random.uniform(size=(1, hidden_size))
        self.bias_output = np.random.uniform(size=(1, output_size))

    def forward(self, x):
        self.hidden_layer_activation = np.dot(x, self.weights_input_hidden) + self.bias_hidden
        self.hidden_layer_output = sigmoid(self.hidden_layer_activation)
        self.output_layer_activation = np.dot(self.hidden_layer_output, self.weights_hidden_output) + self.bias_output
        return sigmoid(self.output_layer_activation)

    def backward(self, x, y, learning_rate=0.1):
        output_error = y - self.output_layer_output
        output_delta = output_error * sigmoid_derivative(self.output_layer_output)

        hidden_layer_error = output_delta.dot(self.weights_hidden_output.T)
        hidden_layer_delta = hidden_layer_error * sigmoid_derivative(self.hidden_layer_output)

        # Updating Weights and Biases
        self.weights_hidden_output += self.hidden_layer_output.T.dot(output_delta) * learning_rate
        self.weights_input_hidden += x.T.dot(hidden_layer_delta) * learning_rate
        self.bias_output += np.sum(output_delta, axis=0, keepdims=True) * learning_rate
        self.bias_hidden += np.sum(hidden_layer_delta, axis=0, keepdims=True) * learning_rate

# Example Usage
if __name__ == "__main__":
    nn = SimpleNN(input_size=2, hidden_size=2, output_size=1)
    
    # Sample data (XOR Problem)
    X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
    y = np.array([[0], [1], [1], [0]])  # Target outputs for XOR

    # Training the neural network
    for epoch in range(10000):
        nn.forward(X)
        nn.backward(X, y)

    # Testing the neural network
    print("Predictions after training:")
    for i in X:
        print(f"Input: {i}, Predicted Output: {nn.forward(i.reshape(1, -1))}")

In this example, we created a basic neural network that learns to solve the XOR problem, which is a classic example of a non-linear separable problem.

Practical Example: Building a Neural Network to Classify Handwritten Digits

To further illustrate the power of neural networks, let’s consider a practical application: classifying handwritten digits (0-9) using the MNIST dataset. The MNIST dataset contains 60,000 training images and 10,000 testing images of handwritten digits.

1. Data Preparation: Load and preprocess the MNIST dataset.
2. Model Architecture: Design a neural network with an input layer (784 neurons for a 28x28 image), a hidden layer (e.g., 128 neurons), and an output layer (10 neurons for each digit).
3. Training: Use backpropagation to adjust weights based on training data.
4. Evaluation: Test the model against the testing dataset.

This process can be performed using popular libraries like TensorFlow or PyTorch for simplicity and efficiency. However, understanding the underlying mathematics and implementation from scratch provides valuable insights into how these frameworks operate.

Conclusion

Building a neural network from scratch offers a profound understanding of its inner workings. By experimenting with the code provided in this article, you can delve deeper into neural network design, training processes, and performance optimization. While pre-built libraries streamline the process, knowing the fundamentals empowers you to innovate and troubleshoot effectively.

For further reading, consider exploring resources such as:

• Neural Networks and Deep Learning
• Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow

Additional Resources

• GitHub Repository: Check out various implementations of neural networks in Python.
• Online Courses: Platforms like Coursera and Udacity offer in-depth courses on deep learning.

By exploring the foundational concepts of neural networks and implementing your own models, you are well on your way to becoming proficient in this exciting field of artificial intelligence.

This article is a summary and expansion based on various sources, including community discussions and implementations available on GitHub. Please ensure proper attribution and acknowledge contributions when using or modifying code from publicly available repositories.