Neurons, Weights, and Layers

Neural networks are inspired by the human brain, but they're actually mathematical functions composed of simple building blocks. Understanding these fundamentals is key to grasping how deep learning works.

What You'll Learn

What artificial neurons are and how they work
How weights and biases affect neuron behavior
How neurons are organized into layers
How information flows through a neural network
The difference between biological and artificial neurons

What is an Artificial Neuron?

An artificial neuron (also called a perceptron) is the basic building block of neural networks. It's a mathematical function that:

Takes multiple inputs (numbers)
Applies weights to each input
Sums them up with a bias
Passes the result through an activation function
Produces an output (number)

Mathematical Representation

Output = Activation(Weight₁ × Input₁ + Weight₂ × Input₂ + ... + Weightₙ × Inputₙ + Bias)

Simple Example

Imagine a neuron that predicts house prices based on size and location:

Input₁: House size (2000 sq ft)
Input₂: Distance from city center (5 miles)
Weight₁: 0.1 (importance of size)
Weight₂: -0.05 (importance of distance)
Bias: 100,000 (base price)

Calculation: 0.1 × 2000 + (-0.05) × 5 + 100,000 = 200 - 0.25 + 100,000 = 100,199.75

Weights: The Learning Parameters

Weights determine how important each input is to the neuron's decision.

How Weights Work

Positive weights: Increase the output
Negative weights: Decrease the output
Large weights: Make the input very important
Small weights: Make the input less important
Zero weights: Ignore the input completely

Example: Email Spam Detection

Input: "Free money" (1 if present, 0 if not)
Weight: 2.5 (high positive weight)
Result: Strong signal that this is spam

Bias: The Baseline

Bias is a constant value added to the weighted sum. It allows the neuron to have a baseline output even when all inputs are zero.

Why Bias Matters

Without bias: Neuron can only output zero when all inputs are zero
With bias: Neuron can learn to output any baseline value
Example: A neuron predicting "likelihood of rain" might have a bias of 0.3 (30% chance even with no weather indicators)

Activation Functions

Activation functions transform the weighted sum into the neuron's final output. They introduce non-linearity, which is essential for neural networks to learn complex patterns.

Common Activation Functions

1. ReLU (Rectified Linear Unit)

f(x) = max(0, x)

Pros: Simple, fast, helps with vanishing gradients
Use case: Most common in hidden layers

2. Sigmoid

f(x) = 1 / (1 + e^(-x))

Pros: Outputs between 0 and 1
Use case: Output layer for binary classification

3. Tanh (Hyperbolic Tangent)

f(x) = (e^x - e^(-x)) / (e^x + e^(-x))

Pros: Outputs between -1 and 1
Use case: Hidden layers, especially in RNNs

Layers: Organizing Neurons

Neural networks organize neurons into layers to process information hierarchically.

Types of Layers

1. Input Layer

Purpose: Receives the raw data
Neurons: One per input feature
Example: For image classification, one neuron per pixel

2. Hidden Layers

Purpose: Process and transform the data
Neurons: Can have any number (architecture choice)
Example: Multiple layers that learn increasingly complex features

3. Output Layer

Purpose: Produces the final prediction
Neurons: Depends on the task
Example: 10 neurons for classifying 10 different objects

Example: Simple Neural Network

Input Layer (3 neurons) → Hidden Layer (4 neurons) → Output Layer (1 neuron)

House Price Prediction:

Input: [Size, Bedrooms, Age]
Hidden: Learns features like "luxury indicator", "location quality"
Output: Predicted price

Information Flow Through Layers

Forward Propagation

Input Layer: Receives raw data
Hidden Layers: Each neuron computes weighted sum + activation
Output Layer: Produces final prediction

Example Flow

Input: [2000, 3, 10] (size, bedrooms, age)

Hidden Layer 1:
- Neuron 1: Learns "size importance"
- Neuron 2: Learns "bedroom importance" 
- Neuron 3: Learns "age importance"
- Neuron 4: Learns "combined features"

Output Layer:
- Single neuron: Combines all features to predict price

Biological vs Artificial Neurons

Similarities

Both receive multiple inputs
Both produce a single output
Both can be connected in networks

Key Differences

| Aspect | Biological Neuron | Artificial Neuron | |--------|------------------|-------------------| | Inputs | Electrical signals | Numbers | | Processing | Chemical reactions | Mathematical operations | | Output | Action potential | Continuous number | | Learning | Synaptic plasticity | Weight updates | | Speed | Milliseconds | Nanoseconds |

Common Misconceptions

❌ "Neurons work like brain cells"

Reality: They're mathematical functions inspired by biology

❌ "More neurons always means better performance"

Reality: Too many neurons can cause overfitting

❌ "Weights are always positive"

Reality: Weights can be positive, negative, or zero

Summary

Artificial neurons are mathematical functions with inputs, weights, bias, and activation
Weights determine the importance of each input
Bias provides a baseline output
Activation functions introduce non-linearity
Layers organize neurons for hierarchical processing
Neural networks are inspired by biology but work mathematically

Self-Check

What are the three main components of an artificial neuron?
How do weights affect a neuron's output?
Why do we need activation functions?
What's the difference between input, hidden, and output layers?

Next Lesson: ReLU, Sigmoid, Tanh: Activation Functions