Multiclass Lab

Multi-class Classification Lab

Optional Lab

Goals

• Explore an example of multi-class classification using neural networks
• Understand how neural networks handle multiple output categories
• Visualize decision boundaries and layer functions

Introduction to Multi-class Classification

Applications

Neural Networks are commonly used for classification tasks such as:

• Classifying photo subjects as {dog, cat, horse, other}
• Categorizing sentence elements into parts of speech {noun, verb, adjective, etc}

Note

In multi-class networks, the final layer has multiple units - one output for each category. The highest value output indicates the predicted category.

Data Preparation and Visualization

• Created a 4-class dataset using Scikit-Learn’s make_blobs function
• Each example has 2 input features (x₀, x₁)
• Examples are color-coded based on their class (0 through 3)

Tip

While generated, this dataset represents many real-world classification problems where several input features map to multiple possible output categories.

Dataset Properties:

• Four unique classes: [0, 1, 2, 3]
• Class representation is an integer for each example
• Shape: X_train is (100, 2), y_train is (100,)
• 100 examples with 2 features each

Model Architecture

Network Design

• Simple 2-layer neural network:
• Input layer: 2 features (x₀, x₁)
• Hidden layer: 2 units with ReLU activation
• Output layer: 4 units with linear activation (one per class)

Neural Network Architecture

tf.random.set_seed(1234)  # for consistent results
model = Sequential([
 Dense(2, activation = 'relu',   name = "L1"),
 Dense(4, activation = 'linear', name = "L2")
])

Compilation and Training

Model Training

model.compile(
 loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
 optimizer=tf.keras.optimizers.Adam(0.01),
)

model.fit(
 X_train,y_train,
 epochs=200
)

Caution

Note the from_logits=True parameter in the loss function - this specifies that the output activation is linear rather than softmax, which is more numerically stable during training.

Understanding How the Network Works

Network Internals

Layer 1 Analysis

• The first layer creates a transformation of the input space:
• Unit 0 separates classes 0 and 1 from classes 2 and 3
  • Points to the left of its decision boundary output zero
  • Points to the right output a positive value
• Unit 1 separates classes 0 and 2 from classes 1 and 3
  • Points above its decision boundary output zero
  • Points below output a positive value

Note

The ReLU activation creates a clear separation: outputs are either 0 or a positive value, creating distinct regions in the feature space.

Layer 2 (Output Layer) Analysis

• This layer operates on the “new features” created by Layer 1
• Each unit in Layer 2 targets one specific class:
• Unit 0 produces maximum values near (0,0) where class 0 has been mapped
• Unit 1 targets the upper left corner (class 1)
• Unit 2 targets the lower right corner (class 2)
• Unit 3 targets the upper right (class 3)

Coordinated Values

• Values are coordinated between units
• Each unit must produce the highest value for its target class
• This coordination is handled by the implicit softmax in the loss function
• Unlike other activations, softmax works across all outputs together

Decision Boundaries

• The model successfully partitions the input space
• Each region corresponds to a different predicted class
• The simple model effectively classifies all training examples correctly

Note

Neural networks automatically learn the appropriate decision boundaries to separate different classes, even with a relatively simple architecture as shown in this example.

Neural networks can effectively solve multi-class classification problems by using multiple output units - one for each possible class. During training, the network learns to coordinate these outputs so that the correct class unit produces the highest value for each input example. While you don’t need to understand all the internal details to use neural networks successfully, seeing how each layer transforms the data provides valuable intuition about their operation.