Problem Statement¶

Business Context¶

In hazardous workplaces like construction sites and industrial plants, ensuring worker safety is critical. Head injuries caused by falling objects or accidents are among the leading causes of fatalities. Safety helmets are essential protective equipment, yet compliance with helmet regulations is often inconsistent, particularly in large-scale operations where manual monitoring is inefficient and prone to errors.

SafeGuard Corp aims to address this issue by automating safety monitoring through an advanced image analysis system. By detecting workers and identifying whether they are wearing helmets, this system will enhance compliance, minimize workplace injuries, and reduce human oversight errors.

Objective¶

Given the challenges faced by SafeGuard Corp in ensuring helmet compliance at hazardous workplaces, they have hired you as a Data Scientist to develop an advanced, machine learning-based solution that achieves the following:

  1. Utilize object detection techniques to accurately identify and locate workers in images captured from construction sites and industrial plants.

  2. Implement a classification model that distinguishes whether the detected workers are wearing helmets or not.

  3. Analyze patterns in the collected image data to understand the factors influencing helmet compliance and the common scenarios where lapses occur.

  4. Integrate the system with existing safety protocols to provide real-time alerts and reports to safety officers, enabling prompt corrective actions.

  5. Ensure the solution is scalable to handle increased volumes of image data from multiple sites, while maintaining high accuracy and efficiency.

This automated system aims to enhance compliance, reduce the risk of head injuries, and streamline the monitoring process, ultimately leading to a safer workplace environment.

Data Description¶

The dataset consists of 640 images, equally divided into two categories:

  1. WithHelmet: 320images showing workers wearing helmets.
  2. WithoutHelmet: 320images showing workers not wearing helmets.

Dataset Characteristics:

  1. Variations in Conditions: Images include diverse environments such as construction sites, factories, and industrial settings, with variations in lighting, angles, and worker postures to simulate real-world conditions.
  2. WorkerActivities: Workers are depicted in different actions such as standing, using tools, or moving, ensuring robust model learning for various scenarios.

Importing Necessary Libraries¶

In [62]:
#import nescessary libraries
import os
import random
import cv2 as cv
import torch
from PIL import Image, ImageDraw
import tensorflow as tf
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import random
import numpy as np
from sklearn.metrics import precision_score, recall_score, confusion_matrix, classification_report
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns
from sklearn.model_selection import train_test_split
import pandas as pd
from tensorflow.keras.preprocessing.image import ImageDataGenerator

%matplotlib inline

import warnings
warnings.filterwarnings(
    'ignore',
    message='.*torch.cuda.amp.autocast.*',
    category=FutureWarning
)

Loading the Data¶

In [2]:
from zipfile import ZipFile

zip_file_path = 'HelmetDetectionDataset.zip'  # <-- Update if needed

# Destination folder to extract files (local folder)
extract_to_path = ''

# Unzipping the file
with ZipFile(zip_file_path, 'r') as zip_ref:
    zip_ref.extractall(extract_to_path)

print(f'File unzipped successfully to {extract_to_path}')

File unzipped successfully to
In [3]:
# NOTE: No double-slash — extract_to_path already ends with '/'
training_data_images_path = 'HelmetDetectionDataset/Training/'
validation_data_images_path = 'HelmetDetectionDataset/Validation/'

Exploratory Data Analysis¶

  1. Class Distribution Analysis: This step involves examining how the different classes are distributed across the dataset. For instance, in a helmet detection task, you would check the number of images with people wearing helmets versus those without. This helps identify if there are any imbalances that might affect model performance.

  2. Visualizing Sample Images: In this phase, we take a closer look at some of the images from the dataset. This could include displaying random samples or specific examples from each class. Visualizations help to understand the quality, diversity, and challenges in the data (e.g., lighting, posture, background, etc.), which is important for choosing the right model and preprocessing techniques.

In [4]:
training_images_with_helmet = len(os.listdir(os.path.join(training_data_images_path, 'WorkersWithHelmet')))
training_images_without_helmet = len(os.listdir(os.path.join(training_data_images_path, 'WorkersWithoutHelmet')))
validation_images_with_helmet = len(os.listdir(os.path.join(validation_data_images_path, 'WorkersWithHelmet')))
# BUG FIX: original code had 'WorkersWithHelmet' here — corrected to 'WorkersWithoutHelmet'
validation_images_without_helmet = len(os.listdir(os.path.join(validation_data_images_path, 'WorkersWithoutHelmet')))

categories = ['Train With Helmet', 'Train Without Helmet', 'Validation With Helmet', 'Validation Without Helmet']
values = [training_images_with_helmet, training_images_without_helmet, validation_images_with_helmet, validation_images_without_helmet]

print("Path to training data with helmet:", os.path.join(training_data_images_path, 'WorkersWithHelmet')
      , "| Number of images:", training_images_with_helmet)
print("Path to training data without helmet:", os.path.join(training_data_images_path, 'WorkersWithoutHelmet')
      , "| Number of images:", training_images_without_helmet)
print("Path to validation data with helmet:", os.path.join(validation_data_images_path, 'WorkersWithHelmet')
      , "| Number of images:", validation_images_with_helmet)
print("Path to validation data without helmet:", os.path.join(validation_data_images_path, 'WorkersWithoutHelmet')
      , "| Number of images:", validation_images_without_helmet)

Path to training data with helmet: HelmetDetectionDataset/Training/WorkersWithHelmet | Number of images: 271
Path to training data without helmet: HelmetDetectionDataset/Training/WorkersWithoutHelmet | Number of images: 280
Path to validation data with helmet: HelmetDetectionDataset/Validation/WorkersWithHelmet | Number of images: 40
Path to validation data without helmet: HelmetDetectionDataset/Validation/WorkersWithoutHelmet | Number of images: 40
In [5]:
plt.figure(figsize=(8, 5))
plt.bar(categories, values, color=['green', 'red', 'blue', 'orange'])  # categories = x-axis, values = heights

# Adding titles and labels
plt.title('Distribution of Training and Validation Images', fontsize=14)
plt.xlabel('Categories', fontsize=12)
plt.ylabel('Number of Images', fontsize=12)
plt.grid(axis='y', linestyle='--', alpha=0.7)

# Show the plot
plt.tight_layout()
plt.show()
No description has been provided for this image

Random images from dataset

In [6]:
def plot_random_images(folder_path, title, num_images=5):
    # Filter only valid image extensions
    valid_ext = ('.jpg', '.jpeg', '.png', '.bmp', '.webp')
    image_files = [
        f for f in os.listdir(folder_path)
        if f.lower().endswith(valid_ext)
    ]

    if not image_files:
        print(f"[ERROR] No image files found in: {folder_path}")
        return

    if len(image_files) < num_images:
        print(f"[WARN] Only {len(image_files)} images available, adjusting num_images.")
        num_images = len(image_files)

    selected_images = random.sample(image_files, num_images)

    fig, axes = plt.subplots(1, num_images, figsize=(15, 5))
    if num_images == 1:
        axes = [axes]

    for i, image_file in enumerate(selected_images):
        image_path = os.path.join(folder_path, image_file)
        try:
            img = mpimg.imread(image_path)
            axes[i].imshow(img)
            axes[i].axis('off')
            axes[i].set_title(f'{title} #{i + 1}', fontsize=10)
        except Exception as e:
            print(f"[ERROR] Could not read {image_file}: {e}")

    plt.tight_layout()
    plt.show()
    plt.close(fig)  # Libera memoria
In [7]:
# Paths to the folders
with_helmet_path_training = os.path.join(training_data_images_path, 'WorkersWithHelmet')
without_helmet_path_training = os.path.join(training_data_images_path, 'WorkersWithoutHelmet')
with_helmet_path_validation = os.path.join(validation_data_images_path, 'WorkersWithHelmet')
without_helmet_path_validation = os.path.join(validation_data_images_path, 'WorkersWithoutHelmet')
In [64]:
# Plot 5 random images with helmet from training data
plot_random_images(with_helmet_path_training, 'With Helmet', num_images=5)
No description has been provided for this image
In [65]:
# Plot 5 random images without helmet from training data
plot_random_images(without_helmet_path_training, 'Without Helmet', num_images=5)
No description has been provided for this image

Observations — Exploratory Data Analysis¶

Class Distribution¶

  • The dataset is balanced by design: 271 training images with helmet vs. 280 without helmet, and 40 vs. 40 in the validation set. No class imbalance correction is required (no class weights, oversampling, or undersampling needed).
  • The Train/Validation split ratio is approximately 88/12, which is acceptable given the reduced dataset size (640 total images).

Visual Distribution Between Classes (Critical Finding)¶

Sample images reveal a significant visual asymmetry between the two classes:

Feature WorkersWithHelmet WorkersWithoutHelmet
Framing Full body or mid-body shot Close-up of face / head only
Background context Construction site, scaffolding, machinery Neutral or indoor background
Persons per image Frequently multiple Single person
Facial resolution Low (person further from camera) High (face is the dominant subject)

Object Detection¶

In [66]:
# Load the YOLOv5 small model — best balance of speed and accuracy for person detection
model = torch.hub.load('ultralytics/yolov5', 'yolov5s')  # yolov5s = small variant

requirements: Ultralytics requirement ['urllib3>=2.6.0 ; python_version > "3.8"'] not found, attempting AutoUpdate...

YOLOv5 🚀 2026-3-29 Python-3.11.13 torch-2.11.0 CPU

Fusing layers... 
YOLOv5s summary: 213 layers, 7225885 parameters, 0 gradients, 16.4 GFLOPs
Adding AutoShape...
In [67]:
def detect_and_count_persons(model, image_path, show_image=False):
    """
    Function to detect and count the number of persons in an image.
    - Displays the original image.
    - Prints the count of persons detected.
    - Shows the image with bounding boxes drawn.

    Args:
        image_path (str): Path to the input image.
    """
    # Perform inference
    results = model(image_path)

    # Extract predictions
    detections = results.xyxy[0]  # Get bounding boxes in xyxy format
    person_count = 0

    # Iterate over detections and count 'person' labels (class 0 in COCO)
    for detection in detections:
        class_id = int(detection[5])  # Class ID is in the 6th position (zero-indexed)
        if class_id == 0 and 0.30 < detection[4]:  # 'person' class in COCO
            person_count += 1

    # Print the count of persons detected
    print(f"Number of persons detected: {person_count}")

    # Display the original image
    if show_image:
      original_image = Image.open(image_path)
      plt.figure(figsize=(10, 8))
      plt.imshow(original_image)
      plt.axis('off')
      plt.title('Original Image')
      plt.show()

      # Display the image with bounding boxes
      print("Displaying image with bounding boxes:")
      results.show()  # Shows images with bounding box

    return results
In [68]:
# detect people in an image and plot the image with the corresponding bounding boxes
img_path = os.path.join(with_helmet_path_training, os.listdir(with_helmet_path_training)[1])
results = detect_and_count_persons(model, img_path, show_image=True)

Number of persons detected: 2
No description has been provided for this image 
Displaying image with bounding boxes:
No description has been provided for this image
In [69]:
img_path_2 = os.path.join(without_helmet_path_training, os.listdir(without_helmet_path_training)[1])
# NOTE: For close-up face images, YOLO may detect 0 persons — this is expected behavior.
# Those images are still used directly (as original) in dataset_generator for the WithoutHelmet class.
results_2 = detect_and_count_persons(model, img_path_2, True)

Number of persons detected: 0
No description has been provided for this image 
Displaying image with bounding boxes:
No description has been provided for this image
In [70]:
def crop_person_images(image_path, results, confidence_threshold=0.30):
    """
    Crops images of persons detected by YOLOv5.
    Only crops persons with a confidence score greater than the threshold.

    Args:
        image_path (str): Path to the input image.
        results (YOLO result object): The result object from YOLO inference.
        confidence_threshold (float): Minimum confidence value for cropping persons. Default is 0.80.

    Returns:
        List of PIL Image: List of cropped images of persons detected.
    """
    # Open the original image
    original_image = Image.open(image_path)

    # Extract predictions (xyxy format: [x1, y1, x2, y2, confidence, class_id])
    detections = results.xyxy[0]
    cropped_images = []

    # Iterate over detections and crop images for 'person' class with high confidence
    for detection in detections:
        x1, y1, x2, y2, confidence, class_id = detection.tolist()
        class_id = int(class_id)

        if class_id == 0 and confidence > confidence_threshold:
            # Crop the image based on the bounding box coordinates
            cropped_image = original_image.crop((x1, y1, x2, y2))
            cropped_images.append(cropped_image)

    return cropped_images

Dataset Creation for Image Classification¶

  1. For the helmet detection task, we begin by storing the cropped images of individuals, which are extracted from the original image using the YOLO algorithm. Along with each cropped image, we also store the corresponding label indicating whether the person is wearing a helmet or not. This label serves as the ground truth for training the model.

  2. To ensure consistency in the dataset, we preprocess the cropped images by resizing them to the same dimensions, typically the input size required by the model (e.g., 224x224 for ResNet) and rescaling it by dividing the pixel values by 255. This step ensures that all input images have uniform size, which is crucial for feeding them into the neural network.

Design Decision — Asymmetric YOLO usage:

  • WithHelmet images → YOLO crops detected persons (label=1). Multiple crops per image are possible.
  • WithoutHelmet images → Original image used directly (label=0). YOLO is intentionally skipped because many of these images are close-up face shots where YOLO person detection fails (YOLO is optimized for full-body bounding boxes). Using the original image ensures no close-up data is lost.
In [71]:
# Creating the dataset from the images
def dataset_generator(with_helmet_path, without_helmet_path):

  dataset = []

  all_helmet_images_name = os.listdir(with_helmet_path)
  all_non_helmet_images_name = os.listdir(without_helmet_path)

  for helmet_image_name in all_helmet_images_name:
    image_path = os.path.join(with_helmet_path, helmet_image_name)
    results = detect_and_count_persons(model, image_path=image_path)
    # Pass the YOLO detection results to crop individual persons from the image
    cropped_images = crop_person_images(image_path, results)

    for croped_image in cropped_images:
      dataset.append([croped_image, 1])

  for non_helmet_image_name in all_non_helmet_images_name:
    img_path = os.path.join(without_helmet_path, non_helmet_image_name)
    original_image = Image.open(img_path)
    # WithoutHelmet: use original image directly — intentional.
    # Close-up face shots are not detected by YOLO as 'persons'.
    # Skipping YOLO here ensures all without-helmet samples are retained.
    dataset.append([original_image, 0])

  return dataset
In [72]:
train_dataset = dataset_generator(with_helmet_path_training, without_helmet_path_training)
validation_dataset = dataset_generator(with_helmet_path_validation, without_helmet_path_validation)

Number of persons detected: 1
Number of persons detected: 2
Number of persons detected: 2
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 2
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 2
Number of persons detected: 3
Number of persons detected: 2
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 2
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 2
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 2
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 2
Number of persons detected: 1
Number of persons detected: 3
Number of persons detected: 0
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 0
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 2
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 2
Number of persons detected: 5
Number of persons detected: 0
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 2
Number of persons detected: 1
Number of persons detected: 2
Number of persons detected: 1
Number of persons detected: 2
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 2
Number of persons detected: 3
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 2
Number of persons detected: 1
Number of persons detected: 0
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 3
Number of persons detected: 1
Number of persons detected: 2
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 3
Number of persons detected: 2
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 0
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 2
Number of persons detected: 1
Number of persons detected: 2
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 2
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 0
Number of persons detected: 1
Number of persons detected: 2
Number of persons detected: 1
Number of persons detected: 2
Number of persons detected: 4
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 0
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 4
Number of persons detected: 3
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 2
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 2
Number of persons detected: 1
Number of persons detected: 2
Number of persons detected: 1
Number of persons detected: 2
Number of persons detected: 3
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 3
Number of persons detected: 1
Number of persons detected: 2
Number of persons detected: 1
Number of persons detected: 2
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 2
Number of persons detected: 2
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 2
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 3
Number of persons detected: 2
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 0
Number of persons detected: 2
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 2
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 2
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 0
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 2
Number of persons detected: 2
Number of persons detected: 2
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 2
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 1
Number of persons detected: 3
Number of persons detected: 1
Number of persons detected: 1
In [17]:
# A function to preprocess the images

def transform_image(image):
    img = image.convert('RGB')       # Ensure 3-channel RGB (handles grayscale/RGBA edge cases)
    img = img.resize((224, 224))     # Resize to ResNet50 input dimensions

    # Convert image to numpy array
    img = np.array(img) / 255.0      # Normalize pixel values to [0, 1]
    return img
In [18]:
def process_images_and_labels(data):
    images = []
    labels = []

    for image, label in data:
        img = transform_image(image)
        images.append(img)
        labels.append(label)

    images = np.array(images)
    labels = np.array(labels)

    return images, labels
In [19]:
train_processed_images, train_labels = process_images_and_labels(train_dataset)
validation_processed_images, validation_labels = process_images_and_labels(validation_dataset)
In [20]:
# Split training dataset into train and test — stratify preserves class ratio in both splits
X_train, X_test, y_train, y_test = train_test_split(
    train_processed_images,
    train_labels,
    test_size=0.1,
    random_state=42,
    stratify=train_labels   # stratify on labels to maintain class distribution
)
X_val, y_val = validation_processed_images, validation_labels

print(X_train.shape, y_train.shape)
print(X_val.shape, y_val.shape)
print(X_test.shape, y_test.shape)

(546, 224, 224, 3) (546,)
(85, 224, 224, 3) (85,)
(61, 224, 224, 3) (61,)

Image Classification¶

In this step, we will load a pretrained ResNet50 model that has already been trained on a large and diverse dataset, such as ImageNet.

  • ResNet50 is a deep convolutional neural network known for its residual connections, which allow it to efficiently train very deep networks without the problem of vanishing gradients.
  • This architecture is widely used for tasks such as image classification, and it can be fine-tuned to recognize specific objects, like helmets, in our case.

Utility Function¶

In [21]:
# defining a function to compute different metrics to check performance of a classification model built using statsmodels
from sklearn.metrics import confusion_matrix, f1_score, accuracy_score, recall_score, precision_score, classification_report
def model_performance_classification(model, predictors, target):
    """
    Function to compute different metrics to check classification model performance

    model: classifier
    predictors: independent variables
    target: dependent variable
    """

    # checking which probabilities are greater than threshold
    predictions = np.round(model.predict(predictors)).flatten()

    # Calculate precision and recall
    accuracy = accuracy_score(target, predictions)
    precision = precision_score(target, predictions)
    recall = recall_score(target, predictions)
    f1 = f1_score(target, predictions)  # to compute F1-score

    # creating a dataframe of metrics
    df_perf = pd.DataFrame(
        {"Accuracy": accuracy, "Recall": recall, "Precision": precision, "F1 Score": f1,},
        index=[0],
    )

    return df_perf

Model 1 ( Base model + output layer)¶

In [22]:
# Load the Pretrained ResNet model
base_model = tf.keras.applications.ResNet50(weights='imagenet', include_top=False, input_shape=(224, 224, 3))
for layer in base_model.layers[:-10]:
    layer.trainable = False  # Freeze all layers except the last 10 (allow fine-tuning of top layers)
In [23]:
model = tf.keras.models.Sequential([
    base_model,
    tf.keras.layers.GlobalAveragePooling2D(),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(1, activation='sigmoid')  # Binary classification: 1 output neuron with sigmoid
])

# Compile the model
model.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),  # Adam optimizer, lr=0.001
    loss='binary_crossentropy',
    metrics=['accuracy']
)
In [24]:
train_datagen = ImageDataGenerator()
In [25]:
#Train the model
epochs = 10
# Batch size
batch_size = 32

history_model_1 = model.fit(train_datagen.flow(X_train, y_train,
                                       shuffle=False),
                    epochs=epochs,
                    batch_size=batch_size,
                    validation_data=(X_val, y_val),
                    verbose=1)

Epoch 1/10
18/18 ━━━━━━━━━━━━━━━━━━━━ 9s 321ms/step - accuracy: 0.8736 - loss: 0.2724 - val_accuracy: 0.5294 - val_loss: 20.4329
Epoch 2/10
18/18 ━━━━━━━━━━━━━━━━━━━━ 2s 122ms/step - accuracy: 0.9396 - loss: 0.1501 - val_accuracy: 0.5294 - val_loss: 18.2573
Epoch 3/10
18/18 ━━━━━━━━━━━━━━━━━━━━ 2s 121ms/step - accuracy: 0.9579 - loss: 0.1145 - val_accuracy: 0.5294 - val_loss: 10.2510
Epoch 4/10
18/18 ━━━━━━━━━━━━━━━━━━━━ 2s 122ms/step - accuracy: 0.9780 - loss: 0.0711 - val_accuracy: 0.8235 - val_loss: 0.6067
Epoch 5/10
18/18 ━━━━━━━━━━━━━━━━━━━━ 2s 121ms/step - accuracy: 0.9780 - loss: 0.0578 - val_accuracy: 0.8824 - val_loss: 0.3422
Epoch 6/10
18/18 ━━━━━━━━━━━━━━━━━━━━ 2s 121ms/step - accuracy: 0.9744 - loss: 0.0555 - val_accuracy: 0.8471 - val_loss: 0.4131
Epoch 7/10
18/18 ━━━━━━━━━━━━━━━━━━━━ 2s 121ms/step - accuracy: 0.9853 - loss: 0.0383 - val_accuracy: 0.4941 - val_loss: 2.2070
Epoch 8/10
18/18 ━━━━━━━━━━━━━━━━━━━━ 2s 121ms/step - accuracy: 0.9927 - loss: 0.0218 - val_accuracy: 0.7294 - val_loss: 0.5760
Epoch 9/10
18/18 ━━━━━━━━━━━━━━━━━━━━ 2s 121ms/step - accuracy: 0.9927 - loss: 0.0258 - val_accuracy: 0.5647 - val_loss: 1.1340
Epoch 10/10
18/18 ━━━━━━━━━━━━━━━━━━━━ 2s 122ms/step - accuracy: 0.9945 - loss: 0.0214 - val_accuracy: 0.5412 - val_loss: 2.3414

Evaluating Model performance on training and validation data.

In [26]:
model_perf_df_val = model_performance_classification(model, X_val, y_val)
model_perf_df_val

3/3 ━━━━━━━━━━━━━━━━━━━━ 3s 780ms/step
Out[26]:
Accuracy Recall Precision F1 Score
0 0.541176 1.0 0.535714 0.697674
In [27]:
model_perf_df_train = model_performance_classification(model, X_train, y_train)
model_perf_df_train

18/18 ━━━━━━━━━━━━━━━━━━━━ 2s 114ms/step
Out[27]:
Accuracy Recall Precision F1 Score
0 0.543956 1.0 0.541436 0.702509
In [ ]:
plt.plot(history_model_1.history['accuracy'], label='Train')
plt.plot(history_model_1.history['val_accuracy'], label='Validation')
plt.title('Model 1 — Accuracy per Epoch')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(loc='upper left')
plt.tight_layout()
plt.show()
No description has been provided for this image

Observations — Model 1 (Base ResNet50 + Output Layer)¶

Training Behavior¶

  • Training accuracy reached 99.45% by epoch 10, while validation accuracy oscillated erratically between 0.49 and 0.88 with no stable upward trend.
  • Validation loss increased sharply from epoch 4 onward (0.34 → 2.34) while training loss continued decreasing — a textbook overfitting signal.
  • The gap between train accuracy (~0.99) and val accuracy (~0.54) confirms the model memorized the training set rather than learning generalizable features.

Performance Metrics¶

Set Accuracy Recall Precision F1
Validation 0.541 1.0 0.536 0.698
Train 0.544 1.0 0.541 0.702
  • Recall = 1.0 with Precision ~0.54 on both sets indicates the model is predicting class 1 (With Helmet) for every sample — it has collapsed to a trivial majority-class predictor rather than learning to discriminate.
  • An F1 of ~0.70 in this context is misleading — it is inflated by perfect Recall at the cost of near-random Precision.

Root Cause¶

  • The model architecture (ResNet50 frozen base + single Dense output layer) provides insufficient capacity for fine-tuning on a small, visually asymmetric dataset.
  • The absence of regularization (no Dropout, no data augmentation) allows the unfrozen top 10 layers to overfit rapidly.
  • The visual distribution mismatch between classes (full-body vs. close-up) likely contributes to the instability in validation accuracy across epochs.

Conclusion¶

Model 1 is not viable for production use. The results establish a performance baseline to improve upon in subsequent models through the addition of hidden layers, Dropout regularization, and data augmentation.

Image Classification Performance Improvement and Final Model Selection¶

Model 2: (Base model + FFN)¶

Lets add a Feed forward neural network with 2 hidden layers. We will be increasing the learning rate while keeping the epochs same.

In [35]:
base_model = tf.keras.applications.ResNet50(weights='imagenet', include_top=False, input_shape=(224, 224, 3))
for layer in base_model.layers[:-10]:
    layer.trainable = False  # Freeze all layers except the last 10
In [36]:
model_FFN = tf.keras.models.Sequential([
    base_model,
    tf.keras.layers.GlobalAveragePooling2D(),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(256, activation='relu'),  # Hidden layer 1: 256 neurons
    tf.keras.layers.Dense(128, activation='relu'),  # Hidden layer 2: 128 neurons
    tf.keras.layers.Dense(1, activation='sigmoid')
])


model_FFN.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),  # Adam, lr=0.001
    loss='binary_crossentropy',
    metrics=['accuracy']
)
In [37]:
epochs = 10
# Batch size
batch_size = 32

history_model_2 = model_FFN.fit(train_datagen.flow(X_train, y_train,
                                       batch_size=batch_size,
                                       seed=42,
                                       shuffle=False),
                    epochs=epochs,
                    validation_data=(X_val, y_val),
                    verbose=1)

Epoch 1/10
18/18 ━━━━━━━━━━━━━━━━━━━━ 10s 337ms/step - accuracy: 0.7967 - loss: 0.4486 - val_accuracy: 0.5294 - val_loss: 39.0943
Epoch 2/10
18/18 ━━━━━━━━━━━━━━━━━━━━ 2s 125ms/step - accuracy: 0.9286 - loss: 0.1932 - val_accuracy: 0.5294 - val_loss: 31.7864
Epoch 3/10
18/18 ━━━━━━━━━━━━━━━━━━━━ 2s 126ms/step - accuracy: 0.9652 - loss: 0.1060 - val_accuracy: 0.9529 - val_loss: 0.2909
Epoch 4/10
18/18 ━━━━━━━━━━━━━━━━━━━━ 2s 124ms/step - accuracy: 0.9762 - loss: 0.0854 - val_accuracy: 0.5529 - val_loss: 3.8086
Epoch 5/10
18/18 ━━━━━━━━━━━━━━━━━━━━ 2s 124ms/step - accuracy: 0.9707 - loss: 0.0710 - val_accuracy: 0.4941 - val_loss: 8.0008
Epoch 6/10
18/18 ━━━━━━━━━━━━━━━━━━━━ 2s 125ms/step - accuracy: 0.9670 - loss: 0.0758 - val_accuracy: 0.6941 - val_loss: 1.8924
Epoch 7/10
18/18 ━━━━━━━━━━━━━━━━━━━━ 2s 124ms/step - accuracy: 0.9872 - loss: 0.0503 - val_accuracy: 0.9529 - val_loss: 0.2125
Epoch 8/10
18/18 ━━━━━━━━━━━━━━━━━━━━ 2s 124ms/step - accuracy: 0.9762 - loss: 0.0647 - val_accuracy: 0.7647 - val_loss: 1.6047
Epoch 9/10
18/18 ━━━━━━━━━━━━━━━━━━━━ 2s 124ms/step - accuracy: 0.9780 - loss: 0.0454 - val_accuracy: 0.8824 - val_loss: 0.8015
Epoch 10/10
18/18 ━━━━━━━━━━━━━━━━━━━━ 2s 124ms/step - accuracy: 0.9744 - loss: 0.0740 - val_accuracy: 0.8118 - val_loss: 0.9594
In [38]:
model_FFN_perf_df_val = model_performance_classification(model_FFN, X_val, y_val)
model_FFN_perf_df_val

3/3 ━━━━━━━━━━━━━━━━━━━━ 3s 908ms/step
Out[38]:
Accuracy Recall Precision F1 Score
0 0.811765 1.0 0.737705 0.849057
In [39]:
model_FFN_perf_df_train = model_performance_classification(model_FFN, X_train, y_train)
model_FFN_perf_df_train

18/18 ━━━━━━━━━━━━━━━━━━━━ 2s 117ms/step
Out[39]:
Accuracy Recall Precision F1 Score
0 0.842491 1.0 0.773684 0.872404
In [40]:
plt.plot(history_model_2.history['accuracy'])
plt.plot(history_model_2.history['val_accuracy'])
plt.title('Model Accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc='upper left')
plt.show()
No description has been provided for this image

Observations — Model 2 (ResNet50 + FFN)¶

Training Behavior¶

  • Training accuracy reached 97.44% by epoch 10, consistent with Model 1.
  • Validation accuracy showed improvement over Model 1 but remains highly unstable, oscillating between 0.49 and 0.95 with no convergence — spikes at epoch 3 (0.95) and epoch 7 (0.95) followed by sharp drops indicate the model is sensitive to batch composition rather than having learned stable representations.
  • Validation loss mirrors this instability (0.29 → 39.09 → 0.21 → 0.96), confirming the model has not generalized reliably.

Performance Metrics¶

Set Accuracy Recall Precision F1
Validation 0.812 1.0 0.738 0.849
Train 0.842 1.0 0.774 0.872
  • Significant improvement over Model 1: Precision jumped from ~0.54 to ~0.74 on validation, and F1 from 0.698 to 0.849.
  • Recall = 1.0 persists on both sets — the model still predicts class 1 (With Helmet) with high frequency, though Precision improvement indicates it is beginning to discriminate rather than blindly predicting the majority class.
  • The train/val accuracy gap (~0.97 vs ~0.81) confirms overfitting is still present, reduced but not controlled.

Compared to Model 1¶

Metric Model 1 Val Model 2 Val Δ
Accuracy 0.541 0.812 +0.271
Precision 0.536 0.738 +0.202
F1 Score 0.698 0.849 +0.151

Root Cause of Remaining Instability¶

  • The addition of two hidden layers (256 → 128 → 1) increased model capacity and improved discrimination, but no regularization (no Dropout, no augmentation) was applied — the model still overfits rapidly on the small training set.
  • The erratic validation curve suggests high sensitivity to the specific samples in each validation batch, likely exacerbated by the visual distribution mismatch between classes identified in EDA.

Conclusion¶

Model 2 demonstrates that adding hidden layers provides a meaningful improvement in discriminative capacity. However, the lack of regularization results in an unstable validation curve and persistent overfitting.

Model 3 (Base model + FFN + Data Augmentation) with dropout¶

Lets add a dropout and data augmentation. We will also decrease the learning rate so that we can reach a global minimnum.

In [41]:
base_model = tf.keras.applications.ResNet50(weights='imagenet', include_top=False, input_shape=(224, 224, 3))
for layer in base_model.layers[:-10]:
    layer.trainable = False  # Freeze all layers except the last 10
In [42]:
model_FFN_DA = tf.keras.models.Sequential([
    base_model,
    tf.keras.layers.GlobalAveragePooling2D(),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(128, activation='relu'),
    tf.keras.layers.Dropout(0.5),   # Drop 50% of neurons to reduce overfitting
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dense(1, activation='sigmoid')
])


model_FFN_DA.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate=0.0001),  # Reduced lr=0.0001 for stable convergence
    loss='binary_crossentropy',
    metrics=['accuracy']
)
In [43]:
# Applying data augmentation
train_datagen = ImageDataGenerator(
    rotation_range=20,           # Rotate images up to 20 degrees
    fill_mode='nearest',
    width_shift_range=0.2,       # Shift width by 20%
    height_shift_range=0.2,      # Shift height by 20%
    shear_range=0.3,
    zoom_range=0.4
)
In [44]:
epochs = 20
# Batch size
batch_size = 32

history_model_3 = model_FFN_DA.fit(train_datagen.flow(X_train, y_train,
                                       batch_size=batch_size,
                                       seed=42,
                                       shuffle=False),
                    epochs=epochs,
                    validation_data=(X_val, y_val),
                    verbose=1)

Epoch 1/20
18/18 ━━━━━━━━━━━━━━━━━━━━ 10s 331ms/step - accuracy: 0.6813 - loss: 0.6144 - val_accuracy: 0.5294 - val_loss: 1.0292
Epoch 2/20
18/18 ━━━━━━━━━━━━━━━━━━━━ 2s 130ms/step - accuracy: 0.8462 - loss: 0.3722 - val_accuracy: 0.5294 - val_loss: 0.9875
Epoch 3/20
18/18 ━━━━━━━━━━━━━━━━━━━━ 2s 128ms/step - accuracy: 0.8608 - loss: 0.3245 - val_accuracy: 0.5294 - val_loss: 0.7618
Epoch 4/20
18/18 ━━━━━━━━━━━━━━━━━━━━ 2s 128ms/step - accuracy: 0.8773 - loss: 0.2664 - val_accuracy: 0.5294 - val_loss: 1.1056
Epoch 5/20
18/18 ━━━━━━━━━━━━━━━━━━━━ 2s 128ms/step - accuracy: 0.8974 - loss: 0.2420 - val_accuracy: 0.5294 - val_loss: 1.1781
Epoch 6/20
18/18 ━━━━━━━━━━━━━━━━━━━━ 2s 127ms/step - accuracy: 0.8993 - loss: 0.2590 - val_accuracy: 0.5294 - val_loss: 0.9329
Epoch 7/20
18/18 ━━━━━━━━━━━━━━━━━━━━ 2s 129ms/step - accuracy: 0.9121 - loss: 0.2139 - val_accuracy: 0.5294 - val_loss: 0.8157
Epoch 8/20
18/18 ━━━━━━━━━━━━━━━━━━━━ 2s 128ms/step - accuracy: 0.9176 - loss: 0.1958 - val_accuracy: 0.5294 - val_loss: 1.0841
Epoch 9/20
18/18 ━━━━━━━━━━━━━━━━━━━━ 2s 128ms/step - accuracy: 0.9231 - loss: 0.2173 - val_accuracy: 0.5294 - val_loss: 1.0222
Epoch 10/20
18/18 ━━━━━━━━━━━━━━━━━━━━ 2s 128ms/step - accuracy: 0.9212 - loss: 0.1899 - val_accuracy: 0.5294 - val_loss: 0.6155
Epoch 11/20
18/18 ━━━━━━━━━━━━━━━━━━━━ 2s 127ms/step - accuracy: 0.9011 - loss: 0.2193 - val_accuracy: 0.5647 - val_loss: 0.5605
Epoch 12/20
18/18 ━━━━━━━━━━━━━━━━━━━━ 2s 124ms/step - accuracy: 0.9359 - loss: 0.1812 - val_accuracy: 0.5294 - val_loss: 1.2235
Epoch 13/20
18/18 ━━━━━━━━━━━━━━━━━━━━ 2s 128ms/step - accuracy: 0.9048 - loss: 0.2366 - val_accuracy: 0.9059 - val_loss: 0.2587
Epoch 14/20
18/18 ━━━━━━━━━━━━━━━━━━━━ 2s 128ms/step - accuracy: 0.9121 - loss: 0.2174 - val_accuracy: 0.9647 - val_loss: 0.1836
Epoch 15/20
18/18 ━━━━━━━━━━━━━━━━━━━━ 2s 133ms/step - accuracy: 0.9304 - loss: 0.1977 - val_accuracy: 0.7647 - val_loss: 0.4899
Epoch 16/20
18/18 ━━━━━━━━━━━━━━━━━━━━ 2s 128ms/step - accuracy: 0.9249 - loss: 0.1897 - val_accuracy: 0.9765 - val_loss: 0.1829
Epoch 17/20
18/18 ━━━━━━━━━━━━━━━━━━━━ 2s 128ms/step - accuracy: 0.9414 - loss: 0.1600 - val_accuracy: 0.9412 - val_loss: 0.2016
Epoch 18/20
18/18 ━━━━━━━━━━━━━━━━━━━━ 2s 130ms/step - accuracy: 0.9359 - loss: 0.1718 - val_accuracy: 0.9765 - val_loss: 0.2135
Epoch 19/20
18/18 ━━━━━━━━━━━━━━━━━━━━ 2s 127ms/step - accuracy: 0.9396 - loss: 0.2070 - val_accuracy: 0.9412 - val_loss: 0.1785
Epoch 20/20
18/18 ━━━━━━━━━━━━━━━━━━━━ 2s 133ms/step - accuracy: 0.9322 - loss: 0.1692 - val_accuracy: 0.9529 - val_loss: 0.1595
In [45]:
model_FFN_DA_perf_df_val = model_performance_classification(model_FFN_DA, X_val, y_val)
model_FFN_DA_perf_df_val

3/3 ━━━━━━━━━━━━━━━━━━━━ 3s 877ms/step
Out[45]:
Accuracy Recall Precision F1 Score
0 0.952941 0.933333 0.976744 0.954545
In [46]:
model_FFN_DA_perf_df_train = model_performance_classification(model_FFN_DA, X_train, y_train)
model_FFN_DA_perf_df_train

18/18 ━━━━━━━━━━━━━━━━━━━━ 2s 114ms/step
Out[46]:
Accuracy Recall Precision F1 Score
0 0.912088 0.836735 1.0 0.911111
In [47]:
plt.plot(history_model_3.history['accuracy'])
plt.plot(history_model_3.history['val_accuracy'])
plt.title('Model Accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc='upper left')
plt.show()
No description has been provided for this image

Observations — Model 3 (ResNet50 + FFN + Dropout + Data Augmentation)¶

Training Behavior¶

  • The training curve shows a clear two-phase learning dynamic:
    • Epochs 1–10: validation accuracy remained flat at 0.5294 — identical to the collapsed baseline seen in Models 1 and 2. The model appeared stalled.
    • Epochs 11–20: a phase transition occurred at epoch 13 where validation accuracy jumped from 0.53 to 0.91, then stabilized between 0.94 and 0.98 for the remaining epochs.
  • This delayed convergence is consistent with the lower learning rate (0.0001) combined with data augmentation — the model explores the loss landscape more slowly but finds a more stable and generalizable minimum.
  • Training accuracy stabilized around 0.91–0.94, notably lower than Models 1 and 2 (~0.99), which is expected: Dropout and augmentation make the training task harder by design, preventing memorization.

Performance Metrics¶

Set Accuracy Recall Precision F1
Validation 0.953 0.933 0.977 0.955
Train 0.912 0.837 1.0 0.911
  • Validation metrics exceed training metrics across Accuracy, Recall, Precision, and F1 — this is the inverse of the overfitting pattern seen in Models 1 and 2, and is the expected behavior when Dropout (active only during training) and augmentation are applied correctly.
  • Recall dropped from 1.0 to 0.933 — the model is no longer predicting class 1 for everything. It now produces genuine False Negatives, which is the sign of a discriminating classifier.
  • Precision reached 0.977 on validation — when the model predicts "With Helmet", it is correct 97.7% of the time.

Compared to Previous Models¶

Metric Model 1 Val Model 2 Val Model 3 Val
Accuracy 0.541 0.812 0.953
Recall 1.0 1.0 0.933
Precision 0.536 0.738 0.977
F1 Score 0.698 0.849 0.955

Key Drivers of Improvement¶

  • Dropout (0.5) prevented co-adaptation of neurons, forcing the network to learn redundant representations robust to feature dropout.
  • Reduced learning rate (0.0001) enabled finer gradient updates, critical for fine-tuning the top ResNet50 layers without overshooting the loss minimum.
  • Data augmentation (rotation, shift, zoom, shear) diversified the training distribution, partially compensating for the visual asymmetry between classes identified in EDA.

Conclusion¶

Model 3 is the strongest candidate so far with an F1 of 0.955 on validation and a stable, converging training curve after epoch 13. The train/val metric inversion confirms effective regularization.

Model 4 (Using a different optimizer and reducing batch size)¶

We will be using the same model architecture as above, but this time we will use Stochastic Gradient Descent optimizer with a larger learning rate to compile our model. We will decrease our batch size so that we can have faster convergences and better generalization.

In [48]:
base_model = tf.keras.applications.ResNet50(weights='imagenet', include_top=False, input_shape=(224, 224, 3))
for layer in base_model.layers[:-10]:
    layer.trainable = False  # Freeze all layers except the last 10
In [49]:
model_FFN_DA_2 = tf.keras.models.Sequential([
    base_model,
    tf.keras.layers.GlobalAveragePooling2D(),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(128, activation='relu'),
    tf.keras.layers.Dropout(0.5),   # Drop 50% of neurons
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dense(1, activation='sigmoid')
])

# BUG FIX: original had misplaced closing parenthesis that excluded loss and metrics from compile()
model_FFN_DA_2.compile(
    optimizer=tf.keras.optimizers.SGD(learning_rate=0.001),  # SGD with momentum default
    loss='binary_crossentropy',
    metrics=['accuracy']
)
In [50]:
# Applying data augmentation
train_datagen = ImageDataGenerator(
    rotation_range=20,
    fill_mode='nearest',
    width_shift_range=0.2,
    height_shift_range=0.2,
    shear_range=0.3,
    zoom_range=0.4
)

epochs = 20

batch_size = 16  # Reduced from 32 — smaller batches = noisier gradients = better generalization with SGD

history_model_4 = model_FFN_DA_2.fit(train_datagen.flow(X_train, y_train,
                                       batch_size=batch_size,
                                       seed=42,
                                       shuffle=True,
                                       ),
                    epochs=epochs,
                    validation_data=(X_val, y_val),
                    verbose=1)

Epoch 1/20
35/35 ━━━━━━━━━━━━━━━━━━━━ 8s 146ms/step - accuracy: 0.5696 - loss: 0.7900 - val_accuracy: 0.6235 - val_loss: 0.6781
Epoch 2/20
35/35 ━━━━━━━━━━━━━━━━━━━━ 2s 67ms/step - accuracy: 0.6447 - loss: 0.6517 - val_accuracy: 0.7647 - val_loss: 0.6675
Epoch 3/20
35/35 ━━━━━━━━━━━━━━━━━━━━ 2s 65ms/step - accuracy: 0.6850 - loss: 0.5724 - val_accuracy: 0.8235 - val_loss: 0.6504
Epoch 4/20
35/35 ━━━━━━━━━━━━━━━━━━━━ 2s 66ms/step - accuracy: 0.7289 - loss: 0.5087 - val_accuracy: 0.5294 - val_loss: 0.6603
Epoch 5/20
35/35 ━━━━━━━━━━━━━━━━━━━━ 2s 65ms/step - accuracy: 0.7473 - loss: 0.4880 - val_accuracy: 0.5294 - val_loss: 0.6608
Epoch 6/20
35/35 ━━━━━━━━━━━━━━━━━━━━ 2s 67ms/step - accuracy: 0.7747 - loss: 0.4513 - val_accuracy: 0.5294 - val_loss: 0.6312
Epoch 7/20
35/35 ━━━━━━━━━━━━━━━━━━━━ 2s 66ms/step - accuracy: 0.7930 - loss: 0.4435 - val_accuracy: 0.5294 - val_loss: 0.6112
Epoch 8/20
35/35 ━━━━━━━━━━━━━━━━━━━━ 2s 66ms/step - accuracy: 0.7912 - loss: 0.4355 - val_accuracy: 0.5294 - val_loss: 0.5256
Epoch 9/20
35/35 ━━━━━━━━━━━━━━━━━━━━ 2s 65ms/step - accuracy: 0.7949 - loss: 0.4321 - val_accuracy: 0.5294 - val_loss: 0.6416
Epoch 10/20
35/35 ━━━━━━━━━━━━━━━━━━━━ 2s 65ms/step - accuracy: 0.8095 - loss: 0.4195 - val_accuracy: 0.5294 - val_loss: 0.5195
Epoch 11/20
35/35 ━━━━━━━━━━━━━━━━━━━━ 2s 65ms/step - accuracy: 0.8004 - loss: 0.4299 - val_accuracy: 0.9176 - val_loss: 0.3656
Epoch 12/20
35/35 ━━━━━━━━━━━━━━━━━━━━ 2s 66ms/step - accuracy: 0.8242 - loss: 0.3855 - val_accuracy: 0.9294 - val_loss: 0.3371
Epoch 13/20
35/35 ━━━━━━━━━━━━━━━━━━━━ 2s 66ms/step - accuracy: 0.8077 - loss: 0.3959 - val_accuracy: 0.9647 - val_loss: 0.2904
Epoch 14/20
35/35 ━━━━━━━━━━━━━━━━━━━━ 2s 66ms/step - accuracy: 0.8059 - loss: 0.4055 - val_accuracy: 0.9294 - val_loss: 0.2720
Epoch 15/20
35/35 ━━━━━━━━━━━━━━━━━━━━ 2s 66ms/step - accuracy: 0.8553 - loss: 0.3597 - val_accuracy: 0.9529 - val_loss: 0.2784
Epoch 16/20
35/35 ━━━━━━━━━━━━━━━━━━━━ 2s 65ms/step - accuracy: 0.8498 - loss: 0.3582 - val_accuracy: 0.9647 - val_loss: 0.2261
Epoch 17/20
35/35 ━━━━━━━━━━━━━━━━━━━━ 2s 66ms/step - accuracy: 0.8590 - loss: 0.3553 - val_accuracy: 0.9647 - val_loss: 0.2203
Epoch 18/20
35/35 ━━━━━━━━━━━━━━━━━━━━ 2s 66ms/step - accuracy: 0.8443 - loss: 0.3458 - val_accuracy: 0.9647 - val_loss: 0.1961
Epoch 19/20
35/35 ━━━━━━━━━━━━━━━━━━━━ 2s 66ms/step - accuracy: 0.8352 - loss: 0.3554 - val_accuracy: 0.9529 - val_loss: 0.1881
Epoch 20/20
35/35 ━━━━━━━━━━━━━━━━━━━━ 2s 66ms/step - accuracy: 0.8681 - loss: 0.3147 - val_accuracy: 0.9059 - val_loss: 0.2862
In [51]:
model_FFN_DA_2_perf_df_val = model_performance_classification(model_FFN_DA_2, X_val, y_val)
model_FFN_DA_2_perf_df_val

3/3 ━━━━━━━━━━━━━━━━━━━━ 4s 1s/step
Out[51]:
Accuracy Recall Precision F1 Score
0 0.905882 1.0 0.849057 0.918367
In [52]:
model_FFN_DA_2_perf_df_train = model_performance_classification(model_FFN_DA_2, X_train, y_train)
model_FFN_DA_2_perf_df_train

18/18 ━━━━━━━━━━━━━━━━━━━━ 2s 118ms/step
Out[52]:
Accuracy Recall Precision F1 Score
0 0.950549 0.996599 0.918495 0.955954
In [53]:
plt.plot(history_model_4.history['accuracy'])
plt.plot(history_model_4.history['val_accuracy'])
plt.title('Model Accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc='upper left')
plt.show()
No description has been provided for this image

Observations — Model 4 (ResNet50 + FFN + Dropout + Data Augmentation + SGD)¶

Training Behavior¶

  • Similar two-phase dynamic observed in Model 3:
    • Epochs 1–10: validation accuracy stalled at 0.5294 (epochs 4–10), with brief improvement at epochs 2–3 (0.76–0.82) followed by collapse — SGD with small batch size (16) produces noisier gradient updates that initially destabilize the validation signal.
    • Epochs 11–20: phase transition at epoch 11 (0.52 → 0.92), stabilizing between 0.91 and 0.97 through epoch 18, then declining slightly to 0.91 at epoch 20 — an early sign of overfitting in the final epochs.
  • Training accuracy reached only 0.87 by epoch 20, consistently lower than Model 3 (~0.93). The smaller batch size (16) introduces higher gradient variance, slowing convergence and explaining the wider train/val gap visible in the curve.
  • Unlike Model 3, validation accuracy surpasses training accuracy from epoch 11 onward — same Dropout inversion effect, but more pronounced due to noisier SGD updates.

Performance Metrics¶

Set Accuracy Recall Precision F1
Validation 0.906 1.0 0.849 0.918
Train 0.951 0.997 0.918 0.956
  • Recall = 1.0 on validation reappears — unlike Model 3 (Recall 0.933), Model 4 is again predicting class 1 more aggressively, suggesting SGD with this configuration has not fully resolved the class discrimination issue.
  • Precision on validation (0.849) is lower than Model 3 (0.977), confirming more False Positives — the model is less conservative when predicting "With Helmet".

Full Model Comparison¶

Metric Model 1 Model 2 Model 3 Model 4
Val Accuracy 0.541 0.812 0.953 0.906
Val Recall 1.0 1.0 0.933 1.0
Val Precision 0.536 0.738 0.977 0.849
Val F1 Score 0.698 0.849 0.955 0.918
Train/Val gap Large Moderate Inverted Inverted
Convergence Unstable Unstable Stable (ep.13+) Stable (ep.11+)

Why Model 3 Outperforms Model 4¶

  • Adam vs SGD: Adam's adaptive learning rates per parameter provide smoother convergence on small datasets. SGD with a fixed learning rate (0.001) requires more epochs and careful tuning to reach the same minimum.
  • Batch size 16 vs 32: smaller batches increase gradient noise, which can help escape local minima but also slows stable convergence and inflates validation variance in the late epochs.
  • Model 4's declining val_accuracy in the final epochs (0.9647 → 0.9059) suggests it would benefit from early stopping or learning rate decay, neither of which was applied.

Conclusion¶

Model 3 is selected as the best model, achieving the highest validation F1 (0.955), highest Precision (0.977), and the most stable convergence curve. Model 4 is a viable alternative but underperforms on Precision and shows marginal deterioration in final epochs. For a safety-critical application where False Positives (missed non-compliant workers) carry operational risk, Model 3's superior Precision is the decisive factor.

Model Performance Comparison and Final Model Selection¶

In [54]:
models_train_comp_df = pd.concat(
    [
        model_perf_df_train.T,
        model_FFN_perf_df_train.T,
        model_FFN_DA_perf_df_train.T,
        model_FFN_DA_2_perf_df_train.T,


    ],
    axis=1,
)
models_train_comp_df.columns = [
    "Model 1: Simple",
    "Model 2: FFN",
    "Model 3: DA",
    'Model 4: SGD'
]

models_val_comp_df = pd.concat(
    [
        model_perf_df_val.T,
        model_FFN_perf_df_val.T,
        model_FFN_DA_perf_df_val.T,
        model_FFN_DA_2_perf_df_val.T

    ],
    axis=1,
)
models_val_comp_df.columns = [
    "Model 1: Simple",
    "Model 2: FFN",
    "Model 3: DA",
    'Model 4: SGD'
]
In [55]:
print('PERFORMANCE OF MODELS ON THE TRAINING SET')
models_train_comp_df

PERFORMANCE OF MODELS ON THE TRAINING SET
Out[55]:
Model 1: Simple Model 2: FFN Model 3: DA Model 4: SGD
Accuracy 0.543956 0.842491 0.912088 0.950549
Recall 1.000000 1.000000 0.836735 0.996599
Precision 0.541436 0.773684 1.000000 0.918495
F1 Score 0.702509 0.872404 0.911111 0.955954
In [56]:
print('PERFORMANCE OF MODELS ON VALIDATION SET')
models_val_comp_df

PERFORMANCE OF MODELS ON VALIDATION SET
Out[56]:
Model 1: Simple Model 2: FFN Model 3: DA Model 4: SGD
Accuracy 0.541176 0.811765 0.952941 0.905882
Recall 1.000000 1.000000 0.933333 1.000000
Precision 0.535714 0.737705 0.976744 0.849057
F1 Score 0.697674 0.849057 0.954545 0.918367

Observations — Model Performance Comparison¶

Validation Set Summary¶

Metric Model 1: Simple Model 2: FFN Model 3: DA Model 4: SGD
Accuracy 0.541 0.812 0.953 0.906
Recall 1.000 1.000 0.933 1.000
Precision 0.536 0.738 0.977 0.849
F1 Score 0.698 0.849 0.955 0.918

Progressive Improvement Trend¶

  • Each architectural addition produced measurable gains on the validation set:
    • Model 1 → Model 2 (+FFN layers): F1 +0.151, Precision +0.202 — hidden layers provided the discriminative capacity missing in the baseline.
    • Model 2 → Model 3 (+Dropout + Augmentation + lower LR): F1 +0.106, Precision +0.239 — regularization was the single most impactful improvement, resolving the majority-class collapse seen in Models 1 and 2.
    • Model 3 → Model 4 (+SGD + smaller batch): F1 -0.037, Precision -0.128 — the optimizer change did not improve generalization; Adam outperformed SGD on this dataset size and configuration.

Training vs. Validation Consistency¶

  • Models 1 and 2: training metrics significantly exceed validation metrics — classic overfitting. Neither model learned generalizable features.
  • Models 3 and 4: validation metrics exceed or match training metrics — Dropout and augmentation successfully inverted the overfitting dynamic.
  • Model 3 is the only model where both train Recall (0.837) and val Recall (0.933) are below 1.0, confirming it is the only model producing genuine discriminative predictions rather than defaulting to a majority-class strategy.

Critical Metric — Precision in a Safety Context¶

  • For helmet compliance detection, a False Positive means a non-compliant worker is classified as wearing a helmet — a direct safety risk.
  • Model 3 achieves Precision = 0.977 on validation, the highest across all models, making it the most reliable for safety-critical deployment.
  • Models 1, 2, and 4 all show Recall = 1.0 on validation, indicating they still predict class 1 too aggressively — inflating Recall at the cost of Precision.

Selected Model¶

Model 3 (ResNet50 + FFN + Dropout + Data Augmentation) is selected as the final model based on highest validation F1 (0.955), highest Precision (0.977), lowest train/val metric gap, and most stable convergence curve across 20 epochs.

Model Performance Check on Test Set¶

In [57]:
best_model = model_FFN_DA  # Model 3 with FFN + Data Augmentation showed best validation performance based on F1/Accuracy trade-off.
In [58]:
predictions = np.round(best_model.predict(X_test))
print(classification_report(y_test, predictions))

2/2 ━━━━━━━━━━━━━━━━━━━━ 1s 828ms/step
              precision    recall  f1-score   support

           0       0.85      1.00      0.92        28
           1       1.00      0.85      0.92        33

    accuracy                           0.92        61
   macro avg       0.92      0.92      0.92        61
weighted avg       0.93      0.92      0.92        61

Lets plot the confusion matrix to gain a deeper insight.

In [59]:
#Compute Confusion Matrix
cm = confusion_matrix(y_test, predictions)

# Confusion Matrix Labels
labels = ['Class 0 (No Helmet)', 'Class 1 (Helmet)']

# Check the content of the confusion matrix to ensure it's correct
print("Confusion Matrix:")

# Plot the confusion matrix
plt.figure(figsize=(8, 6))
sns.heatmap(cm, annot=True, fmt='d', cmap='Blues',
            xticklabels=labels, yticklabels=labels,
            cbar=False, linewidths=1, linecolor='black',
            square=True)

# Add Titles and Axis Labels
plt.title('Confusion Matrix', fontsize=16)
plt.xlabel('Predicted Labels', fontsize=12)
plt.ylabel('True Labels', fontsize=12)

# # Ensure the plot window is shown
plt.show()

Confusion Matrix:
No description has been provided for this image

Lets plot some images of the test dataset along with the predictions.

In [60]:
num_images = 5

# Select 5 random images from the test set
selected_images = sample = np.array(random.sample(list(X_test), num_images))

class_dict = {0: 'No Helmet Detected', 1: 'Helmet Detected'}

# Plot the images
plt.figure(figsize=(15, 5))
for i, image_file in enumerate(selected_images):
    confidence = best_model.predict(np.expand_dims(image_file, axis=0), verbose=2)
    prediction = int(np.round(confidence))
    if prediction == 0:
        confidence = (1 - confidence) * 100
    else:
        confidence = confidence[0][0] * 100
    detected_class = class_dict[prediction]
    plt.subplot(1, num_images, i + 1)  # Arrange images in 1 row
    plt.imshow(image_file)
    plt.axis('off')  # Hide axes
    plt.title(f' image #{i + 1} \nPredicted Class:{detected_class}\nConfidence:{np.round(confidence,2)}%', fontsize=10)
plt.tight_layout()
plt.show()

1/1 - 1s - 579ms/step
1/1 - 0s - 34ms/step
1/1 - 0s - 33ms/step
1/1 - 0s - 43ms/step
1/1 - 0s - 27ms/step
No description has been provided for this image

Observations — Model Performance on Test Set¶

Classification Report¶

Class Precision Recall F1 Support
0 — No Helmet 0.85 1.00 0.92 28
1 — Helmet 1.00 0.85 0.92 33
Overall Accuracy 0.92 61

Confusion Matrix Breakdown¶

Predicted No Helmet Predicted Helmet
Actual No Helmet 28 (TN) 0 (FP)
Actual Helmet 5 (FN) 28 (TP)

Key Findings¶

  • Zero False Positives (FP = 0): the model never classified a non-compliant worker as wearing a helmet. In a safety-critical context this is the most important result — no undetected compliance violations on the test set.
  • 5 False Negatives (FN = 5): 5 workers wearing helmets were classified as non-compliant. These are operationally acceptable errors — they trigger unnecessary alerts but do not represent a safety risk.
  • Precision = 1.0 on class 1 (Helmet): every positive prediction was correct, confirming the model's conservative and reliable behavior when flagging compliance.
  • Recall = 1.0 on class 0 (No Helmet): all 28 non-compliant workers were correctly identified — no safety violations went undetected.

Generalization Assessment¶

  • Test accuracy (0.92) is consistent with validation accuracy (0.953), confirming the model generalizes well to unseen data without significant degradation.
  • The macro average F1 (0.92) equals the weighted average F1 (0.92), indicating balanced performance across both classes despite the visual distribution asymmetry identified in EDA.

Sample Predictions¶

  • The model assigns high confidence to correct predictions: 99.72%, 99.7%, and 99.99% on helmet-wearing workers.
  • "No Helmet Detected" predictions at 83.73% and 99.45% are also correct — close-up face images (WithoutHelmet class) are classified reliably at test time.
  • Note: the confidence display format shows [[ 83.73 ]]% for class 0 predictions due to a numpy array scalar conversion — a cosmetic issue from the DeprecationWarning visible in the output. It does not affect model correctness.

Conclusion¶

The final model (Model 3) achieves 92% test accuracy with zero False Positives, making it suitable for production deployment as a first-pass automated compliance screening tool. The 5 False Negatives represent a 15.2% miss rate on the helmet class — acceptable for a system designed to assist human safety officers rather than replace them entirely.

Actionable Insights and Recommendations¶

Insights¶

  • Transfer learning with ResNet50 is effective for small safety datasets: even with only 640 images, fine-tuning the top 10 layers of a pretrained ImageNet model produced a 92% accurate classifier — demonstrating that domain-specific CV systems can be built without large proprietary datasets.

  • Regularization was the decisive factor in model quality: Models 1 and 2 collapsed to majority-class predictors (Recall = 1.0, Precision ~0.54) despite high training accuracy. The addition of Dropout (0.5) and data augmentation in Model 3 was the single intervention that unlocked genuine discrimination between classes (F1 0.955 on validation).

  • Adam outperforms SGD on small, imbalanced visual datasets: Model 4 demonstrated that SGD with batch size 16 introduces gradient noise that destabilizes early training and reduces final Precision (0.849 vs 0.977), without compensating gains in other metrics.

  • The two-stage pipeline (YOLO → ResNet50) introduces a critical asymmetry: YOLO is applied only to the WithHelmet class because WithoutHelmet images are close-up face shots that fall below YOLO's person detection confidence threshold. This architectural decision preserves all training samples but introduces a visual distribution mismatch that the model must compensate for through augmentation.

  • Zero False Positives on the test set is the most operationally significant result: the system never cleared a non-compliant worker as compliant — the exact failure mode that would defeat the purpose of automated safety monitoring.

  • The 5 False Negatives (15.2% miss rate on helmet class) represent the primary production risk: workers wearing helmets may occasionally trigger unnecessary alerts, adding noise to safety officer workflows but not creating undetected hazards.

Recommendations¶

  • Threshold tuning for production deployment: lower the classification threshold below 0.5 to further reduce False Negatives at the cost of acceptable increase in False Positives. For safety-critical applications, maximizing Recall on class 0 (No Helmet) is the priority — the current 100% Recall on this class should be preserved.

  • Fix the confidence display bug before deployment: the DeprecationWarning on numpy scalar conversion produces malformed confidence strings ([[ 83.73 ]]%). Replace int(np.round(confidence)) with int(np.round(float(confidence))) to resolve it.

  • Collect visually consistent training data: the primary dataset limitation is the framing mismatch between classes. Future data collection for WorkersWithoutHelmet should target full-body or mid-body shots in construction site contexts, matching the visual distribution of WorkersWithHelmet images.

  • Replace YOLOv5s with YOLOv8n for production inference: YOLOv8n offers better detection performance on partially occluded and distant persons — directly addressing the low confidence scores (0.31–0.35) observed in the object detection phase.

  • Add early stopping and model checkpointing: both Models 3 and 4 showed slight degradation in final epochs. Implementing tf.keras.callbacks.EarlyStopping with monitor='val_loss', patience=5 and ModelCheckpoint saving the best weights would capture peak performance automatically.

  • Wrap the pipeline for real-time deployment: package the full inference chain (frame extraction → YOLO person detection → crop → ResNet50 classification → alert trigger) as a single callable service. Flag predictions with confidence below 0.75 for human review rather than automated action.

  • Establish a monitoring baseline: log prediction confidence distributions and class ratios per site in production. A drift in the ratio of No Helmet detections over time is a leading indicator of either model degradation or genuine compliance deterioration — both requiring intervention.

Executive Summary¶

SafeGuard Corp commissioned an automated helmet compliance detection system to reduce manual monitoring errors and improve worker safety at hazardous industrial sites. This project delivered a fully functional two-stage computer vision pipeline validated on a 640-image dataset, achieving production-viable performance metrics.

Pipeline Architecture¶

The system operates in two sequential stages:

  1. Object Detection (YOLOv5s): detects and crops individual workers from site images. Applied exclusively to the WithHelmet class — close-up face images in the WithoutHelmet class bypass YOLO by design, as full-body detection is not applicable to that visual distribution.

  2. Image Classification (ResNet50 + FFN): classifies each cropped image as compliant (helmet present) or non-compliant (no helmet), using a pretrained ImageNet backbone fine-tuned on the target dataset.

Model Selection¶

Four model configurations were evaluated. Model 3 was selected as the final model:

Model Architecture Val F1 Val Precision Val Accuracy
Model 1 ResNet50 + Output layer 0.698 0.536 0.541
Model 2 ResNet50 + FFN 0.849 0.738 0.812
Model 3 ResNet50 + FFN + Dropout + Augmentation 0.955 0.977 0.953
Model 4 ResNet50 + FFN + Dropout + Augmentation + SGD 0.918 0.849 0.906

Final Model — Test Set Performance¶

Metric Result
Overall Accuracy 92%
False Positives (non-compliant cleared as compliant) 0
False Negatives (compliant flagged as non-compliant) 5
Precision — No Helmet class 0.85
Recall — No Helmet class 1.00
Precision — Helmet class 1.00
Recall — Helmet class 0.85

The most operationally critical result is zero False Positives: the system did not clear a single non-compliant worker as compliant across the entire test set.

Key Technical Findings¶

  • Dropout (0.5) and data augmentation were the decisive interventions that resolved majority-class collapse in early models, enabling genuine binary discrimination.
  • Adam optimizer consistently outperformed SGD on this dataset size and configuration.
  • The visual distribution mismatch between classes (full-body vs. close-up) is the primary dataset limitation and the most significant risk factor for production generalization.

Production Readiness Assessment¶

Dimension Status
Binary classification accuracy ✅ Production-viable (92% test)
Safety-critical metric (FP = 0) ✅ Verified on test set
Pipeline completeness ✅ End-to-end YOLO → classifier
Dataset size ⚠️ Small (640 images) — monitor for drift
Visual class consistency ⚠️ Mismatch between classes — requires data collection
Real-time inference packaging ⚠️ Pending — requires deployment wrapper

Recommended Next Steps¶

  1. Collect WithoutHelmet images in construction site context to resolve the visual distribution mismatch.
  2. Tune the classification threshold below 0.5 to further eliminate False Negatives.
  3. Replace YOLOv5s with YOLOv8n for improved detection on occluded and distant workers.
  4. Package the pipeline as a real-time inference service with confidence-based human review routing for low-certainty predictions.
  5. Implement production monitoring for prediction drift as a leading indicator of model degradation or genuine compliance shifts across sites.

Power Ahead!