Case Study 9 — Federated Edge AI (LoRA + Differential Privacy)

Student: José Antonio González Villalón - josegzzv@msn.com Deliverable: Gonzalez_Jose_FederatedAI_Lab.ipynb Structure: follows the Case Study 1 → 8 template (Part 0 → Part 5 + Reflection).

This notebook implements the healthcare monitoring network described in the assignment specification: 3 simulated edge clients, a federated server, a privacy engine and LoRA parameter-efficient fine-tuning on a DistilBERT backbone.

Layer	Implementation
Model	distilbert-base-uncased (pretrained), 3-class head (Normal / Warning / Critical)
PEFT	PEFT LoraConfig, rank r, adapters on q_lin and v_lin only
Privacy	Opacus DP-SGD (per-sample clipping + Gaussian noise) with an RDP accountant per client
Federation	FedAvg (McMahan et al., 2017), sample-weighted, adapters-only payload
Data	Non-IID vital-sign shards rendered to clinical text; raw data never leaves the client

How to run: execute top to bottom. Part 3 runs the baseline, Part 4 answers Q1–Q10, Part 5 runs the five parameter-exploration experiments (Q11–Q15) and writes results/*.json, results/summary.csv and results/deltas.csv.

Headline result of this run. The system learns and differential privacy consumes the result. With DP disabled the identical pipeline reaches 0.6333; under the prescribed (ε = 10, δ = 1e-3) budget it finishes at 0.3667 — a utility cost of 0.2667. Tightening the budget tenfold to ε = 1 changes accuracy by one test sample. The cost is incurred by applying DP-SGD at this data scale at all, not by the choice of budget.

# ---- Part 0: Setup & Config (Ungraded) ----
# Install dependencies once in your environment before running:
# %pip install torch transformers peft opacus pandas numpy matplotlib plotly scikit-learn tqdm ipywidgets

import os, gc, sys, math, time, json, copy, random, resource
import numpy as np
import pandas as pd
import torch

SEED = 42
random.seed(SEED); np.random.seed(SEED); torch.manual_seed(SEED)
torch.set_num_threads(2)          # keep CPU timings comparable across runs

# ---- Environment stamp (reproducibility): the versions behind every number below ----
import platform, importlib
_env = {"python": platform.python_version(), "os": platform.platform()}
for _m in ("torch", "transformers", "peft", "opacus", "numpy", "pandas"):
    try:
        _env[_m] = importlib.import_module(_m).__version__
    except Exception as _e:
        _env[_m] = f"n/a ({_e.__class__.__name__})"
print("Environment:", " | ".join(f"{_k}={_v}" for _k, _v in _env.items()))
print("Threads:", torch.get_num_threads(), "| device: cpu | secure_rng: off")

# ---- Optimized Configuration (V1.2.0) ----
# Federated Learning
NUM_CLIENTS = 3
ROUNDS = 5
LOCAL_EPOCHS = 1
BATCH_SIZE = 16
LEARNING_RATE = 1e-3
CLIENT_SELECTION = 1.0  # all clients participate

# LoRA
LORA_R = 4
LORA_ALPHA = 8
LORA_DROPOUT = 0.1
TARGET_MODULES = ["q_lin", "v_lin"]

# Differential Privacy
EPSILON = 10.0
DELTA = 1e-3
MAX_GRAD_NORM = 2.0

# Model / Data
MODEL_NAME = "distilbert-base-uncased"
NUM_CLASSES = 3          # Normal, Warning, Critical
MAX_LENGTH = 64
SAMPLES_PER_CLIENT = 50
TRAIN_SPLIT = 0.8

# Single dict used by every helper below, so an experiment is "one variable changed".
BASE_CFG = dict(
    NUM_CLIENTS=NUM_CLIENTS, ROUNDS=ROUNDS, LOCAL_EPOCHS=LOCAL_EPOCHS, BATCH_SIZE=BATCH_SIZE,
    LEARNING_RATE=LEARNING_RATE, CLIENT_SELECTION=CLIENT_SELECTION,
    LORA_R=LORA_R, LORA_ALPHA=LORA_ALPHA, LORA_DROPOUT=LORA_DROPOUT,
    TARGET_MODULES=TARGET_MODULES,
    EPSILON=EPSILON, DELTA=DELTA, MAX_GRAD_NORM=MAX_GRAD_NORM, DP_ENABLED=True,
    MODEL_NAME=MODEL_NAME, NUM_CLASSES=NUM_CLASSES, MAX_LENGTH=MAX_LENGTH,
    SAMPLES_PER_CLIENT=SAMPLES_PER_CLIENT, TRAIN_SPLIT=TRAIN_SPLIT,
)

RESULTS_DIR = "results"; os.makedirs(RESULTS_DIR, exist_ok=True)
RESULTS = {}   # tag -> measured result dict, consumed by the analysis cells

print("Seed:", SEED)
print("Clients:", NUM_CLIENTS, "| Rounds:", ROUNDS, "| Target (eps, delta):", (EPSILON, DELTA))

W0901 14:33:36.868000 30483 site-packages/torch/distributed/elastic/multiprocessing/redirects.py:29] NOTE: Redirects are currently not supported in Windows or MacOs.

Environment: python=3.11.13 | os=macOS-26.6.2-arm64-arm-64bit | torch=2.11.0 | transformers=5.16.1 | peft=0.20.0 | opacus=1.6.0 | numpy=1.26.4 | pandas=2.3.2
Threads: 2 | device: cpu | secure_rng: off
Seed: 42
Clients: 3 | Rounds: 5 | Target (eps, delta): (10.0, 0.001)

Part 1 — Data Simulation & Client Shards

Three edge devices, three different care settings, therefore three different label priors — the label-skew form of non-IID data, which is what actually occurs when each hospital ward contributes its own patients.

Client	Setting	Prior (Normal / Warning / Critical)
0	General ward	0.60 / 0.30 / 0.10
1	Step-down unit	0.34 / 0.33 / 0.33
2	ICU	0.15 / 0.30 / 0.55

Vital signs are drawn from label-conditioned Gaussians with deliberate overlap (a heart rate of 105 bpm is genuinely ambiguous between Warning and Critical), then rendered into a short clinical sentence so a language backbone can be used. Tokenization and the 80/20 split happen on the client; only adapter tensors are ever transmitted.

# ---- Part 1: Data Simulation & Client Shards (Q1 baseline) ----
# Each edge client is a wearable/bedside monitor in a different care setting, so the
# label distribution is non-IID by construction (label-skew, the realistic medical case).
# Raw readings are rendered to clinical text locally; only adapter weights ever leave
# the device, which is what satisfies the "no raw data leaves the client" requirement.

from torch.utils.data import TensorDataset, DataLoader

LABELS = ["Normal", "Warning", "Critical"]

# Class priors per client: 0 = general ward, 1 = step-down unit, 2 = ICU.
CLIENT_PRIORS = [
    [0.60, 0.30, 0.10],
    [0.34, 0.33, 0.33],
    [0.15, 0.30, 0.55],
]

# Label-conditioned vital-sign distributions: (mean, std) per class.
VITALS = {
    0: dict(hr=(74, 7),   sbp=(118, 8),  dbp=(76, 5),   temp=(36.7, 0.25), spo2=(98, 1.0)),
    1: dict(hr=(101, 9),  sbp=(142, 10), dbp=(89, 6),   temp=(37.9, 0.35), spo2=(94, 1.5)),
    2: dict(hr=(129, 12), sbp=(163, 13), dbp=(101, 8),  temp=(39.1, 0.5),  spo2=(88, 2.5)),
}


def render_vitals(hr, sbp, dbp, temp, spo2):
    """Serialize one reading as the clinical text the language model consumes."""
    return (f"patient vitals heart rate {hr:.0f} bpm blood pressure {sbp:.0f} over {dbp:.0f} mmhg "
            f"temperature {temp:.1f} celsius oxygen saturation {spo2:.0f} percent")


def make_client_data(n_samples, priors, rng):
    """Draw a label-imbalanced shard for one client and render it to text."""
    texts, labels = [], []
    for _ in range(n_samples):
        y = int(rng.choice(len(LABELS), p=priors))
        v = VITALS[y]
        hr, sbp, dbp = rng.normal(*v["hr"]), rng.normal(*v["sbp"]), rng.normal(*v["dbp"])
        temp = rng.normal(*v["temp"])
        spo2 = np.clip(rng.normal(*v["spo2"]), 70, 100)
        texts.append(render_vitals(hr, sbp, dbp, temp, spo2))
        labels.append(y)
    return texts, np.array(labels, dtype=np.int64)


def build_clients(cfg, tokenizer):
    """Tokenize each shard and split it 80/20 locally; the split never leaves the device."""
    rng = np.random.default_rng(SEED)
    clients = []
    for cid in range(cfg["NUM_CLIENTS"]):
        priors = CLIENT_PRIORS[cid % len(CLIENT_PRIORS)]
        texts, y = make_client_data(cfg["SAMPLES_PER_CLIENT"], priors, rng)
        enc = tokenizer(texts, padding="max_length", truncation=True,
                        max_length=cfg["MAX_LENGTH"], return_tensors="pt")
        n_tr = int(cfg["SAMPLES_PER_CLIENT"] * cfg["TRAIN_SPLIT"])
        idx = rng.permutation(cfg["SAMPLES_PER_CLIENT"])
        tr, te = idx[:n_tr], idx[n_tr:]
        y_t = torch.tensor(y)
        clients.append(dict(
            id=cid, priors=priors,
            train=TensorDataset(enc["input_ids"][tr], enc["attention_mask"][tr], y_t[tr]),
            test=TensorDataset(enc["input_ids"][te], enc["attention_mask"][te], y_t[te]),
            label_counts=np.bincount(y[tr], minlength=cfg["NUM_CLASSES"]).tolist(),
        ))
    return clients

Part 2 — DistilBERT + LoRA + Differential Privacy

Three design decisions drive every number in this study:

1. Frozen backbone, rank-r adapters. LoRA replaces the update ΔW of a d×d projection with B·A, where A ∈ R^{r×d} and B ∈ R^{d×r}. For DistilBERT (d = 768, 6 layers, 2 adapted projections) that is 2·r·d·layers·projections parameters — 73,728 at r = 4 — instead of d² · layers · projections ≈ 7.08 M.
2. The classification head must also be trained, because a 3-class head does not exist in the pretrained checkpoint. It is declared in modules_to_save, so it is trainable and part of the federated payload. Ignoring it would understate the communication cost by roughly 9×, so it is counted explicitly throughout.
3. DP-SGD, not output perturbation. Opacus clips each per-sample gradient to MAX_GRAD_NORM (bounding one patient's influence — the sensitivity) and adds Gaussian noise N(0, σ²C²) to the summed gradient. σ is calibrated once against the total number of steps across all rounds, so the cumulative budget lands on the target ε rather than spending ε per round.

# ---- Part 2: DistilBERT + LoRA + Differential Privacy (Q2-Q7 machinery) ----
# Backbone is frozen; only rank-r LoRA adapters on the attention projections
# (q_lin / v_lin) and the classification head are trainable and communicated.

from transformers import (AutoTokenizer, AutoModelForSequenceClassification,
                          DistilBertConfig, DistilBertTokenizerFast)
from peft import LoraConfig, get_peft_model, TaskType
from opacus import PrivacyEngine
from opacus.accountants import RDPAccountant
from opacus.accountants.utils import get_noise_multiplier
from opacus.validators import ModuleValidator


def load_backbone(cfg):
    """Load the pretrained checkpoint. If the model hub is unreachable, fall back to a
    config-only initialization: identical architecture and parameter counts, random
    weights (all efficiency/privacy metrics are checkpoint-independent; accuracy is not)."""
    try:
        model = AutoModelForSequenceClassification.from_pretrained(
            cfg["MODEL_NAME"], num_labels=cfg["NUM_CLASSES"])
        tokenizer = AutoTokenizer.from_pretrained(cfg["MODEL_NAME"])
        return model, tokenizer, "pretrained"
    except Exception as exc:
        print("Hub unavailable, using config-init backbone:", type(exc).__name__)
        model = AutoModelForSequenceClassification.from_config(
            DistilBertConfig(num_labels=cfg["NUM_CLASSES"]))
        tokenizer = DistilBertTokenizerFast(vocab_file="vocab.txt", do_lower_case=True)
        return model, tokenizer, "config-init"


def build_lora_model(cfg, backbone):
    """Attach LoRA adapters; PEFT freezes everything else automatically."""
    lcfg = LoraConfig(
        task_type=TaskType.SEQ_CLS,
        r=cfg["LORA_R"], lora_alpha=cfg["LORA_ALPHA"], lora_dropout=cfg["LORA_DROPOUT"],
        target_modules=cfg["TARGET_MODULES"], bias="none",
        modules_to_save=["pre_classifier", "classifier"],  # new head must also be trained
    )
    return get_peft_model(copy.deepcopy(backbone), lcfg)


def param_report(model, backbone):
    """Q2 evidence: trainable vs full-model parameters.
    base_total = parameters a full fine-tune would have to ship every round."""
    total = sum(p.numel() for p in model.parameters())
    trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
    lora_only = sum(p.numel() for n, p in model.named_parameters()
                    if p.requires_grad and "lora_" in n)
    base_total = sum(p.numel() for p in backbone.parameters())
    return dict(total=total, base_total=base_total, trainable=trainable,
                lora_only=lora_only, head_only=trainable - lora_only,
                reduction_pct=100.0 * (1 - trainable / base_total),
                lora_only_reduction_pct=100.0 * (1 - lora_only / base_total))


def trainable_state(model):
    """The federated payload: only tensors that require grad are exchanged."""
    return {n: p.detach().clone() for n, p in model.named_parameters() if p.requires_grad}


def load_trainable_state(model, state):
    """Server -> client broadcast of the aggregated adapter."""
    with torch.no_grad():
        for n, p in model.named_parameters():
            if n in state:
                p.copy_(state[n])

W0831 00:30:09.994000 20600 site-packages/torch/distributed/elastic/multiprocessing/redirects.py:29] NOTE: Redirects are currently not supported in Windows or MacOs.

Part 3 — Federated Server Loop

Protocol executed each round: broadcast the global adapter → each client runs LOCAL_EPOCHS of local DP-SGD → clients upload adapters → server performs sample-weighted FedAvg → evaluate the global model on every client's held-out shard. The server never sees a gradient that has not already been clipped and noised on the device, so the privacy guarantee holds against an honest-but-curious server.

# ---- Part 3: Federated Server Loop with DP-SGD clients (Q1-Q10) ----
# Protocol per round: broadcast global adapter -> each client runs DP-SGD locally ->
# clients upload adapter deltas -> server does sample-weighted FedAvg -> evaluate.

def local_train(model, dataset, cfg, noise_multiplier, accountant):
    """One client's local update. Returns (payload, wall_time, steps).

    Opacus adds two things to plain SGD: per-sample gradient clipping to
    MAX_GRAD_NORM (bounds one patient's influence = the sensitivity) and Gaussian
    noise of scale sigma * MAX_GRAD_NORM on the summed gradient."""
    model.train()
    loader = DataLoader(dataset, batch_size=cfg["BATCH_SIZE"], shuffle=True)
    optimizer = torch.optim.AdamW([p for p in model.parameters() if p.requires_grad],
                                  lr=cfg["LEARNING_RATE"])
    sample_rate = cfg["BATCH_SIZE"] / len(dataset)
    steps, t0 = 0, time.time()

    if cfg["DP_ENABLED"]:
        engine = PrivacyEngine(accountant="rdp")
        # poisson_sampling=False keeps step counts deterministic for the lab; the RDP
        # accountant still assumes Poisson subsampling, so reported epsilon is conservative.
        model, optimizer, loader = engine.make_private(
            module=model, optimizer=optimizer, data_loader=loader,
            noise_multiplier=noise_multiplier, max_grad_norm=cfg["MAX_GRAD_NORM"],
            poisson_sampling=False)

    for _ in range(cfg["LOCAL_EPOCHS"]):
        for input_ids, attn, y in loader:
            optimizer.zero_grad(set_to_none=True)
            out = model(input_ids=input_ids, attention_mask=attn, labels=y)
            out.loss.backward()
            optimizer.step()
            steps += 1
            if cfg["DP_ENABLED"]:
                # One accountant per client, stepped every round -> cumulative privacy loss.
                accountant.step(noise_multiplier=noise_multiplier, sample_rate=sample_rate)

    dt = time.time() - t0
    inner = model._module if hasattr(model, "_module") else model   # unwrap GradSampleModule
    payload = {k.replace("_module.", ""): v for k, v in trainable_state(inner).items()}
    return payload, dt, steps


def fedavg(payloads, weights):
    """Sample-weighted FedAvg over LoRA adapters + head (McMahan et al., 2017)."""
    w = np.asarray(weights, dtype=np.float64); w = w / w.sum()
    return {k: sum(float(wi) * p[k] for wi, p in zip(w, payloads)) for k in payloads[0]}


@torch.no_grad()
def evaluate(model, clients):
    """Global model accuracy, averaged over the clients' held-out local test shards."""
    model.eval()
    per_client = []
    for c in clients:
        correct = n = 0
        for input_ids, attn, y in DataLoader(c["test"], batch_size=32):
            correct += (model(input_ids=input_ids, attention_mask=attn)
                        .logits.argmax(-1) == y).sum().item()
            n += y.numel()
        per_client.append(correct / max(n, 1))
    return float(np.mean(per_client)), per_client

# ---- Instrumentation: communication, latency and memory (Q4, Q8, Q9) ----

def bandwidth_report(pr, cfg):
    """Bytes on the wire per round: LoRA federated learning vs full-model FedAvg (fp32)."""
    b = 4  # bytes per fp32 parameter
    per_round_lora = pr["trainable"] * b * cfg["NUM_CLIENTS"] * 2   # uplink + downlink
    per_round_full = pr["base_total"] * b * cfg["NUM_CLIENTS"] * 2
    return dict(payload_bytes_per_client=pr["trainable"] * b,
                full_bytes_per_client=pr["base_total"] * b,
                per_round_lora_bytes=per_round_lora, per_round_full_bytes=per_round_full,
                total_lora_bytes=per_round_lora * cfg["ROUNDS"],
                total_full_bytes=per_round_full * cfg["ROUNDS"],
                savings_pct=100.0 * (1 - per_round_lora / per_round_full))


@torch.no_grad()
def latency_report(model, cfg, iters=30, warmup=5):
    """Edge inference latency: single-sample p50/p95 plus batch-16 throughput."""
    model.eval()
    ids = torch.randint(0, 1000, (1, cfg["MAX_LENGTH"])); attn = torch.ones_like(ids)
    for _ in range(warmup):
        model(input_ids=ids, attention_mask=attn)          # warm caches / lazy init
    lat = []
    for _ in range(iters):
        t0 = time.perf_counter(); model(input_ids=ids, attention_mask=attn)
        lat.append((time.perf_counter() - t0) * 1000)
    ids16 = torch.randint(0, 1000, (16, cfg["MAX_LENGTH"])); a16 = torch.ones_like(ids16)
    t0 = time.perf_counter()
    for _ in range(5):
        model(input_ids=ids16, attention_mask=a16)
    batch_ms = (time.perf_counter() - t0) / 5 * 1000
    return dict(p50_ms=float(np.percentile(lat, 50)), p95_ms=float(np.percentile(lat, 95)),
                mean_ms=float(np.mean(lat)), batch16_ms=batch_ms,
                throughput_sps=16 / (batch_ms / 1000))


def memory_report(pr):
    """Static footprint (fp32 / int8) plus the peak RSS observed for this process.

    ru_maxrss is platform-dependent: kilobytes on Linux, bytes on macOS/BSD.
    Normalizing here keeps the reported figure comparable across machines."""
    peak = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
    peak_mb = peak / 1e6 if sys.platform == "darwin" else peak / 1024
    return dict(weights_fp32_mb=pr["base_total"] * 4 / 1e6,
                weights_int8_mb=pr["base_total"] * 1 / 1e6,
                adapter_fp32_kb=pr["trainable"] * 4 / 1e3,
                peak_rss_mb=peak_mb)

# ---- Federated driver: one call = one fully instrumented experiment ----

def run(cfg_updates=None, tag="baseline", verbose=True):
    """Run the whole federated system end-to-end and return every metric Q1-Q15 needs."""
    cfg = dict(BASE_CFG); cfg.update(cfg_updates or {})
    random.seed(SEED); np.random.seed(SEED); torch.manual_seed(SEED)

    backbone, tokenizer, weight_source = load_backbone(cfg)
    clients = build_clients(cfg, tokenizer)

    global_model = build_lora_model(cfg, backbone)
    validator_issues = len(ModuleValidator.validate(global_model, strict=False))
    pr = param_report(global_model, backbone)

    n_train = int(cfg["SAMPLES_PER_CLIENT"] * cfg["TRAIN_SPLIT"])
    sample_rate = cfg["BATCH_SIZE"] / n_train
    steps_per_round = math.ceil(n_train / cfg["BATCH_SIZE"]) * cfg["LOCAL_EPOCHS"]

    # Calibrate sigma ONCE against the total number of steps across all rounds, so the
    # cumulative budget over the whole federation lands on the target EPSILON.
    noise_multiplier = 0.0
    if cfg["DP_ENABLED"]:
        noise_multiplier = get_noise_multiplier(
            target_epsilon=cfg["EPSILON"], target_delta=cfg["DELTA"],
            sample_rate=sample_rate, steps=steps_per_round * cfg["ROUNDS"], accountant="rdp")

    accountants = [RDPAccountant() for _ in clients]
    global_state = trainable_state(global_model)
    baseline_acc = 1.0 / cfg["NUM_CLASSES"]          # uniform random guess

    round_logs = []
    for r in range(cfg["ROUNDS"]):
        payloads, times, weights = [], [], []
        for c, acct in zip(clients, accountants):
            local = build_lora_model(cfg, backbone)   # fresh client model each round
            load_trainable_state(local, global_state) # server broadcast
            p, dt, _ = local_train(local, c["train"], cfg, noise_multiplier, acct)
            payloads.append(p); times.append(dt); weights.append(len(c["train"]))
            del local; gc.collect()      # free the per-client model: 3 rounds of leaked
                                         # DistilBERT copies is enough to OOM a 8 GB host

        global_state = fedavg(payloads, weights)
        load_trainable_state(global_model, global_state)
        acc, per_client = evaluate(global_model, clients)
        eps = [a.get_epsilon(delta=cfg["DELTA"]) if cfg["DP_ENABLED"] else float("inf")
               for a in accountants]

        round_logs.append(dict(round=r + 1, acc=acc, per_client_acc=per_client,
                               client_times=times, avg_client_time_sec=float(np.mean(times)),
                               round_wall_sec=float(np.sum(times)),
                               epsilon_per_client=eps,
                               epsilon_mean=float(np.mean(eps)) if cfg["DP_ENABLED"] else None))
        if verbose:
            e = f" | eps={eps[0]:.3f}" if cfg["DP_ENABLED"] else " | eps=inf (DP off)"
            print(f"[{tag}] Round {r+1}/{cfg['ROUNDS']} | Acc={acc:.3f} | "
                  f"Avg client train time={np.mean(times):.2f}s{e}")

    res = dict(tag=tag, cfg=cfg, weight_source=weight_source, validator_issues=validator_issues,
               params=pr, rounds=round_logs, baseline_acc=baseline_acc,
               final_acc=round_logs[-1]["acc"], improvement=round_logs[-1]["acc"] - baseline_acc,
               noise_multiplier=float(noise_multiplier), sample_rate=float(sample_rate),
               steps_per_round=steps_per_round, total_steps=steps_per_round * cfg["ROUNDS"],
               bandwidth=bandwidth_report(pr, cfg), latency=latency_report(global_model, cfg),
               memory=memory_report(pr),
               client_label_counts=[c["label_counts"] for c in clients],
               total_wall_sec=float(sum(rl["round_wall_sec"] for rl in round_logs)))
    RESULTS[tag] = res
    with open(os.path.join(RESULTS_DIR, f"{tag}.json"), "w") as fh:
        json.dump(res, fh, indent=2, default=float)
    return res


# Baseline run with the Optimized Configuration V1.2.0.
baseline_res = run({}, tag="baseline")
print(f"\nBaseline (random guess): {baseline_res['baseline_acc']:.3f} | "
      f"Final global accuracy: {baseline_res['final_acc']:.3f} | "
      f"Improvement: {baseline_res['improvement']:+.3f}")
print(f"Backbone weights: {baseline_res['weight_source']} | "
      f"sigma={baseline_res['noise_multiplier']:.3f} | "
      f"cumulative eps={baseline_res['rounds'][-1]['epsilon_per_client'][0]:.3f} "
      f"(target {EPSILON}) at delta={DELTA}")

[transformers] DistilBertForSequenceClassification LOAD REPORT from: distilbert-base-uncased
Key                     | Status     | 
------------------------+------------+-
vocab_transform.weight  | UNEXPECTED | 
vocab_transform.bias    | UNEXPECTED | 
vocab_layer_norm.weight | UNEXPECTED | 
vocab_layer_norm.bias   | UNEXPECTED | 
vocab_projector.bias    | UNEXPECTED | 
pre_classifier.weight   | MISSING    | 
pre_classifier.bias     | MISSING    | 
classifier.bias         | MISSING    | 
classifier.weight       | MISSING    | 

Notes:
- UNEXPECTED:	can be ignored when loading from different task/architecture; not ok if you expect identical arch.
- MISSING:	those params were newly initialized because missing from the checkpoint. Consider training on your downstream task.
/opt/anaconda3/envs/cv-ml-py311/lib/python3.11/site-packages/opacus/privacy_engine.py:98: UserWarning: Secure RNG turned off. This is perfectly fine for experimentation as it allows for much faster training performance, but remember to turn it on and retrain one last time before production with ``secure_mode`` turned on.
  warnings.warn(
[transformers] `use_return_dict` is deprecated! Use `return_dict` instead!
/var/folders/sy/bq80xgcs6130507ldlh7sntm0000gn/T/ipykernel_20600/3409225596.py:31: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
  out.loss.backward()

[baseline] Round 1/5 | Acc=0.467 | Avg client train time=0.74s | eps=4.513
[baseline] Round 2/5 | Acc=0.467 | Avg client train time=0.68s | eps=6.249
[baseline] Round 3/5 | Acc=0.533 | Avg client train time=0.72s | eps=7.644
[baseline] Round 4/5 | Acc=0.400 | Avg client train time=0.72s | eps=8.873
[baseline] Round 5/5 | Acc=0.367 | Avg client train time=0.70s | eps=9.991

Baseline (random guess): 0.333 | Final global accuracy: 0.367 | Improvement: +0.033
Backbone weights: pretrained | sigma=0.919 | cumulative eps=9.991 (target 10.0) at delta=0.001

Part 4 — Visualizations and Observational Analysis (Q1–Q10)

# ---- Part 4: Visualizations ----
import matplotlib.pyplot as plt

df = pd.DataFrame([{ "round": r["round"], "acc": r["acc"],
                     "avg_client_time_sec": r["avg_client_time_sec"],
                     "epsilon": r["epsilon_per_client"][0] } for r in baseline_res["rounds"]])

fig, axes = plt.subplots(1, 3, figsize=(15, 4))
axes[0].plot(df["round"], df["acc"], marker="o", label="Global model")
axes[0].axhline(baseline_res["baseline_acc"], ls="--", color="grey", label="Random guess (1/3)")
axes[0].set_title("Global Accuracy per Round"); axes[0].set_xlabel("Round")
axes[0].set_ylabel("Accuracy"); axes[0].legend()

axes[1].plot(df["round"], df["epsilon"], marker="s", color="firebrick")
axes[1].axhline(EPSILON, ls="--", color="grey", label=f"Budget eps={EPSILON}")
axes[1].set_title("Cumulative Privacy Loss per Client"); axes[1].set_xlabel("Round")
axes[1].set_ylabel("epsilon (delta=1e-3)"); axes[1].legend()

axes[2].bar(df["round"], df["avg_client_time_sec"], color="steelblue")
axes[2].set_title("Average Client Training Time per Round"); axes[2].set_xlabel("Round")
axes[2].set_ylabel("Seconds")
plt.tight_layout(); plt.show()

# Non-IID check: label distribution per client shard.
lab = pd.DataFrame(baseline_res["client_label_counts"], columns=LABELS)
lab.index.name = "client"
print("Local training label distribution (non-IID by design):"); display(lab)

Local training label distribution (non-IID by design):

	Normal	Warning	Critical
client
0	23	15	2
1	12	15	13
2	5	14	21

# ---- Q1-Q5: Performance metrics (computed from the baseline run) ----
b   = baseline_res
pr  = b["params"]; bw = b["bandwidth"]

# Q1 - Final global accuracy and improvement over baseline.
maj = max(np.sum(b["client_label_counts"], axis=0)) / np.sum(b["client_label_counts"])
print("Q1  final_accuracy      =", f"{b['final_acc']:.4f}")
print("Q1  random_baseline     =", f"{b['baseline_acc']:.4f}",
      f"| improvement = {b['improvement']:+.4f} ({b['improvement']/b['baseline_acc']*100:+.1f}% relative)")
print("Q1  majority_class_rate =", f"{maj:.4f}   # stricter baseline given the non-IID skew")

# Q2 - LoRA parameter reduction. Show the calculation explicitly.
#     LoRA adds 2 * r * d parameters per adapted projection: A (d x r) and B (r x d).
d_model, n_layers, n_proj = 768, 6, len(TARGET_MODULES)
expected_lora = 2 * LORA_R * d_model * n_layers * n_proj
print(f"\nQ2  expected LoRA params = 2*r*d*layers*projections = 2*{LORA_R}*{d_model}*{n_layers}*{n_proj} = {expected_lora:,}")
print(f"Q2  measured lora_only   = {pr['lora_only']:,}  | classification head = {pr['head_only']:,}")
print(f"Q2  trainable / full     = {pr['trainable']:,} / {pr['base_total']:,}"
      f"  -> reduction = {pr['reduction_pct']:.3f}%")
print(f"Q2  adapters only        -> reduction = {pr['lora_only_reduction_pct']:.4f}%")

# Q3 - Privacy budget utilization.
eps_final = b["rounds"][-1]["epsilon_per_client"]
print(f"\nQ3  sigma (noise multiplier) = {b['noise_multiplier']:.4f} | clipping C = {MAX_GRAD_NORM}"
      f" | sample rate q = {b['sample_rate']:.3f} | steps/client = {b['total_steps']}")
for i, e in enumerate(eps_final):
    print(f"Q3  client {i}: cumulative eps = {e:.3f} / {EPSILON} "
          f"({e/EPSILON*100:.1f}% of budget) at delta = {DELTA}")

# Q4 - Communication bandwidth savings.
print(f"\nQ4  LoRA payload  = {bw['payload_bytes_per_client']/1e3:,.1f} KB per client per round")
print(f"Q4  full-model    = {bw['full_bytes_per_client']/1e6:,.1f} MB per client per round")
print(f"Q4  per round (uplink+downlink, {NUM_CLIENTS} clients): "
      f"{bw['per_round_lora_bytes']/1e6:.3f} MB vs {bw['per_round_full_bytes']/1e6:.1f} MB")
print(f"Q4  whole training ({ROUNDS} rounds): {bw['total_lora_bytes']/1e6:.2f} MB vs "
      f"{bw['total_full_bytes']/1e6:.1f} MB -> savings = {bw['savings_pct']:.3f}%")

# Q5 - Training time per client and per round.
times = np.array([r["client_times"] for r in b["rounds"]])
print(f"\nQ5  mean client time/round = {times.mean():.2f}s (std {times.std():.2f}s, "
      f"min {times.min():.2f}s, max {times.max():.2f}s)")
print(f"Q5  per-client means       = {[f'{t:.2f}s' for t in times.mean(axis=0)]}")
print(f"Q5  serial wall clock      = {b['total_wall_sec']:.1f}s | "
      f"parallel-equivalent (clients train concurrently) = {times.max(axis=1).sum():.1f}s")

Q1  final_accuracy      = 0.3667
Q1  random_baseline     = 0.3333 | improvement = +0.0333 (+10.0% relative)
Q1  majority_class_rate = 0.3667   # stricter baseline given the non-IID skew

Q2  expected LoRA params = 2*r*d*layers*projections = 2*4*768*6*2 = 73,728
Q2  measured lora_only   = 73,728  | classification head = 592,899
Q2  trainable / full     = 666,627 / 66,955,779  -> reduction = 99.004%
Q2  adapters only        -> reduction = 99.8899%

Q3  sigma (noise multiplier) = 0.9186 | clipping C = 2.0 | sample rate q = 0.400 | steps/client = 15
Q3  client 0: cumulative eps = 9.991 / 10.0 (99.9% of budget) at delta = 0.001
Q3  client 1: cumulative eps = 9.991 / 10.0 (99.9% of budget) at delta = 0.001
Q3  client 2: cumulative eps = 9.991 / 10.0 (99.9% of budget) at delta = 0.001

Q4  LoRA payload  = 2,666.5 KB per client per round
Q4  full-model    = 267.8 MB per client per round
Q4  per round (uplink+downlink, 3 clients): 15.999 MB vs 1606.9 MB
Q4  whole training (5 rounds): 80.00 MB vs 8034.7 MB -> savings = 99.004%

Q5  mean client time/round = 0.71s (std 0.06s, min 0.63s, max 0.92s)
Q5  per-client means       = ['0.73s', '0.70s', '0.71s']
Q5  serial wall clock      = 10.7s | parallel-equivalent (clients train concurrently) = 3.9s

Section A.1 — Performance Metrics (Q1–Q5)

Measured run: seed 42, Optimized Configuration V1.2.0, pretrained DistilBERT backbone.

---

Q1 — Final global accuracy and improvement over baseline.

Metric	Value
Final global accuracy (round 5)	0.3667
Uniform random baseline (1/3)	0.3333
Absolute improvement	+0.0334 (+10.0 % relative)
Majority-class baseline	0.3667
Peak accuracy (round 3)	0.5333
Accuracy trajectory (rounds 1→5)	0.467 → 0.467 → 0.533 → 0.400 → 0.367
Per-client accuracy (round 5)	0.30 / 0.50 / 0.30

Observation. The final round is not the best round. The model climbs to 0.5333 — 1.60× the random baseline — by round 3, then loses that ground over rounds 4 and 5, finishing level with the majority-class baseline and below where it started.

Insight. Accuracy under DP-SGD is not monotone in the number of rounds, so reporting only the final round understates what this system achieved by 0.166. The control run in Q7 rules out underfitting: without privacy the identical pipeline reaches 0.6333, so the capacity to learn this task is present. What each additional round adds is three more noised updates whose gradient signal — on 40 samples per client — no longer exceeds the noise injected to satisfy the budget, so late rounds degrade the aggregate instead of refining it. The practical consequence for a production federation is a stopping rule: hold out a validation shard and keep the best round, rather than running to the ε ceiling. One caveat bounds every accuracy claim below: the test set holds 30 samples, so a single example is worth 0.0333 and differences of that order are noise.

---

Q2 — LoRA parameter reduction.

Quantity	Value
Full DistilBERT (base_total)	66,955,779
LoRA adapters only	73,728
Classification head (pre_classifier + classifier)	592,899
Total trainable / communicated	666,627
Reduction vs full fine-tune	99.004 %
Reduction, adapters only	99.890 %

Calculation: 2 · r · d · layers · projections = 2 · 4 · 768 · 6 · 2 = 73,728, which matches the measured count exactly. The full-rank alternative for the same projections is 768² · 6 · 2 = 7,077,888, so LoRA is a 96× reduction on the attention updates.

Observation. The 99 % headline is real, but 88.9 % of the trainable parameters are the classification head, not the adapters.

Insight. At r = 4 the adapter is effectively free and the head dominates the payload; any further communication optimization must target the head (freeze pre_classifier, or use a low-rank head), not the rank. This inverts a naive "just lower r" tuning strategy — and it is why Experiment 2 (Q12) moves the total payload by only 33 % despite a 4× rank increase.

---

Q3 — Privacy budget utilization.

Parameter	Value
Noise multiplier σ (calibrated)	0.9186
Clipping norm C	2.0
Sampling rate q = B/n	0.400
DP-SGD steps per client	15 (3 per round × 5 rounds)
Cumulative ε per client, δ = 1e-3	9.991 / 9.991 / 9.991 → 99.9 % of budget
ε accumulation by round	4.513 → 6.249 → 7.644 → 8.873 → 9.991

All three clients spend an identical budget because they hold identical shard sizes and therefore identical (q, steps); the accountant is per client, and since each patient record lives on exactly one device, the federation-level guarantee is the per-client one under parallel composition — not their sum.

Observation. The accumulation is concave: round 1 alone costs 4.513 of the 10.0 budget, round 5 only 1.118.

Insight. Read against the Q1 trajectory this is uncomfortable — the rounds that are cheapest in privacy (4 and 5) are precisely the ones that degraded accuracy. The schedule is also fixed at design time: adding a sixth round does not cost "a little more privacy", it invalidates the calibration and requires re-solving σ (visible in Q13, where 8 rounds forced σ from 0.9186 to 1.0773). A production system needs the round budget as a first-class, immutable configuration input, with the accountant as the enforcement point that refuses further rounds once ε is exhausted.

---

Q4 — Communication bandwidth savings.

Payload (fp32)	LoRA-FL	Full-model FedAvg
Per client, per round	2.67 MB	267.82 MB
Per round (3 clients, up + down)	16.00 MB	1,606.94 MB
Whole 5-round training	80.00 MB	8,034.69 MB
Savings	99.004 %	—

Observation. The federation transfers 80 MB instead of 8.03 GB — a 100× reduction.

Insight. This is the metric that decides whether federated learning is deployable at all on wearables. On an LTE-M/NB-IoT link at ~100 kbps, 267 MB per client per round is roughly 6 hours of uplink; 2.67 MB is about 3.5 minutes. The same argument applies to cost: at typical cellular IoT rates the full-model variant is unaffordable at fleet scale, while the LoRA variant is a rounding error. Two further reductions are available and not applied here — fp16 payloads (−50 %) and adapter sparsification — but the 99 % step is the one that changes feasibility.

---

Q5 — Training time per client and per round.

Metric	Value
Mean client training time per round	0.71 s (σ 0.06 s, min 0.64 s, max 0.92 s)
Per-client means (clients 0/1/2)	0.73 s / 0.70 s / 0.71 s
Serial wall-clock for 5 rounds	10.7 s
Parallel-equivalent (clients train concurrently)	3.9 s
Same workload without DP	0.65 s → DP overhead ≈ +10.5 %

Observation. Client time is nearly uniform, and the DP compute overhead is only 10.5 %.

Insight. The uniformity is an artifact of a homogeneous simulation: every client has 40 samples and the same CPU. Real federations are heterogeneous, and the round time is set by the slowest participant — the straggler problem. The gap between 10.7 s serial and 3.9 s parallel is the value the server realizes by running clients concurrently, which is why production FL uses deadline-based partial aggregation (accept the first k of n) rather than waiting for all clients. Read against Q7, this is the study's cheapest number and its most misleading one: differential privacy costs 10.5 % in compute and 89 % of the achievable accuracy — a reminder that the DP tax is not paid in the place a systems engineer would instinctively look for it.

# ---- Q6-Q10: System behaviour ----
# Q7 needs a privacy-free control: identical seed and configuration, DP disabled.
control_res = run({"DP_ENABLED": False}, tag="control_nodp")

lat = b["latency"]; mem = b["memory"]
dp_cost = b["final_acc"] - control_res["final_acc"]

print(f"\nQ6  guarantee: ({b['rounds'][-1]['epsilon_per_client'][0]:.2f}, {DELTA})-DP per client "
      f"over {b['total_steps']} DP-SGD steps; sensitivity bounded by C={MAX_GRAD_NORM}, "
      f"sigma={b['noise_multiplier']:.3f}")
print(f"Q6  exp(eps) = {math.exp(b['rounds'][-1]['epsilon_per_client'][0]):.1f}x  "
      f"# worst-case likelihood ratio for a membership-inference adversary")

print(f"\nQ7  accuracy with DP    = {b['final_acc']:.4f}")
print(f"Q7  accuracy without DP = {control_res['final_acc']:.4f} -> utility cost = {dp_cost:+.4f}")
print(f"Q7  time with DP = {np.mean([r['avg_client_time_sec'] for r in b['rounds']]):.2f}s/client/round"
      f" vs without DP = {np.mean([r['avg_client_time_sec'] for r in control_res['rounds']]):.2f}s"
      f"  # per-sample gradients are the DP compute overhead")

print(f"\nQ8  single-sample latency p50 = {lat['p50_ms']:.1f} ms | p95 = {lat['p95_ms']:.1f} ms "
      f"(seq len {MAX_LENGTH}, CPU)")
print(f"Q8  batch-16 latency = {lat['batch16_ms']:.1f} ms -> {lat['throughput_sps']:.1f} samples/s")

print(f"\nQ9  backbone fp32 = {mem['weights_fp32_mb']:.1f} MB | int8-quantized = {mem['weights_int8_mb']:.1f} MB")
print(f"Q9  adapter payload kept on device = {mem['adapter_fp32_kb']:.1f} KB")
print(f"Q9  peak process RSS during DP training = {mem['peak_rss_mb']:.0f} MB")

# Q10 - Explicit scoring rubric so the composite score is auditable.
U = np.clip((b["final_acc"] - b["baseline_acc"]) / (1 - b["baseline_acc"]), 0, 1)      # utility
P = 1.0 if b["rounds"][-1]["epsilon_per_client"][0] <= EPSILON else EPSILON / b["rounds"][-1]["epsilon_per_client"][0]
E = pr["reduction_pct"] / 100                                                          # parameter efficiency
C = b["bandwidth"]["savings_pct"] / 100                                                # comm efficiency
L = min(1.0, 100.0 / lat["p95_ms"])            # 100 ms = real-time alerting threshold
M = min(1.0, 512.0 / mem["weights_fp32_mb"])   # 512 MB = mid-range wearable gateway budget
WEIGHTS = dict(utility=0.30, privacy=0.20, param_eff=0.15, comm_eff=0.15, latency=0.10, memory=0.10)
scores = dict(utility=U, privacy=P, param_eff=E, comm_eff=C, latency=L, memory=M)
composite = sum(WEIGHTS[k] * scores[k] for k in WEIGHTS) * 100
print("\nQ10 component scores:", {k: round(v, 3) for k, v in scores.items()})
print(f"Q10 composite system score = {composite:.1f}/100")

[transformers] DistilBertForSequenceClassification LOAD REPORT from: distilbert-base-uncased
Key                     | Status     | 
------------------------+------------+-
vocab_transform.weight  | UNEXPECTED | 
vocab_transform.bias    | UNEXPECTED | 
vocab_layer_norm.weight | UNEXPECTED | 
vocab_layer_norm.bias   | UNEXPECTED | 
vocab_projector.bias    | UNEXPECTED | 
pre_classifier.weight   | MISSING    | 
pre_classifier.bias     | MISSING    | 
classifier.bias         | MISSING    | 
classifier.weight       | MISSING    | 

Notes:
- UNEXPECTED:	can be ignored when loading from different task/architecture; not ok if you expect identical arch.
- MISSING:	those params were newly initialized because missing from the checkpoint. Consider training on your downstream task.

[control_nodp] Round 1/5 | Acc=0.367 | Avg client train time=0.69s | eps=inf (DP off)
[control_nodp] Round 2/5 | Acc=0.367 | Avg client train time=0.67s | eps=inf (DP off)
[control_nodp] Round 3/5 | Acc=0.367 | Avg client train time=0.63s | eps=inf (DP off)
[control_nodp] Round 4/5 | Acc=0.367 | Avg client train time=0.63s | eps=inf (DP off)
[control_nodp] Round 5/5 | Acc=0.633 | Avg client train time=0.61s | eps=inf (DP off)

Q6  guarantee: (9.99, 0.001)-DP per client over 15 DP-SGD steps; sensitivity bounded by C=2.0, sigma=0.919
Q6  exp(eps) = 21829.6x  # worst-case likelihood ratio for a membership-inference adversary

Q7  accuracy with DP    = 0.3667
Q7  accuracy without DP = 0.6333 -> utility cost = -0.2667
Q7  time with DP = 0.71s/client/round vs without DP = 0.65s  # per-sample gradients are the DP compute overhead

Q8  single-sample latency p50 = 17.3 ms | p95 = 25.8 ms (seq len 64, CPU)
Q8  batch-16 latency = 102.0 ms -> 156.9 samples/s

Q9  backbone fp32 = 267.8 MB | int8-quantized = 67.0 MB
Q9  adapter payload kept on device = 2666.5 KB
Q9  peak process RSS during DP training = 2336 MB

Q10 component scores: {'utility': 0.05, 'privacy': 1.0, 'param_eff': 0.99, 'comm_eff': 0.99, 'latency': 1.0, 'memory': 1.0}
Q10 composite system score = 71.2/100

Section A.2 — System Behaviour (Q6–Q10)

---

Q6 — Differential privacy guarantees and practical implications.

The system provides (9.991, 1e-3)-differential privacy per client over 15 DP-SGD steps, enforced by per-sample clipping at C = 2.0 and Gaussian noise with σ = 0.9186, accounted with Rényi DP and converted to (ε, δ).

Formally: for any two shards differing in one patient, the probability of the released adapter falling in any set differs by at most e^ε (plus δ). Practically: e^9.991 ≈ 2.2 × 10⁴. That is a very weak worst-case bound — an adversary is permitted a 21,830× likelihood ratio when testing whether a given patient was in the training set. The specification's own configuration table calls this "Privacy Settings (Relaxed for Education)", and the number makes the reason concrete.

Insight. Three implications matter for deployment. (i) ε = 10 is not a HIPAA/GDPR answer; regulated deployments target ε ≤ 1–3. (ii) The guarantee is per client per release; it does not cover the model's outputs after deployment, side channels, or the fact that the server learns which clients participated — DP is one control, not the control set. (iii) δ = 1e-3 is loose for a medical fleet: it permits a 1-in-1,000 failure of the guarantee, which at 10,000 patients is 10 expected individuals; δ ≤ 1/n is the defensible setting. The Q7 results add a fourth, and it is the sharpest: at this data scale the system pays the full utility price of differential privacy while receiving a guarantee too weak to claim in a regulated filing. That is the worst cell of the trade-off matrix — and it is the configuration the assignment prescribes.

---

Q7 — Privacy–utility trade-off.

Configuration	σ	Cumulative ε	Final accuracy	vs control
DP disabled (control)	0.0	∞	0.6333	—
ε = 10 (baseline)	0.9186	9.991	0.3667	−0.2667
ε = 1 (Experiment 1)	4.7656	0.993	0.4000	−0.2333

Observation. Differential privacy costs 0.2667 accuracy at the prescribed budget: the control reaches 0.6333 while the private run finishes at 0.3667, so DP consumes 89 % of the improvement over the random baseline that the pipeline is otherwise capable of. The second row is the surprising one — tightening the budget tenfold moved accuracy by +0.0333, one test sample, in the direction opposite to the conventional story.

Insight. Between ε = 1 and ε = 10 there is no measurable utility difference at this scale, and the mechanism explains why. σ rises 5.19× (0.9186 → 4.7656) for the tenfold budget reduction, confirming the strongly sub-linear ε↔σ relationship — but with only 15 optimizer steps on 40 samples per client, the noise norm already exceeds the clipped gradient norm at σ = 0.92. Once the update is noise-dominated, adding more noise changes little; both budgets land the model in the same degenerate regime. This reverses the usual tuning advice: at this data scale, relaxing ε buys nothing, so there is no reason to accept a weak guarantee. Utility is recovered only by changing the regime — more data per client (lower q at fixed batch size, so less privacy cost per step) and more clients (FedAvg averages k independent noise draws, reducing aggregate noise variance by 1/k). Federated scale, not budget relaxation, is the lever.

---

Q8 — Edge inference latency and real-time suitability.

Metric	Value (CPU, 2 threads, seq len 64)
Single-sample p50	17.3 ms
Single-sample p95	25.8 ms
Batch-16 latency	102.0 ms
Batch-16 throughput	156.9 samples/s

Observation. p95 of 25.8 ms sits well inside a 100 ms interactive budget.

Insight. Vital-sign monitoring samples at 1 Hz or slower, so a 17 ms inference has roughly 58× headroom per patient-second — the model is not the bottleneck; radio duty-cycling and sensor sampling are. Two deployment caveats: this is a desktop-class CPU restricted to 2 threads, and a Cortex-A53-class wearable SoC is typically 5–10× slower, which pushes p95 to ~130–260 ms — still acceptable for this workload; and LoRA adapters should be merged into the base weights (merge_and_unload()) before deployment, since keeping them separate adds an extra matmul per adapted projection on every forward pass.

---

Q9 — Memory utilization and deployment feasibility.

Component	Size
DistilBERT backbone, fp32	267.8 MB
Same backbone, int8-quantized	67.0 MB
Adapter payload exchanged per round	2.67 MB
Peak process RSS during DP training	2,336 MB

Observation. Inference is cheap; DP training is not — peak RSS is ~8.7× the model size.

Insight. The multiplier comes from Opacus materializing per-sample gradients: memory scales with batch_size × trainable_params rather than trainable_params. This is the single most important edge-deployment constraint in the system, with three standard mitigations: BatchMemoryManager (virtual batches that preserve the accounting while capping physical memory), a smaller physical batch with gradient accumulation, or moving training to a gateway tier while the wearable performs inference only. A 267 MB fp32 backbone plus a 2.3 GB training peak rules out a microcontroller-class device entirely and points to a gateway-trains / device-infers topology; int8 quantization at 67 MB makes the inference side feasible on a mid-range wearable, and the 2.67 MB adapter is small enough to store several task-specific adapters on-device and swap them.

Platform note. resource.getrusage().ru_maxrss returns kilobytes on Linux but bytes on macOS/BSD. memory_report() normalizes for this; without the guard the figure above is misreported by a factor of 1,024 on macOS.

---

Q10 — Overall system performance score.

Component	Weight	Score	Basis
Utility	0.30	0.050	(acc − 1/3) / (1 − 1/3)
Privacy	0.20	1.000	budget met: ε 9.991 ≤ 10 at δ = 1e-3
Parameter efficiency	0.15	0.990	99.004 % reduction
Communication efficiency	0.15	0.990	99.004 % savings
Latency	0.10	1.000	p95 25.8 ms ≤ 100 ms target
Memory	0.10	1.000	267.8 MB ≤ 512 MB gateway budget
Composite	1.00	71.2 / 100	weighted sum

Observation. The composite is carried entirely by the engineering components; utility is the sole failing dimension (0.050).

Insight. The control run makes that number interpretable rather than merely bad. At 0.6333 the privacy-free pipeline scores 0.450 on the same utility scale and would lift the composite to 83.2 — so the 12.0-point gap is the price of the privacy component, paid to obtain a guarantee (ε = 9.991) that Q6 shows is too weak to claim in a regulated setting. Note also what the rubric conceals: scoring the privacy dimension on whether the budget was met rather than on whether the resulting ε is defensible awards a perfect 1.000 here. The finding is therefore not that the system underperforms but that it is operating outside its viable regime. The infrastructure would survive a design review unchanged; the interventions, ordered by expected effect, are (1) more data per client, (2) more clients, (3) a re-calibrated round budget that stops at the accuracy peak rather than the ε ceiling, and (4) a class-balanced or focal loss for the ward/ICU label skew. None is an architecture change.

Part 5 — Parameter Exploration (Q11–Q15)

Each experiment changes one variable at a time against the Optimized Configuration V1.2.0 and re-runs the entire federated system: data is re-sharded, σ is re-calibrated for the new (q, steps) pair, and every metric is re-measured. Nothing below is extrapolated.

# ---- Part 5: Experiments Q11-Q15 (one variable at a time) ----
# Every experiment is a full re-run of the federated system, not a simulation:
# data is re-sharded, sigma is re-calibrated, and all metrics are re-measured.

EXPERIMENTS = [
    ("exp11_eps1",      {"EPSILON": 1.0},                              "Q11 Privacy impact: EPSILON 10.0 -> 1.0"),
    ("exp12_r16",       {"LORA_R": 16, "LORA_ALPHA": 32},              "Q12 LoRA optimization: LORA_R 4 -> 16"),
    ("exp13_scale",     {"NUM_CLIENTS": 2, "ROUNDS": 8},               "Q13 Federated scale: 3 clients/5 rounds -> 2/8"),
    ("exp14_intensity", {"LOCAL_EPOCHS": 3, "LEARNING_RATE": 5e-4},    "Q14 Training intensity: 1 epoch@1e-3 -> 3 epochs@5e-4"),
    ("exp15_scarcity",  {"SAMPLES_PER_CLIENT": 25},                    "Q15 Data constraints: 50 -> 25 samples/client"),
]

for tag, updates, title in EXPERIMENTS:
    print("\n" + "=" * 90); print(title); print("=" * 90)
    run(updates, tag=tag)

# Consolidated before/after comparison table.
def summarize(res):
    e = res["rounds"][-1]["epsilon_per_client"][0]
    return dict(
        run=res["tag"],
        final_acc=round(res["final_acc"], 4),
        improvement=round(res["improvement"], 4),
        eps_final=(None if not res["cfg"]["DP_ENABLED"] else round(e, 3)),
        sigma=round(res["noise_multiplier"], 4),
        trainable_params=res["params"]["trainable"],
        param_reduction_pct=round(res["params"]["reduction_pct"], 3),
        payload_kb=round(res["bandwidth"]["payload_bytes_per_client"] / 1e3, 1),
        total_comm_mb=round(res["bandwidth"]["total_lora_bytes"] / 1e6, 3),
        avg_client_s=round(float(np.mean([r["avg_client_time_sec"] for r in res["rounds"]])), 2),
        total_train_s=round(res["total_wall_sec"], 1),
    )

summary = pd.DataFrame([summarize(RESULTS[t]) for t in
                        ["baseline", "control_nodp"] + [t for t, _, _ in EXPERIMENTS]])
summary.to_csv(os.path.join(RESULTS_DIR, "summary.csv"), index=False)
display(summary)

==========================================================================================
Q11 Privacy impact: EPSILON 10.0 -> 1.0
==========================================================================================

[transformers] DistilBertForSequenceClassification LOAD REPORT from: distilbert-base-uncased
Key                     | Status     | 
------------------------+------------+-
vocab_transform.weight  | UNEXPECTED | 
vocab_transform.bias    | UNEXPECTED | 
vocab_layer_norm.weight | UNEXPECTED | 
vocab_layer_norm.bias   | UNEXPECTED | 
vocab_projector.bias    | UNEXPECTED | 
pre_classifier.weight   | MISSING    | 
pre_classifier.bias     | MISSING    | 
classifier.bias         | MISSING    | 
classifier.weight       | MISSING    | 

Notes:
- UNEXPECTED:	can be ignored when loading from different task/architecture; not ok if you expect identical arch.
- MISSING:	those params were newly initialized because missing from the checkpoint. Consider training on your downstream task.
/opt/anaconda3/envs/cv-ml-py311/lib/python3.11/site-packages/opacus/privacy_engine.py:98: UserWarning: Secure RNG turned off. This is perfectly fine for experimentation as it allows for much faster training performance, but remember to turn it on and retrain one last time before production with ``secure_mode`` turned on.
  warnings.warn(
/var/folders/sy/bq80xgcs6130507ldlh7sntm0000gn/T/ipykernel_20600/3409225596.py:31: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
  out.loss.backward()

[exp11_eps1] Round 1/5 | Acc=0.400 | Avg client train time=0.66s | eps=0.413
[exp11_eps1] Round 2/5 | Acc=0.400 | Avg client train time=0.71s | eps=0.600
[exp11_eps1] Round 3/5 | Acc=0.433 | Avg client train time=0.69s | eps=0.750
[exp11_eps1] Round 4/5 | Acc=0.400 | Avg client train time=0.69s | eps=0.877
[exp11_eps1] Round 5/5 | Acc=0.400 | Avg client train time=0.69s | eps=0.993

==========================================================================================
Q12 LoRA optimization: LORA_R 4 -> 16
==========================================================================================

[transformers] DistilBertForSequenceClassification LOAD REPORT from: distilbert-base-uncased
Key                     | Status     | 
------------------------+------------+-
vocab_transform.weight  | UNEXPECTED | 
vocab_transform.bias    | UNEXPECTED | 
vocab_layer_norm.weight | UNEXPECTED | 
vocab_layer_norm.bias   | UNEXPECTED | 
vocab_projector.bias    | UNEXPECTED | 
pre_classifier.weight   | MISSING    | 
pre_classifier.bias     | MISSING    | 
classifier.bias         | MISSING    | 
classifier.weight       | MISSING    | 

Notes:
- UNEXPECTED:	can be ignored when loading from different task/architecture; not ok if you expect identical arch.
- MISSING:	those params were newly initialized because missing from the checkpoint. Consider training on your downstream task.

[exp12_r16] Round 1/5 | Acc=0.400 | Avg client train time=0.72s | eps=4.513
[exp12_r16] Round 2/5 | Acc=0.400 | Avg client train time=0.68s | eps=6.249
[exp12_r16] Round 3/5 | Acc=0.500 | Avg client train time=0.69s | eps=7.644
[exp12_r16] Round 4/5 | Acc=0.367 | Avg client train time=0.77s | eps=8.873
[exp12_r16] Round 5/5 | Acc=0.367 | Avg client train time=0.66s | eps=9.991

==========================================================================================
Q13 Federated scale: 3 clients/5 rounds -> 2/8
==========================================================================================

[transformers] DistilBertForSequenceClassification LOAD REPORT from: distilbert-base-uncased
Key                     | Status     | 
------------------------+------------+-
vocab_transform.weight  | UNEXPECTED | 
vocab_transform.bias    | UNEXPECTED | 
vocab_layer_norm.weight | UNEXPECTED | 
vocab_layer_norm.bias   | UNEXPECTED | 
vocab_projector.bias    | UNEXPECTED | 
pre_classifier.weight   | MISSING    | 
pre_classifier.bias     | MISSING    | 
classifier.bias         | MISSING    | 
classifier.weight       | MISSING    | 

Notes:
- UNEXPECTED:	can be ignored when loading from different task/architecture; not ok if you expect identical arch.
- MISSING:	those params were newly initialized because missing from the checkpoint. Consider training on your downstream task.

[exp13_scale] Round 1/8 | Acc=0.500 | Avg client train time=0.70s | eps=3.497
[exp13_scale] Round 2/8 | Acc=0.500 | Avg client train time=0.75s | eps=4.835
[exp13_scale] Round 3/8 | Acc=0.500 | Avg client train time=0.74s | eps=5.914
[exp13_scale] Round 4/8 | Acc=0.500 | Avg client train time=0.68s | eps=6.853
[exp13_scale] Round 5/8 | Acc=0.500 | Avg client train time=0.71s | eps=7.715
[exp13_scale] Round 6/8 | Acc=0.500 | Avg client train time=0.71s | eps=8.514
[exp13_scale] Round 7/8 | Acc=0.500 | Avg client train time=0.71s | eps=9.267
[exp13_scale] Round 8/8 | Acc=0.500 | Avg client train time=0.70s | eps=9.992

==========================================================================================
Q14 Training intensity: 1 epoch@1e-3 -> 3 epochs@5e-4
==========================================================================================

[transformers] DistilBertForSequenceClassification LOAD REPORT from: distilbert-base-uncased
Key                     | Status     | 
------------------------+------------+-
vocab_transform.weight  | UNEXPECTED | 
vocab_transform.bias    | UNEXPECTED | 
vocab_layer_norm.weight | UNEXPECTED | 
vocab_layer_norm.bias   | UNEXPECTED | 
vocab_projector.bias    | UNEXPECTED | 
pre_classifier.weight   | MISSING    | 
pre_classifier.bias     | MISSING    | 
classifier.bias         | MISSING    | 
classifier.weight       | MISSING    | 

Notes:
- UNEXPECTED:	can be ignored when loading from different task/architecture; not ok if you expect identical arch.
- MISSING:	those params were newly initialized because missing from the checkpoint. Consider training on your downstream task.

[exp14_intensity] Round 1/5 | Acc=0.467 | Avg client train time=2.22s | eps=4.116
[exp14_intensity] Round 2/5 | Acc=0.433 | Avg client train time=2.17s | eps=5.928
[exp14_intensity] Round 3/5 | Acc=0.367 | Avg client train time=2.15s | eps=7.426
[exp14_intensity] Round 4/5 | Acc=0.367 | Avg client train time=2.12s | eps=8.764
[exp14_intensity] Round 5/5 | Acc=0.367 | Avg client train time=2.18s | eps=9.991

==========================================================================================
Q15 Data constraints: 50 -> 25 samples/client
==========================================================================================

[transformers] DistilBertForSequenceClassification LOAD REPORT from: distilbert-base-uncased
Key                     | Status     | 
------------------------+------------+-
vocab_transform.weight  | UNEXPECTED | 
vocab_transform.bias    | UNEXPECTED | 
vocab_layer_norm.weight | UNEXPECTED | 
vocab_layer_norm.bias   | UNEXPECTED | 
vocab_projector.bias    | UNEXPECTED | 
pre_classifier.weight   | MISSING    | 
pre_classifier.bias     | MISSING    | 
classifier.bias         | MISSING    | 
classifier.weight       | MISSING    | 

Notes:
- UNEXPECTED:	can be ignored when loading from different task/architecture; not ok if you expect identical arch.
- MISSING:	those params were newly initialized because missing from the checkpoint. Consider training on your downstream task.

[exp15_scarcity] Round 1/5 | Acc=0.267 | Avg client train time=0.38s | eps=3.843
[exp15_scarcity] Round 2/5 | Acc=0.267 | Avg client train time=0.39s | eps=5.742
[exp15_scarcity] Round 3/5 | Acc=0.267 | Avg client train time=0.37s | eps=7.312
[exp15_scarcity] Round 4/5 | Acc=0.267 | Avg client train time=0.38s | eps=8.708
[exp15_scarcity] Round 5/5 | Acc=0.267 | Avg client train time=0.37s | eps=9.991

	run	final_acc	improvement	eps_final	sigma	trainable_params	param_reduction_pct	payload_kb	total_comm_mb	avg_client_s	total_train_s
0	baseline	0.3667	0.0333	9.991	0.9186	666627	99.004	2666.5	79.995	0.71	10.7
1	control_nodp	0.6333	0.3000	NaN	0.0000	666627	99.004	2666.5	79.995	0.65	9.7
2	exp11_eps1	0.4000	0.0667	0.993	4.7656	666627	99.004	2666.5	79.995	0.69	10.3
3	exp12_r16	0.3667	0.0333	9.991	0.9186	887811	98.674	3551.2	106.537	0.71	10.6
4	exp13_scale	0.5000	0.1667	9.992	1.0773	666627	99.004	2666.5	85.328	0.71	11.4
5	exp14_intensity	0.3667	0.0333	9.991	1.3599	666627	99.004	2666.5	79.995	2.17	32.5
6	exp15_scarcity	0.2667	-0.0667	9.991	1.1938	666627	99.004	2666.5	79.995	0.38	5.7

# ---- Per-experiment delta versus baseline (what actually changed, and by how much) ----
base = summarize(RESULTS["baseline"])
rows = []
for tag, updates, title in EXPERIMENTS:
    s = summarize(RESULTS[tag])
    rows.append({
        "experiment": tag,
        "changed": ", ".join(f"{k}={v}" for k, v in updates.items()),
        "d_accuracy": round(s["final_acc"] - base["final_acc"], 4),
        "d_eps": (None if s["eps_final"] is None else round(s["eps_final"] - base["eps_final"], 3)),
        "d_sigma": round(s["sigma"] - base["sigma"], 4),
        "d_params": s["trainable_params"] - base["trainable_params"],
        "d_payload_kb": round(s["payload_kb"] - base["payload_kb"], 1),
        "d_total_comm_mb": round(s["total_comm_mb"] - base["total_comm_mb"], 3),
        "d_avg_client_s": round(s["avg_client_s"] - base["avg_client_s"], 2),
        "d_total_train_s": round(s["total_train_s"] - base["total_train_s"], 1),
    })
deltas = pd.DataFrame(rows)
deltas.to_csv(os.path.join(RESULTS_DIR, "deltas.csv"), index=False)
display(deltas)

# Convergence curves for every configuration, on one axis.
plt.figure(figsize=(9, 5))
for tag in ["baseline", "control_nodp"] + [t for t, _, _ in EXPERIMENTS]:
    r = RESULTS[tag]
    plt.plot([x["round"] for x in r["rounds"]], [x["acc"] for x in r["rounds"]],
             marker="o", label=f"{tag} (final {r['final_acc']:.3f})")
plt.axhline(1/3, ls="--", color="grey", label="Random guess")
plt.xlabel("Federated round"); plt.ylabel("Global accuracy")
plt.title("Convergence across all configurations"); plt.legend(fontsize=8); plt.tight_layout(); plt.show()

	experiment	changed	d_accuracy	d_eps	d_sigma	d_params	d_payload_kb	d_total_comm_mb	d_avg_client_s	d_total_train_s
0	exp11_eps1	EPSILON=1.0	0.0333	-8.998	3.8470	0	0.0	0.000	-0.02	-0.4
1	exp12_r16	LORA_R=16, LORA_ALPHA=32	0.0000	0.000	0.0000	221184	884.7	26.542	0.00	-0.1
2	exp13_scale	NUM_CLIENTS=2, ROUNDS=8	0.1333	0.001	0.1587	0	0.0	5.333	0.00	0.7
3	exp14_intensity	LOCAL_EPOCHS=3, LEARNING_RATE=0.0005	0.0000	0.000	0.4413	0	0.0	0.000	1.46	21.8
4	exp15_scarcity	SAMPLES_PER_CLIENT=25	-0.1000	0.000	0.2752	0	0.0	0.000	-0.33	-5.0

Section B — Experimental Results (Q11–Q15)

Measured results (seed 42, pretrained backbone). The test set holds 30 samples, so accuracy deltas of ±0.0333 or less are within one sample and are reported as no measurable effect.

Run	Change	Final acc	Δ vs base	σ	ε	Trainable	Total comm	Avg client time
baseline	—	0.3667	—	0.9186	9.991	666,627	80.00 MB	0.71 s
control	DP off	0.6333	+0.267	0.0	∞	666,627	80.00 MB	0.65 s
Q11	ε 10 → 1	0.4000	+0.033 (noise)	4.7656	0.993	666,627	80.00 MB	0.69 s
Q12	r 4 → 16	0.3667	0.000	0.9186	9.991	887,811	106.54 MB	0.71 s
Q13	3 cl/5 rd → 2/8	0.5000	+0.133	1.0773	9.992	666,627	85.33 MB	0.71 s
Q14	1 ep@1e-3 → 3 ep@5e-4	0.3667	0.000	1.3599	9.991	666,627	80.00 MB	2.17 s
Q15	50 → 25 samples/client	0.2667	−0.100	1.1938	9.991	666,627	80.00 MB	0.38 s

Only three results exceed the noise band: the DP control, Q13 and Q15.

---

Q11 — Experiment 1: Privacy impact (EPSILON 10.0 → 1.0).

Metric	Before	After	Δ
Noise multiplier σ	0.9186	4.7656	×5.19
Cumulative ε (δ = 1e-3)	9.991	0.993	−9.0
Final accuracy	0.3667	0.4000	+0.0333 (one test sample)
Accuracy trajectory	0.467→0.533→0.367	0.400 (flat, all 5 rounds)	—
Avg client time	0.71 s	0.69 s	≈ 0

Observation. A 10× privacy tightening changed accuracy by one test sample — and in the wrong direction for the conventional trade-off — while training time was unchanged.

Insight. Time is unaffected because DP-SGD's cost is per-sample gradient computation and clipping, not noise sampling: privacy strength is free in compute. The accuracy result is not "stronger privacy helped"; it is that at this data scale the two budgets are indistinguishable, because both sit in the noise-dominated regime described in Q7 — the ε = 1 curve is flat at 0.400 across all five rounds, with no learning signal at any point. Read with the Q7 control, the operative variable is DP on/off, not the budget level. That removes the usual argument for accepting a weak guarantee: since ε = 1 costs nothing relative to ε = 10 here, there is no utility case for operating at ε = 10. Restoring utility requires leaving the noise-dominated regime — more data and more clients — not a looser budget.

---

Q12 — Experiment 2: LoRA optimization (LORA_R 4 → 16, alpha 8 → 32).

Metric	Before	After	Δ
LoRA adapter parameters	73,728	294,912	×4 (exactly linear in r)
Total trainable	666,627	887,811	+33.2 %
Parameter reduction	99.004 %	98.674 %	−0.33 pp
Payload per client per round	2,666.5 KB	3,551.2 KB	+33.2 %
Total communication (5 rounds)	80.00 MB	106.54 MB	+26.5 MB
Final accuracy	0.3667	0.3667	0.000
Avg client time	0.71 s	0.71 s	≈ 0

Observation. Quadrupling the rank quadrupled the adapter parameters exactly, raised total communication by a third, and bought no accuracy — every intermediate-round difference is within one test sample.

Insight. Capacity is not the binding constraint. With 40 samples per client and 15 noised steps, a 4× larger adapter is 4× more parameters to estimate from the same data under the same privacy budget — the DP noise is injected into a larger parameter space and the effective signal-to-noise per coordinate falls, so the added expressiveness is spent fitting noise. The scaling relations are exactly linear and worth stating as design rules: adapter parameters = 2·r·d·L·P, so payload and per-round bandwidth are linear in r while the accuracy return is flat. This experiment purchased 26.5 MB of additional traffic for nothing. Note also that total payload grew only 33 % for a 4× rank increase, because the classification head (592,899 params) dominates — the Q2 insight, quantified.

---

Q13 — Experiment 3: Federated scale (NUM_CLIENTS 3 → 2, ROUNDS 5 → 8).

Metric	Before	After	Δ
Final accuracy	0.3667	0.5000	+0.1333 (best private result)
Accuracy trajectory	0.467→0.533→0.367 (peaks, decays)	0.500 (flat, all 8 rounds)	stable
DP-SGD steps per client	15	24	+60 %
Noise multiplier σ	0.9186	1.0773	+17.3 % (re-calibrated)
Cumulative ε	9.991	9.992	unchanged (budget respected)
Total communication	80.00 MB	85.33 MB	+6.7 %
Total training wall-clock	10.7 s	11.4 s	+6.5 %

Observation. Removing a client but adding three rounds produced the highest accuracy of any private configuration, at a 6.7 % communication cost and the same ε — and the curve is flat at 0.500 across all eight rounds rather than peaking and decaying like the baseline.

Insight. The stability is the more interesting half. Dropping the ICU client removed the most skewed shard, so the remaining two ([23, 15, 2] and [12, 15, 13]) disagree less, FedAvg over their updates is a smaller-variance operation, and the aggregate stops oscillating — the Q1 degradation does not occur. The accountant meanwhile behaves exactly as designed: 60 % more steps forced σ up 17.3 % so cumulative ε still landed on 9.992, demonstrating that rounds and noise are exchangeable at fixed privacy, not additive. Caution on the magnitude: 0.500 on a 20-sample two-client test set is ±0.05 per sample, so the direction is trustworthy and the precise value is not. Generalizable rule: when a federation is step-limited, spend budget on rounds; when it is data-limited, spend it on clients.

---

Q14 — Experiment 4: Training intensity (LOCAL_EPOCHS 1 → 3, LR 1e-3 → 5e-4).

Metric	Before	After	Δ
DP-SGD steps per client	15	45	×3
Noise multiplier σ	0.9186	1.3599	+48.0 %
Cumulative ε	9.991	9.991	unchanged
Final accuracy	0.3667	0.3667	0.000
Accuracy trajectory	0.467→0.533→0.367	0.467→0.433→0.367→0.367→0.367	decays sooner
Avg client time	0.71 s	2.17 s	×3.04
Total training wall-clock	10.7 s	32.5 s	+204 %

Observation. Tripling local computation tripled the privacy-relevant step count, forced σ up 48 %, cost 3.04× the client time — and delivered no accuracy change at all.

Insight. This is the sharpest trade-off in the study: local epochs are not free under DP. In non-private federated learning, more local work is the standard way to buy accuracy without buying communication. Under DP-SGD the accountant charges per step, so tripling local epochs triples the privacy cost, which is repaid by raising σ — and the added noise cancels the added optimization. Halving the learning rate simultaneously removed the compensating step size. The trajectory is diagnostic: it decays like the baseline but reaches its floor two rounds earlier, because more noised steps per round means the Q1 degradation arrives sooner. System-level conclusion: under a fixed privacy budget the communication/computation trade-off inverts — prefer more rounds of light local work (Q13) over fewer rounds of heavy local work (Q14), because rounds also buy noise averaging across clients while local epochs buy only noise.

---

Q15 — Experiment 5: Data constraints (SAMPLES_PER_CLIENT 50 → 25).

Metric	Before	After	Δ
Local training samples per client	40	20	−50 %
Sampling rate q = B/n	0.400	0.800	×2
DP-SGD steps per client	15	10	−33 %
Noise multiplier σ	0.9186	1.1938	+30.0 %
Cumulative ε	9.991	9.991	unchanged
Final accuracy	0.3667	0.2667	−0.1000 (below random)
Per-client accuracy (final)	0.30 / 0.50 / 0.30	0.40 / 0.20 / 0.20	degraded
Local label counts	[23,15,2] [12,15,13] [5,14,21]	[12,8,0] [8,6,6] [4,5,11]	class lost
Avg client time	0.71 s	0.38 s	−46.5 %

Observation. Halving the data cost 10 accuracy points and produced the only configuration to finish below the random baseline — and client 0's shard now contains zero Critical cases.

Insight. Scarcity compounds through three channels at once, which is why the damage exceeds the naive "half the data" expectation. (i) Fewer steps: 10 instead of 15. (ii) A worse privacy position: with BATCH_SIZE = 16 and only 20 samples, q doubles to 0.8 — each step now touches 80 % of the shard, so the per-step privacy cost rises and σ must increase 30 % for the same ε. (iii) A lost class: client 0 holds no Critical examples and cannot contribute any gradient signal for that class, yet still contributes a full-weight update to FedAvg, so the aggregate inherits its blind spot. The batch-size interaction is the non-obvious trap — BATCH_SIZE must be re-tuned whenever shard size changes, or the privacy accounting silently degrades utility. Both failures are invisible in the global average and visible only in the per-client breakdown, which argues for per-client accuracy and per-client class coverage as mandatory federated monitoring metrics rather than optional diagnostics.

# ---- Final configuration retained: Experiment 5 (data constraints), per the deliverable spec ----
# The submitted notebook must keep the Experiment 5 configuration active.
SAMPLES_PER_CLIENT = 25          # was 50
BASE_CFG["SAMPLES_PER_CLIENT"] = SAMPLES_PER_CLIENT

final_res = RESULTS["exp15_scarcity"]
print("Active final configuration (Experiment 5):")
for k in ["NUM_CLIENTS", "ROUNDS", "LOCAL_EPOCHS", "BATCH_SIZE", "LEARNING_RATE",
          "LORA_R", "LORA_ALPHA", "EPSILON", "DELTA", "MAX_GRAD_NORM", "SAMPLES_PER_CLIENT"]:
    print(f"  {k:20s} = {BASE_CFG[k]}")
print(f"\nExperiment 5 final accuracy = {final_res['final_acc']:.4f} "
      f"| cumulative eps = {final_res['rounds'][-1]['epsilon_per_client'][0]:.3f} "
      f"| sigma = {final_res['noise_multiplier']:.4f}")
print(f"Artifacts written to ./{RESULTS_DIR}/ (per-run JSON, summary.csv, deltas.csv)")

Active final configuration (Experiment 5):
  NUM_CLIENTS          = 3
  ROUNDS               = 5
  LOCAL_EPOCHS         = 1
  BATCH_SIZE           = 16
  LEARNING_RATE        = 0.001
  LORA_R               = 4
  LORA_ALPHA           = 8
  EPSILON              = 10.0
  DELTA                = 0.001
  MAX_GRAD_NORM        = 2.0
  SAMPLES_PER_CLIENT   = 25

Experiment 5 final accuracy = 0.2667 | cumulative eps = 9.991 | sigma = 1.1938
Artifacts written to ./results/ (per-run JSON, summary.csv, deltas.csv)

Reproducibility Notes

Seed control. SEED = 42 is set once in Part 0 and re-applied at the top of every run() call, so each experiment re-shards its data and re-draws its DP noise from the same starting state. Every number in this notebook comes from that single seed: run-to-run variance of the Gaussian noise draw is not characterized here, and the three effects that exceed the resolution floor (the no-DP control, Q13, Q15) would need three to five seeds under a paired design before they could carry confidence intervals.

Resolution floor. The held-out set is 30 samples (10 per client), so one example is worth 0.0333 accuracy. Deltas at or below that band are reported as no measurable effect throughout Parts 4 and 5, and should not be read as real differences.

Environment. Executed top to bottom in a single kernel (cv-ml-py311, Python 3.11, macOS, CPU only — no GPU/MPS path is taken; torch.set_num_threads(2) in Part 0 fixes the thread count so the timing and latency numbers are comparable across runs). Library floors are those of the assignment specification: torch>=2.0.1, transformers>=4.40.0, peft>=0.10.0, opacus>=1.4.0. The exact versions behind this run are printed by the environment stamp at the end of Part 0, so the record travels with the notebook rather than depending on an external environment file.

What is not bit-deterministic. Opacus runs with secure_mode=False (the warning in Part 3 is expected and documented by Opacus), and the DistilBERT checkpoint is fetched from the Hugging Face hub on first use with a freshly initialized 3-class head. A different library minor version, or a different head initialization, moves accuracies within the ±0.0333 band without changing any conclusion drawn here — absolute values should be compared against that band rather than digit for digit.

Execution order. Cells must run in order: Part 3 produces baseline_res, Part 4 consumes it, Part 5 fills RESULTS, and the final cell retains the Experiment 5 configuration required by the deliverable spec. Re-running Part 5 as a block is safe; re-running a single experiment in isolation is not, because σ is calibrated from the configuration active at call time.

Artifacts. Every run writes results/<tag>.json (per-round accuracy, per-client accuracy, client timings, ε per client), plus results/summary.csv and results/deltas.csv. All figures in the accompanying report are rebuilt from those files, so they can be regenerated without re-training.

Naming and logging conventions. One tag per configuration (baseline, control_nodp, exp11_eps1, exp12_r16, exp13_scale, exp14_intensity, exp15_scarcity) is used consistently for the log lines, the RESULTS dictionary, the JSON filenames and the report tables, so any printed line can be traced to its serialized record.

Reflection

System behaviour

The five rounds of federated training exercised every component the specification targets, and they separate cleanly into two verdicts.

The infrastructure works and is production-shaped. LoRA reduced the trainable and communicated surface from 66.96 M parameters to 666,627 — a 99.004 % reduction that turned 8.03 GB of federated traffic into 80.00 MB. The Opacus RDP accountant tracked cumulative privacy loss to 9.991 of a 10.0 budget on all three clients and, in every experiment, absorbed changes in step count by re-calibrating σ (0.9186 → 1.0773 at 8 rounds, → 1.3599 at 3 local epochs, → 1.1938 at half data) so that cumulative ε never drifted. Edge inference measured 17.3 ms p50 / 25.8 ms p95, with roughly 58× headroom against a 1 Hz vital-sign stream. FedAvg over adapters and the head behaved as specified, and raw patient data never left a client.

The learning configuration does not work, and the control run localizes the cause precisely. With differential privacy disabled the identical pipeline reaches 0.6333 — 1.90× the random baseline — so the task is learnable, the backbone is adequate and the aggregation is sound. Under the prescribed (ε = 10, δ = 1e-3) budget the same run finishes at 0.3667, level with the majority-class baseline. DP-SGD consumes 89 % of the achievable improvement. The composite score of 71.2/100 decomposes into near-perfect scores on privacy (1.00), parameter efficiency (0.99), communication (0.99), latency (1.00) and memory (1.00) against 0.05 on utility; the privacy-free pipeline would score 0.450 on utility and lift the composite to 83.2, so the 12.0-point gap is exactly the price of the privacy component.

A second behavioural finding deserves equal weight: accuracy under DP-SGD is not monotone in rounds. The baseline peaked at 0.5333 in round 3 and decayed to 0.3667 by round 5, while the ε accumulation curve was concave — round 1 cost 4.513 of the budget and round 5 only 1.118. The rounds that were cheapest in privacy were the ones that destroyed accuracy.

Experimental results

The five experiments produced four rules that generalize beyond this lab, and one methodological warning.

1. Differential privacy, not the budget, is what costs utility. DP off → on costs 0.2667; ε 10 → 1 costs nothing measurable (+0.0333 = one test sample). Below a threshold of data per client, the update is noise-dominated at any budget, so relaxing ε buys nothing — and therefore there is no reason to accept a weak guarantee.
2. Capacity is not the binding constraint under a fixed privacy budget. LoRA rank 4 → 16 quadrupled adapter parameters, added 26.5 MB of traffic, and changed accuracy by 0.000 — the DP noise is simply spread over a larger parameter space.
3. Rounds beat local epochs under DP. 8 rounds × 2 clients gained +0.133 accuracy for +6.7 % communication and +6.5 % wall-clock; 3 local epochs gained 0.000 for +204 % wall-clock. The accountant charges per step regardless of where the step happens, so steps followed by aggregation (which averages noise) are worth more than steps that are not.
4. Shard size and batch size are coupled through the accountant. Halving the data doubled q to 0.8, forced σ up 30 %, emptied a class from one client's shard, and dropped the model below random — visible only in the per-client breakdown.
5. Methodological warning: with a 30-sample test set, one example is worth 0.0333. Only three results in this study exceed that band (the DP control, Q13 and Q15); Q11, Q12 and Q14 are all reported as no measurable effect, and a submission that reported them as real effects would be over-claiming.

Deployment recommendations

High-security medical (tight privacy constraints). Target ε ≤ 1.0 with δ ≤ 1e-5 (δ ≤ 1/n, where n is the patient population; the lab's 1e-3 permits a 1-in-1,000 failure and is not defensible for a fleet). Q11 makes this cheaper than it looks: at this data scale ε = 1 costs nothing relative to ε = 10. What the tight budget requires is leaving the noise-dominated regime — ≥ 20 clients and ≥ 500 samples per client, which lowers q at a fixed batch size and averages ~20 independent noise draws per aggregation. Keep LORA_R = 4 (Q12), keep LOCAL_EPOCHS = 1 (Q14), and buy progress with rounds under a re-calibrated σ (Q13). Add secure aggregation so the server sees only the sum. Operationally, treat the accountant as an enforcement gate that refuses round k+1 once the budget is exhausted; hold out a validation shard and keep the best round rather than the last, since Q1 shows the final round is not the best one; log (σ, C, q, steps, ε, δ) per round as immutable audit evidence; and pin the model, adapter and accounting-library versions per release so a guarantee can be reproduced from the record.

Resource-constrained IoT (edge efficiency). Adopt a gateway-trains / device-infers split. The measured 2,336 MB peak RSS during DP training rules out on-wearable training, while int8-quantized inference at 67.0 MB and ~17 ms is comfortable on a mid-range device. Keep LORA_R = 4 and BATCH_SIZE = 8–16 with Opacus BatchMemoryManager to cap physical memory while preserving the accounting; merge adapters into the base weights before deployment. Transmit fp16 adapters (2.67 MB → 1.33 MB per client per round) and schedule rounds on Wi-Fi or charging windows. Two device-side adapters at 2.67 MB each are cheap enough to keep resident, allowing task switching without re-downloading a backbone.

Performance-critical (throughput / latency). Run inference with merged adapters and int8 weights on a server tier, batch at 16–32, and target p95 under 100 ms with horizontal replicas; the measured baseline was 156.9 samples/s at 102.0 ms for a batch of 16. For training throughput, Q13 and Q14 are decisive: use more rounds of one local epoch with deadline-based partial aggregation (accept the first k of n) rather than deep local training, since round time is set by the slowest participant and the 10.7 s serial vs 3.9 s parallel gap is the concurrency headroom available. One recommendation is explicitly not made here: relaxing ε to buy throughput. Q11 measured no utility difference between ε = 1 and ε = 10, and Q5 measured only a 10.5 % compute overhead for DP-SGD, so a looser budget purchases neither accuracy nor speed while weakening the guarantee by an order of magnitude. If throughput binds, the levers are batch size, quantization and replica count — not ε.

Limitations

Three limitations bound every claim above. The evaluation set is 30 samples (10 per client), so one sample is worth 0.0333 accuracy and the Q11, Q12 and Q14 deltas all fall inside that band. The data is synthetic and label-conditioned; real vital-sign streams are temporally correlated, which breaks the independent-sample assumption underlying record-level DP accounting when one patient contributes many readings, and would require user-level differential privacy in production. And a single seed was used throughout, so the run-to-run variance of the DP noise draw is not characterized — the three findings that exceed the noise band would need 3–5 seeds to carry confidence intervals.

References

• McMahan, B. et al. (2017). Communication-Efficient Learning of Deep Networks from Decentralized Data. AISTATS.
• Hu, E. et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685.
• Dwork, C. & Roth, A. (2014). The Algorithmic Foundations of Differential Privacy.
• Abadi, M. et al. (2016). Deep Learning with Differential Privacy. ACM CCS.
• Mironov, I. (2017). Rényi Differential Privacy. IEEE CSF.