Applied Data Science

Final Project Proposal

CICIoT2023 vs CIC IIoT 2025

CROSS-DATASET EVALUATION OF IOT/IIOT INTRUSION DETECTION MODEL GENERALIZATION ON CIC IIOT 2025

Research gap · 5 linked reasons

1

New Dataset

CIC IIoT 2025 Released

Latest IIoT benchmark — synchronized sensor time-series and network traffic. 50 attack types, 7 categories, 40 industrial devices. More representative than its predecessor.

unb.ca/cic · IIoT-Dataset-2025

2

Threat Landscape

Gap

820K Attacks / Day

Up 46% from the previous year (ORDR). The IoT ecosystem keeps expanding — more devices, wider attack surface, older datasets no longer representative enough.

ordr.net · IoT Security Statistics

3

Dataset Bias

Gap

97.7% Class Imbalance

CICIoT2023 is dominated by attack traffic at 97.7%. 99% accuracy looks great, but the model is mostly guessing the majority class — not actually learning.

CICIoT2023 · class analysis

4

ML Implementation

Gap

Dataset → IDS Model

CIC IIoT 2025 is relevant as an ML benchmark for IDS. But IoT security data is complex — multi-class performance falls far below binary classification.

Feature Selection · IDS · Benchmark

5

Generalization

Cross-Dataset Eval

A model that performs well on one dataset may not generalize to another domain. Cross-dataset evaluation proves the model learns general attack patterns, not memorized training data.

Cross-domain · Generalization · IDS

02 / 09

Objectives

1

Evaluate the cross-domain generalization capability of ML-based IDS between IoT and IIoT environments.

2

Quantify the generalization gap between in-dataset and cross-dataset evaluation scenarios.

3

Identify robust network traffic features that remain effective across different datasets.

4

Establish a cross-dataset benchmark using Decision Tree, Random Forest, and XGBoost.

Contributions

Provides a baseline benchmark for the newly released DataSense (CIC IIoT 2025) dataset.

Supports the development of more reliable and generalizable IDS models for real-world IIoT deployment.

Reduces the risk of overestimating IDS performance caused by single-dataset evaluation.

Contributes empirical evidence on the impact of domain shift between IoT and IIoT datasets.

Serves as a reference for future research on cross-dataset intrusion detection evaluation.

03 / 09

Attribute	CICIoT 2023	CIC IIoT 2025 · DataSense
Domain	Consumer IoT · smart home / campus	Industrial IoT · factory / plant floor
Devices	105 heterogeneous IoT devices	40 IIoT + OT industrial devices
Modality	Network traffic (flow / packet)	Sensor time-series + network traffic
Attacks	33 types · 7 categories	50 types · 7 categories
Features	~48 features per flow	Multi-modal: sensor + network features
Labels	Benign + 33 attack classes	Benign + 50 attack types
Challenge	97.7% class imbalance	Domain shift from IoT to IIoT

CICIoT 2023 unb.ca/cic/datasets/iotdataset-2023.html CIC IIoT 2025 unb.ca/cic/datasets/iiot-dataset-2025.html

04 / 09

2.3.1 / [10]

Evaluating Ensemble Versus Non-Ensemble Machine Learning Performance with Preprocessing Techniques for IoT Intrusion Detection on CICIoT2023

Firdaus et al.

Penelitian ini membandingkan model ensemble dan non-ensemble pada CICIoT2023 dengan random undersampling dan seleksi fitur hybrid MI + RFFI. Lima model diuji menggunakan stratified 5-fold cross-validation, dan XGBoost menjadi yang terbaik dengan Macro F1 0,8891 ± 0,0008. Temuan utamanya menegaskan bahwa normalisasi Min-Max penting untuk stabilitas performa, tetapi seluruh evaluasi masih berada dalam domain dataset yang sama.

GapStrong preprocessing, but cross-domain robustness remains untested.

2.3.2 / [11]

Accelerating Classification for IoT Attack Detection Using Decision Tree Model with Gini Impurity Tree-Based Feature Selection Technique

Dzaki et al.

Studi ini mengevaluasi enam teknik feature selection pada Decision Tree untuk CICIoT2023, lalu memilih Gini Impurity Tree-based sebagai pendekatan paling efisien. Subset 10 fitur yang dihasilkan memangkas waktu latih secara signifikan dan menghasilkan pohon yang lebih dangkal, tetapi eksperimen masih fokus pada performa in-dataset.

GapEfficient feature selection, but no cross-domain validation.

2.3.3 / [22]

Interpretable Intrusion Detection for IoT: A CNN-BiLSTM Permutation Importance Framework for Deep Feature Selection

Al-Shibly et al.

Penelitian ini adalah salah satu yang memakai DataSense (CIC IIoT 2025) secara langsung. Dari 94 fitur, 20 fitur terpilih lalu dipakai untuk melatih classifier ringan, dengan Random Forest memberi hasil terbaik. Namun evaluasi masih biner dan sepenuhnya in-dataset, sehingga multiclass cross-dataset generalization belum terjawab.

GapUseful DataSense baseline, but still binary and single-domain.

2.3.4 / [28]

Cross-Dataset Temporal and Semantic Generalization of Intrusion Detection Models for the Future Internet

Elangovan et al.

Studi ini mengevaluasi generalisasi lintas dataset pada empat dataset dari 2017 hingga 2024 tanpa retraining. Tree-ensemble menunjukkan degradasi yang paling rendah dibanding model lain, dan fitur flow/temporal lebih stabil antar dataset. Meski begitu, arah evaluasinya berbeda dari pasangan CICIoT2023 → DataSense yang menjadi fokus penelitian ini.

GapCross-dataset gap is real, but not yet tested on CICIoT2023 → DataSense.

05 / 09

06 / 09

System design diagram for the cross-dataset evaluation pipeline

Train on CICIoT2023, then align features and test on CIC IIoT 2025 to measure the generalization gap.

07 / 09

3

The final result is a three-part answer to the research problem.

Generalization Gap

Measures how far performance drops when the model moves from in-dataset testing to cross-dataset testing.

Robust Feature Analysis

Identifies which features stay useful across both datasets and which ones are too dataset-specific.

Baseline Recommendation

Gives a practical starting point for the best model-feature combination to carry forward.

Rancangan timeline pengerjaan proyek akhir

08 / 09

Closing

Thank You

09 / 09