Intelligent Gesture and Eye-Tracking Recognition Using Multimodal Deep Learning for Accessible Human–Computer Interfaces

Abstract

Accessible human–computer interaction requires reliable control when conventional keyboards, mice, and touch interfaces are difficult to use. This paper proposes a multimodal deep learning architecture combining camera-derived hand gestures, appearance-based gaze estimation, temporal context, and confidence-aware decision fusion. The design introduces independent encoders for hand landmarks and ocular imagery, synchronized temporal modeling, adaptive modality gating, dwell-based selection, and an abstention mechanism to reduce unintended activation. Privacy-sensitive inference is intended to run locally, with explicit consent and configurable calibration. A subject-disjoint experimental protocol is specified to evaluate classification, gaze error, end-to-end latency, task completion, false activations, and subgroup robustness. Illustrative numerical tables and plots demonstrate how such a study could be reported; these figures are not empirical measurements. The proposed framework prioritizes error-aware accessibility rather than classification accuracy alone, and identifies deployment challenges involving lighting, motor variability, occlusion, fatigue, and calibration drift.