CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta content="text/html; charset=utf-8" http-equiv="content-type"/> <title>OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning</title>  <meta content="width=device-width, initial-scale=1, shrink-to-fit=no" name="viewport"/> <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/css/bootstrap.min.css" rel="stylesheet" type="text/css"/> <link href="/static/browse/0.3.4/css/ar5iv.0.7.9.min.css" rel="stylesheet" type="text/css"/> <link href="/static/browse/0.3.4/css/ar5iv-fonts.0.7.9.min.css" rel="stylesheet" type="text/css"/> <link href="/static/browse/0.3.4/css/latexml_styles.css" rel="stylesheet" type="text/css"/> <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/js/bootstrap.bundle.min.js"></script> <script src="https://cdnjs.cloudflare.com/ajax/libs/html2canvas/1.3.3/html2canvas.min.js"></script> <script src="/static/browse/0.3.4/js/addons_new.js"></script> <script src="/static/browse/0.3.4/js/feedbackOverlay.js"></script> <meta content="Earables, wearables, health monitoring, on-device machine learning, embedded artificial intelligence, privacy-preserving computing." lang="en" name="keywords"/> <base href="/html/2410.04775v1/"/></head> <body> <nav class="ltx_page_navbar"> <nav class="ltx_TOC"> <ol class="ltx_toclist"> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S1" title="In OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">1 Introduction</a></li> <li class="ltx_tocentry ltx_tocentry_section"> <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S2" title="In OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">2 Design Principles</a> <ol class="ltx_toclist ltx_toclist_section"> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S2.SS1" title="In 2. Design Principles ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">2.1 Local Computation and Energy Efficiency</a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S2.SS2" title="In 2. Design Principles ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">2.2 Dynamic Sensor Access and Multitasking</a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S2.SS3" title="In 2. Design Principles ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">2.3 Unified Communication Across Devices</a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S2.SS4" title="In 2. Design Principles ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">2.4 Privacy by Design</a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_section"> <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S3" title="In OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">3 Hardware Architecture</a> <ol class="ltx_toclist ltx_toclist_section"> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S3.SS1" title="In 3. Hardware Architecture ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">3.1 Form Factor and Sensor Placement</a></li> <li class="ltx_tocentry ltx_tocentry_subsection"> <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S3.SS2" title="In 3. Hardware Architecture ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">3.2 Compute Subsystem</a> <ol class="ltx_toclist ltx_toclist_subsection"> <li class="ltx_tocentry ltx_tocentry_subsubsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S3.SS2.SSS1" title="In 3.2. Compute Subsystem ‣ 3. Hardware Architecture ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">3.2.1 Main Processor and CNN Accelerator</a></li> <li class="ltx_tocentry ltx_tocentry_subsubsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S3.SS2.SSS2" title="In 3.2. Compute Subsystem ‣ 3. Hardware Architecture ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">3.2.2 Specialised Computational Units</a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S3.SS3" title="In 3. Hardware Architecture ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">3.3 Wireless Subsystem</a></li> <li class="ltx_tocentry ltx_tocentry_subsection"> <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S3.SS4" title="In 3. Hardware Architecture ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">3.4 Sensing Subsystem</a> <ol class="ltx_toclist ltx_toclist_subsection"> <li class="ltx_tocentry ltx_tocentry_subsubsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S3.SS4.SSS1" title="In 3.4. Sensing Subsystem ‣ 3. Hardware Architecture ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">3.4.1 Photoplethysmography sensor (PPG)</a></li> <li class="ltx_tocentry ltx_tocentry_subsubsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S3.SS4.SSS2" title="In 3.4. Sensing Subsystem ‣ 3. Hardware Architecture ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">3.4.2 Inertial Measurement Unit</a></li> <li class="ltx_tocentry ltx_tocentry_subsubsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S3.SS4.SSS3" title="In 3.4. Sensing Subsystem ‣ 3. Hardware Architecture ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">3.4.3 Temperature</a></li> <li class="ltx_tocentry ltx_tocentry_subsubsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S3.SS4.SSS4" title="In 3.4. Sensing Subsystem ‣ 3. Hardware Architecture ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">3.4.4 Microphones</a></li> </ol> </li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_section"> <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S4" title="In OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">4 Software Architecture</a> <ol class="ltx_toclist ltx_toclist_section"> <li class="ltx_tocentry ltx_tocentry_subsection"> <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S4.SS1" title="In 4. Software Architecture ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">4.1 Communication Subsystem</a> <ol class="ltx_toclist ltx_toclist_subsection"> <li class="ltx_tocentry ltx_tocentry_subsubsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S4.SS1.SSS1" title="In 4.1. Communication Subsystem ‣ 4. Software Architecture ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">4.1.1 Inter-Module Communication (IMC)</a></li> <li class="ltx_tocentry ltx_tocentry_subsubsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S4.SS1.SSS2" title="In 4.1. Communication Subsystem ‣ 4. Software Architecture ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">4.1.2 Communication Interface</a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_subsection"> <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S4.SS2" title="In 4. Software Architecture ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">4.2 Sense and Compute Subsystem</a> <ol class="ltx_toclist ltx_toclist_subsection"> <li class="ltx_tocentry ltx_tocentry_subsubsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S4.SS2.SSS1" title="In 4.2. Sense and Compute Subsystem ‣ 4. Software Architecture ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">4.2.1 Sensor Distribution Module</a></li> <li class="ltx_tocentry ltx_tocentry_subsubsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S4.SS2.SSS2" title="In 4.2. Sense and Compute Subsystem ‣ 4. Software Architecture ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">4.2.2 Machine Learning Engine</a></li> <li class="ltx_tocentry ltx_tocentry_subsubsection"> <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S4.SS2.SSS3" title="In 4.2. Sense and Compute Subsystem ‣ 4. Software Architecture ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">4.2.3 Audio Manager</a> <ol class="ltx_toclist ltx_toclist_subsubsection"> <li class="ltx_tocentry ltx_tocentry_paragraph"><a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S4.SS2.SSS3.Px1" title="In 4.2.3. Audio Manager ‣ 4.2. Sense and Compute Subsystem ‣ 4. Software Architecture ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">Time-Critical Pipelines</a></li> <li class="ltx_tocentry ltx_tocentry_paragraph"><a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S4.SS2.SSS3.Px2" title="In 4.2.3. Audio Manager ‣ 4.2. Sense and Compute Subsystem ‣ 4. Software Architecture ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">Non-Time-Critical Pipelines</a></li> </ol> </li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S4.SS3" title="In 4. Software Architecture ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">4.3 Load Balancing Subsystem</a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S5" title="In OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">5 Smartphone Application</a></li> <li class="ltx_tocentry ltx_tocentry_section"> <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S6" title="In OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">6 Applications of OmniBuds</a> <ol class="ltx_toclist ltx_toclist_section"> <li class="ltx_tocentry ltx_tocentry_subsection"> <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S6.SS1" title="In 6. Applications of OmniBuds ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">6.1 Existing Applications</a> <ol class="ltx_toclist ltx_toclist_subsection"> <li class="ltx_tocentry ltx_tocentry_subsubsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S6.SS1.SSS1" title="In 6.1. Existing Applications ‣ 6. Applications of OmniBuds ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">6.1.1 Vital Signs Monitoring</a></li> <li class="ltx_tocentry ltx_tocentry_subsubsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S6.SS1.SSS2" title="In 6.1. Existing Applications ‣ 6. Applications of OmniBuds ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">6.1.2 Multi-modal Contextual Recognition</a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_subsection"> <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S6.SS2" title="In 6. Applications of OmniBuds ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">6.2 Future Applications</a> <ol class="ltx_toclist ltx_toclist_subsection"> <li class="ltx_tocentry ltx_tocentry_subsubsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S6.SS2.SSS1" title="In 6.2. Future Applications ‣ 6. Applications of OmniBuds ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">6.2.1 Emotion Recognition and Augmented Feedback</a></li> <li class="ltx_tocentry ltx_tocentry_subsubsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S6.SS2.SSS2" title="In 6.2. Future Applications ‣ 6. Applications of OmniBuds ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">6.2.2 Gesture-Based Interface for Cognitive Augmentation</a></li> <li class="ltx_tocentry ltx_tocentry_subsubsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S6.SS2.SSS3" title="In 6.2. Future Applications ‣ 6. Applications of OmniBuds ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">6.2.3 Personalised Fitness Coaching with Motion Analysis</a></li> <li class="ltx_tocentry ltx_tocentry_subsubsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S6.SS2.SSS4" title="In 6.2. Future Applications ‣ 6. Applications of OmniBuds ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">6.2.4 Adaptive Personal Assistant Based on Physical and Cognitive State</a></li> <li class="ltx_tocentry ltx_tocentry_subsubsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S6.SS2.SSS5" title="In 6.2. Future Applications ‣ 6. Applications of OmniBuds ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">6.2.5 Acoustic Augmented Reality for Situational Awareness</a></li> </ol> </li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S7" title="In OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">7 Conclusion</a></li> </ol></nav> </nav> <div class="ltx_page_main"> <div class="ltx_page_content"> <article class="ltx_document ltx_authors_1line ltx_leqno"> <h1 class="ltx_title ltx_title_document">OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning</h1> <div class="ltx_authors"> Alessandro Montanari Ashok Thangarajan Khaldoon Al-Naimi Andrea Ferlini Yang Liu Ananta Narayanan Balaji Fahim Kawsar Nokia Bell Labs, Cambridge (UK) </div> <div class="ltx_abstract"> <h6 class="ltx_title ltx_title_abstract">Abstract.</h6> Sensory earables have evolved from basic audio enhancement devices into sophisticated platforms for clinical-grade health monitoring and wellbeing management. This paper introduces OmniBuds, an advanced sensory earable platform integrating multiple biosensors and onboard computation powered by a machine learning accelerator, all within a real-time operating system (RTOS). The platform’s dual-ear symmetric design, equipped with precisely positioned kinetic, acoustic, optical, and thermal sensors, enables highly accurate and real-time physiological assessments. Unlike conventional earables that rely on external data processing, OmniBuds leverage real-time onboard computation to significantly enhance system efficiency, reduce latency, and safeguard privacy by processing data locally. This capability includes executing complex machine learning models directly on the device. We provide a comprehensive analysis of OmniBuds’ design, hardware and software architecture demonstrating its capacity for multi-functional applications, accurate and robust tracking of physiological parameters, and advanced human-computer interaction. </div> <div class="ltx_keywords">Earables, wearables, health monitoring, on-device machine learning, embedded artificial intelligence, privacy-preserving computing. </div> <section class="ltx_section" id="S1"> <h2 class="ltx_title ltx_title_section"> 1. Introduction</h2> <div class="ltx_para" id="S1.p1"> Sensory earables have transcended their initial promise of enhanced audio experiences to become powerful tools for accurate health monitoring and personal wellbeing management. Central to this transformation is their ability to leverage the ear’s unique anatomical proximity to critical vascular and acoustic structures, enabling precise physiological sensing while minimising the effects of motion artefacts <cite class="ltx_cite ltx_citemacro_citep">(Röddiger et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib34" title="">2022</a>; Choudhury, <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib12" title="">2021</a>)</cite>. </div> <div class="ltx_para" id="S1.p2"> The integration of advanced biosensors capable of tracking accurate vital markers allows earables to offer a comprehensive assessment of an individual’s physical, physiological, and social context. Recent academic studies have revealed the extensive potential of these devices in a variety of applications. Earables have demonstrated significant promise in the precise tracking of fitness metrics <cite class="ltx_cite ltx_citemacro_citep">(Prakash et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib33" title="">2019</a>; Ferlini et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib17" title="">2019</a>, <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib15" title="">2021a</a>; Atallah et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib5" title="">2014</a>; Ferlini et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib16" title="">2021b</a>)</cite>, monitoring cardiovascular and respiratory parameters <cite class="ltx_cite ltx_citemacro_citep">(Ferlini et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib18" title="">2021c</a>; Romero et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib35" title="">2024</a>; Butkow et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib8" title="">2023</a>)</cite>, hearing screening <cite class="ltx_cite ltx_citemacro_citep">(Shahid et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib36" title="">2024</a>; Demirel et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib13" title="">2023</a>; Chan et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib10" title="">2023</a>)</cite> and modelling motor symptoms associated with neurological disorders such as Parkinson’s disease <cite class="ltx_cite ltx_citemacro_citep">(Goverdovsky et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib22" title="">2017</a>; Bleichner and Debener, <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib7" title="">2017</a>; Kidmose et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib26" title="">2013</a>; Kalanadhabhatta et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib23" title="">2021</a>)</cite>. They have also been utilised to support dementia patients through cognitive assistance <cite class="ltx_cite ltx_citemacro_citep">(Franklin et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib19" title="">2021</a>)</cite> and to enhance auditory function via augmented hearing <cite class="ltx_cite ltx_citemacro_citep">(Veluri et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib39" title="">2024a</a>, <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib40" title="">b</a>, <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib38" title="">2023</a>; Yang et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib43" title="">2020</a>; Yang and Choudhury, <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib42" title="">2021</a>; Demirel et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib14" title="">2024</a>)</cite>. Additionally, earables are pioneering the next frontier of personal computing devices, advancing human-computer interaction (HCI) by enabling seamless and intuitive engagement with devices worn on or around the body <cite class="ltx_cite ltx_citemacro_citep">(Röddiger et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib34" title="">2022</a>)</cite>. This growing body of research underscores the transformative role of earables, particularly in healthcare and HCI, as they pave the way for innovative developments in personal computing. </div> <div class="ltx_para" id="S1.p3"> Yet, despite remarkable strides, the ultimate challenge remains: creating an earable platform that seamlessly combines high performance sensing, functional utility, and design efficiency within the compact form factor demanded by the ear. It is within this delicate balance that the next breakthrough in sensory earables lies. </div> <div class="ltx_para" id="S1.p4"> In 2018, we introduced eSense <cite class="ltx_cite ltx_citemacro_citep">(Kawsar et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib24" title="">2018a</a>)</cite>, a multisensory in-ear platform aimed at advancing research into intelligent earables. By integrating motion and audio sensors with Bluetooth Low Energy (BLE) connectivity, eSense facilitated the monitoring of activities such as speech and movement, while offering APIs to enable developers to access sensor data. This platform provided a foundation for exploring innovative capabilities; however, it also highlighted key challenges, including the need for enhanced sensor accuracy, system integration, design optimisation, user experience, and societal acceptance. These findings revealed both the limitations of eSense and its significant potential for further development and wider application in earable technology. </div> <figure class="ltx_figure" id="S1.F1"><img alt="Refer to caption" class="ltx_graphics ltx_centering ltx_img_landscape" height="631" id="S1.F1.g1" src="x1.png" width="830"/> <figcaption class="ltx_caption ltx_centering">Figure 1. OmniBuds main HW components and vital signs monitored.</figcaption> </figure> <div class="ltx_para" id="S1.p5"> Our reflection suggests that the true utility of sensory earables hinges on their ability to deliver precise and reliable sensing, a task that demands careful sensor placement to fully exploit the anatomical advantages of the ear. The inherent symmetry of the ears offers a unique opportunity, where spatial redundancy can significantly enhance the accuracy of physiological observations. Expanding beyond kinetic and acoustic sensors to incorporate advanced modalities such as optical PPG and temperature sensors introduces an array of new possibilities for health and wellbeing monitoring. In dual-ear configurations, this spatial redundancy unlocks multi-dimensional, robust health assessments, vastly increasing the scope of applications for these devices. </div> <div class="ltx_para" id="S1.p6"> Currently, platforms like eSense operate as passive data collectors, with no onboard computation, relying entirely on external systems for processing. This design introduces inefficiencies, including communication bottlenecks, increased latency, and critical privacy risks due to the need for constant data transmission. We envision integrating local computational capabilities and sufficient storage within ergonomically designed earables. This shift would enable real-time processing, facilitate the execution of machine learning models directly on the device, and drastically enhance system efficiency and responsiveness. Privacy concerns would also be mitigated, as sensitive data could remain on the device without being transmitted externally. Moreover, the inclusion of powerful and programmable digital signal processing (DSP) can further amplify their impact, enabling augmented auditory experiences, spatial hearing enhancements, and even therapeutic interventions. </div> <div class="ltx_para" id="S1.p7"> We anticipate that, with these advancements, earables can evolve from simple firmware-driven devices into fully developed programmable software platforms. This transformation would allow multiple applications to coexist, efficiently multitask, and optimally access sensors, creating a rich, integrated user experience untethered from secondary devices. However, advanced sensing, computing, and intervention capabilities must not compromise the primary user experience—the core functionality of earables as wireless audio devices. Therefore, a unified and efficient communication framework is essential to seamlessly support both wireless audio and these additional functionalities, ensuring a cohesive and high-quality user experience without interference. </div> <div class="ltx_para" id="S1.p8"> These advancements represent a significant shift in the development of earables, positioning them at the forefront of personal health technology and human-computer interaction. In this paper, we introduce OmniBuds, a state-of-the-art sensory earable platform that exemplifies this shift. With an advanced array of carefully placed sensors, onboard computation powered by a machine learning accelerator, local storage, and programmable digital signal processors (DSPs), OmniBuds is designed to operate within a software stack built on a real-time operating system (RTOS). Its functionally and spatially symmetric dual-ear design opens new opportunities in accurate bio-sensing, targeted interventions, and multi-application functionality, all while ensuring ultra-efficient and privacy-preserving operation in a compact and ergonomic form factor. </div> <div class="ltx_para" id="S1.p9"> The paper first outlines the design principles behind OmniBuds, followed by a detailed explanation of its hardware and software architecture. Each key subsystem is examined demonstrating the platform’s capabilities. Finally, we present applications of OmniBuds and explore their vast potential for future innovation, emphasising their transformative role in the advancement of earable technology. </div> </section> <section class="ltx_section" id="S2"> <h2 class="ltx_title ltx_title_section"> 2. Design Principles</h2> <div class="ltx_para" id="S2.p1"> The design of OmniBuds was driven by a central philosophy: integrating hardware and software in a way that maximises performance, efficiency, and versatility for earable computing. Unlike many other devices, where hardware and software development may occur in silos, OmniBuds was conceived as a cohesive platform where the two elements are deeply intertwined. </div> <div class="ltx_para" id="S2.p2"> At the core of this design is the recognition that hardware choices dictate software capabilities, and vice versa. This interdependence enabled us to develop a platform that meets the current technical requirements of researchers. Additionally, it is designed to anticipate future needs in the field, particularly in terms of scalability, flexibility, and real-time processing. </div> <div class="ltx_para" id="S2.p3"> In the following sections, we will explore the key design principles that guided the design of OmniBuds, highlighting how these principles shaped both the device’s hardware and software. </div> <section class="ltx_subsection" id="S2.SS1"> <h3 class="ltx_title ltx_title_subsection"> 2.1. Local Computation and Energy Efficiency</h3> <div class="ltx_para" id="S2.SS1.p1"> A fundamental principle guiding the design of OmniBuds is the prioritisation of local computation. Unlike conventional earbuds, which often rely on external devices to handle intensive processing tasks, OmniBuds are designed to manage complex workloads directly on the device. This approach significantly reduces the need for constant communication with external systems, a major source of power consumption in wearable devices. </div> <div class="ltx_para" id="S2.SS1.p2"> For instance, integrating a CNN accelerator enables OmniBuds to run sophisticated machine learning models on-device, such as those used for speech recognition. The power of the CNN accelerator lies in its ability to execute these complex models in milliseconds, without needing to stream audio data to an external device for processing. This enhances the responsiveness of the earbuds while also significantly reducing battery drain by keeping all computations local. The ability to process speech directly within the device enables applications where real-time interaction and low-latency responses are critical, such as hands-free control of devices or real-time assistance in challenging environments<cite class="ltx_cite ltx_citemacro_citep">(Veluri et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib39" title="">2024a</a>, <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib40" title="">b</a>, <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib38" title="">2023</a>)</cite>. </div> <div class="ltx_para" id="S2.SS1.p3"> Following this principle enables OmniBuds to support a wide range of applications, from real-time biometric monitoring to interactive audio experiences, all while ensuring extended battery life and data privacy. </div> </section> <section class="ltx_subsection" id="S2.SS2"> <h3 class="ltx_title ltx_title_subsection"> 2.2. Dynamic Sensor Access and Multitasking</h3> <div class="ltx_para" id="S2.SS2.p1"> One of the central design principles of OmniBuds is the integration of sensor flexibility and multitasking <cite class="ltx_cite ltx_citemacro_citep">(Min et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib30" title="">2022</a>, <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib29" title="">2023</a>)</cite>, which is achieved by tightly coupling hardware and software design. The platform leverages intelligent sensors capable of performing many functions autonomously, reducing the need to constantly wake up the main microcontroller (MCU). These sensors are equipped with built-in hardware functionalities, such as basic data processing and event detection, allowing them to handle critical tasks independently. </div> <div class="ltx_para" id="S2.SS2.p2"> On the software side, OmniBuds supports a multitasking environment that allows multiple applications to access sensor data simultaneously. Rather than restricting sensor access to a single application at a time, the system is designed to allow multiple applications to share sensor data without conflicts or excessive resource consumption. This requires a coordinated approach at the system level, where the OmniBuds software system manages access and ensures that each application can retrieve the data it needs without impacting the performance of other applications. For instance, physiological monitoring applications can access PPG sensor data while another application may simultaneously use the same data stream for user authentication <cite class="ltx_cite ltx_citemacro_citep">(Yadav et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib41" title="">2018</a>)</cite> or facial expression detection <cite class="ltx_cite ltx_citemacro_citep">(Choi et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib11" title="">2022</a>)</cite>. The system efficiently manages this data access, ensuring that sensor data is shared across applications without compromising performance or power efficiency. </div> <div class="ltx_para" id="S2.SS2.p3"> This holistic design—integrating flexible, autonomous sensors with multitasking software—ensures that OmniBuds remain adaptable to a wide range of experimental or real-world applications. Researchers and developers can take full advantage of the platform’s ability to run multiple, independent applications simultaneously, maximising the potential of each sensor and minimising unnecessary power consumption. </div> </section> <section class="ltx_subsection" id="S2.SS3"> <h3 class="ltx_title ltx_title_subsection"> 2.3. Unified Communication Across Devices</h3> <div class="ltx_para" id="S2.SS3.p1"> A key design principle of OmniBuds is to enable seamless communication between devices and software modules, ensuring that the platform remains both scalable and developer-friendly. This capability is underpinned by the integration of a Bluetooth Classic and Bluetooth Low Energy (BLE) chip at the hardware level and a unified set of APIs at the software level. This integration allows OmniBuds to abstract away the complexities of device-to-device communication, enabling modules to interact seamlessly, whether running locally on the earbuds or across multiple devices in the ecosystem. </div> <div class="ltx_para" id="S2.SS3.p2"> The unified communication framework simplifies multi-device coordination while providing developers with a consistent interface for building robust and scalable applications. This framework allows applications to communicate across earbuds, or with external systems, without requiring developers to manage the intricacies of local versus remote execution. For example, real-time data from one earbud can be transmitted to the other or to an external device, such as a smartphone, while maintaining a high level of performance and low latency. </div> <div class="ltx_para" id="S2.SS3.p3"> OmniBuds’ communication capabilities are particularly suited for research scenarios where multiple devices may need to work in concert. Whether managing sensor data, synchronising tasks, or coordinating between devices, the unified BLE communication framework ensures that developers can focus on building applications rather than dealing with the complexities of communication protocols. </div> </section> <section class="ltx_subsection" id="S2.SS4"> <h3 class="ltx_title ltx_title_subsection"> 2.4. Privacy by Design</h3> <div class="ltx_para" id="S2.SS4.p1"> In modern wearable systems, especially those handling sensitive physiological data, privacy is a non-negotiable requirement. OmniBuds address this through a dual-layer approach, combining local data processing at the hardware level with secure communication channels in the software. </div> <div class="ltx_para" id="S2.SS4.p2"> By prioritising local computation, OmniBuds minimise the need to transmit sensitive data externally, reducing the risk of unauthorised access. When external communication is necessary, the platform uses encrypted Bluetooth Low Energy (BLE) to protect data integrity. This approach ensures that OmniBuds meet the stringent privacy requirements of health-focused research, safeguarding user data throughout all operations. </div> <div class="ltx_para" id="S2.SS4.p3"> Demonstrating the synergy between hardware and software, OmniBuds set a new standard in earable technology. Designed to address current research demands and anticipate future needs, they combine seamless hardware-software integration with a strong focus on privacy and flexibility, positioning them as a leader in earable computing research. </div> </section> </section> <section class="ltx_section" id="S3"> <h2 class="ltx_title ltx_title_section"> 3. Hardware Architecture</h2> <figure class="ltx_figure" id="S3.F2"><img alt="Refer to caption" class="ltx_graphics ltx_centering ltx_img_landscape" height="361" id="S3.F2.g1" src="x2.png" width="830"/> <figcaption class="ltx_caption ltx_centering">Figure 2. OmniBuds Hardware Block Diagram.</figcaption> </figure> <figure class="ltx_figure" id="S3.F3"><img alt="Refer to caption" class="ltx_graphics ltx_centering ltx_img_landscape" height="373" id="S3.F3.g1" src="x3.png" width="664"/> <figcaption class="ltx_caption ltx_centering">Figure 3. OmniBuds PCBs.</figcaption> </figure> <div class="ltx_para" id="S3.p1"> OmniBuds are fully functional True Wireless Stereo (TWS) earbuds with standard features like music playback, calls, Active Noise Cancellation (ANC), and Acoustic Transparency (or Pass-through). What distinguishes them is the integration of additional sensors and computational units, transforming them into a powerful platform for earable computing research. Both earbuds share identical hardware, enabling multi-device computation and sensing. This symmetry allows computational and sensing tasks to be distributed between the two earbuds, such as one performing signals pre-processing while the other handles intensive machine learning tasks, or both working together for enhanced spatial sensing. </div> <div class="ltx_para" id="S3.p2"> At the core of OmniBuds hardware (Figure <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S3.F2" title="Figure 2 ‣ 3. Hardware Architecture ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">2</a>) is a dual-core microcontroller (MCU) with a CNN accelerator for efficient on-device machine learning. A dedicated audio DSP handles audio processing, and the BioHub processor manages PPG data for vital signs estimation, all designed to emphasise local computation for real-time data processing and machine learning. </div> <div class="ltx_para" id="S3.p3"> OmniBuds are equipped with a comprehensive suite of sensors selected for their low-power characteristics and advanced functionalities: a 9-axis IMU, a 3-wavelength PPG sensor, a medical-grade temperature sensor, and three microphones. Each earbud includes 1GB of non-volatile memory and 8MB of RAM, offering ample capacity for complex applications, such as large datasets and sophisticated machine learning models. For interaction, the earbuds feature a button and an RGB LED. Figure <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S3.F3" title="Figure 3 ‣ 3. Hardware Architecture ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">3</a> shows the OmniBuds PCBs, consisting of a mix of rigid, flex-rigid, and flex designs. </div> <div class="ltx_para" id="S3.p4"> The earbuds are powered by a 105mAh battery, providing up to 6 hours of music playback and approximately 8 hours of PPG sensing. The charging case, featuring a 630mAh battery, allows for multiple recharges of the earbuds, extending their total usage time. </div> <div class="ltx_para" id="S3.p5"> In the following sections, we first analyse the OmniBuds form factor and the positioning of the sensors before delving into the main HW submodules: computation, communication and sensing. </div> <section class="ltx_subsection" id="S3.SS1"> <h3 class="ltx_title ltx_title_subsection"> 3.1. Form Factor and Sensor Placement</h3> <figure class="ltx_figure" id="S3.F4"><img alt="Refer to caption" class="ltx_graphics ltx_centering ltx_img_landscape" height="437" id="S3.F4.g1" src="x4.png" width="664"/> <figcaption class="ltx_caption ltx_centering">Figure 4. Placement of the sensors in OmniBuds.</figcaption> </figure> <div class="ltx_para" id="S3.SS1.p1"> Form factor and ergonomics design are crucial in wearables, impacting usability, performance, and sensor data quality. When designing OmniBuds, we considered factors affecting comfort, sensor accuracy, and manufacturing ease. Enclosed in an injection-moulded plastic cover, OmniBuds measure 4.3 x 3 x 2.2 cm and weigh around 12g, roughly twice as much as other commercial true wireless earbuds (e.g., Apple AirPods Pro 2: 5.3 grams <cite class="ltx_cite ltx_citemacro_citep">(app, <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib2" title="">[n. d.]</a>)</cite>, Bose Quite Comfort Earbuds II: 6.24 grams <cite class="ltx_cite ltx_citemacro_citep">(bos, <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib3" title="">[n. d.]</a>)</cite>) Despite this, OmniBuds remain comfortable even when worn for extended periods. OmniBuds feature a more capable and intricate sensor suite than comparable earbuds, requiring a larger battery. Specifically, the battery accounts for about 20% of the earbud weight (approximately 2.4 grams). </div> <div class="ltx_para" id="S3.SS1.p2"> Sensor placement is a critical element of the OmniBuds design, where performance, user comfort, and manufacturing feasibility are carefully balanced. Each positioning decision is driven by extensive validation studies and the need to optimise the quality of data collected without compromising wearability or ease of production. Figure <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S3.F4" title="Figure 4 ‣ 3.1. Form Factor and Sensor Placement ‣ 3. Hardware Architecture ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">4</a> illustrates the placement of the various sensors within the OmniBuds. </div> <div class="ltx_para" id="S3.SS1.p3"> The arrangement of the three microphones is particularly strategic. Two outward-facing microphones, positioned at the top and bottom of each earbud, facilitate essential acoustic features such as ANC, Transparency mode, and beamforming for voice capture. The positioning allows the microphones to target sounds from the user’s environment while enabling effective noise cancellation by capturing ambient noise from multiple angles. Meanwhile, the inward-facing microphone, located within the speaker chamber, serves multiple purposes. While its primary role is as a reference microphone to enhance ANC performance, it also acts as a sensitive detector for internal body sounds, as explored in many studies <cite class="ltx_cite ltx_citemacro_citep">(Ferlini et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib15" title="">2021a</a>; Ma et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib28" title="">2021</a>; Truong et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib37" title="">2022</a>)</cite>. To maximize the accuracy of in-ear recordings, the earbuds are designed to ensure a snug fit in the ear canal, exploiting the occlusion effect for higher-quality body sound detection. The combination of interchangeable ear-tips and an ergonomic earbud shape helps achieve this fit, ensuring both acoustic performance and user comfort. </div> <div class="ltx_para" id="S3.SS1.p4"> The placement of the 9-axis Inertial Measurement Unit (IMU) is equally deliberate. Positioned centrally on the outer plastic cover of the earbuds, the IMU is optimally located to capture a broad range of movements, from subtle facial gestures to larger head and body movements supporting diverse applications that rely on motion data. </div> <div class="ltx_para" id="S3.SS1.p5"> The photoplethysmography sensor (PPG) is crucial in OmniBuds since it supports the measurement of all vital signs (except temperature) directly or indirectly. The accuracy of PPG-derived metrics depends on several factors, including sensor location. In designing OmniBuds, we conducted a systematic validation study to determine the optimal placement for a PPG sensor in an earable <cite class="ltx_cite ltx_citemacro_citep">(Ferlini et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib18" title="">2021c</a>)</cite>. We explored three placements: concha, ear canal, and behind the auricle (or pinna). Our study concluded that the best location for the PPG sensor in OmniBuds is facing towards the concha. Although in-ear canal placement provides slightly more accurate data, the concha placement preserves a higher level of user comfort and practicality for everyday use, which are key factors for long-term wearability in earables. </div> <div class="ltx_para" id="S3.SS1.p6"> Finally, the location of the temperature sensor is determined by the need to measure body core temperature with high accuracy, while accounting for usability and manufacturing constraints. While placing the sensor in the ear canal would provide the most accurate readings <cite class="ltx_cite ltx_citemacro_citep">(Ferlini et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib18" title="">2021c</a>)</cite>, it is instead placed against the skin of the concha, which still offers a reliable proxy for body core temperature. This location strikes the ideal balance between accuracy and manufacturing feasibility, allowing for effective temperature sensing without requiring overly complex computational corrections. </div> <div class="ltx_para" id="S3.SS1.p7"> In summary, the positioning of sensors in OmniBuds reflects a careful evaluation of trade-offs, balancing scientific rigour with practical considerations to ensure the best possible user experience and data quality. </div> </section> <section class="ltx_subsection" id="S3.SS2"> <h3 class="ltx_title ltx_title_subsection"> 3.2. Compute Subsystem</h3> <section class="ltx_subsubsection" id="S3.SS2.SSS1"> <h4 class="ltx_title ltx_title_subsubsection"> 3.2.1. Main Processor and CNN Accelerator</h4> <div class="ltx_para" id="S3.SS2.SSS1.p1"> At the core of the OmniBuds’ computation system is a low-power dual-core MCU (Cortex-M4F and RISC-V) with a floating point unit, providing robust processing power. The MCU features 512KB of Flash memory and 128KB of SRAM, making it well-suited for managing real-time tasks and coordinating the various sensors and subsystems. </div> <div class="ltx_para" id="S3.SS2.SSS1.p2"> To further enhance its computational capabilities, OmniBuds integrates a dedicated hardware-based Convolutional Neural Network (CNN) accelerator. The accelerator consists of 64 convolutional processors, each equipped with its own pooling engine, input cache, weight memory (from 8-bit down to 1-bit width), and convolution engine, enabling highly parallelised execution of machine learning models. The CNN accelerator supports up to 64 layers and can handle a maximum input size of 1024x1024, allowing it to execute a wide range of neural network architectures, including feed-forward models, residual networks, recurrent models and encoder-decoder designs. </div> </section> <section class="ltx_subsubsection" id="S3.SS2.SSS2"> <h4 class="ltx_title ltx_title_subsubsection"> 3.2.2. Specialised Computational Units</h4> <div class="ltx_para" id="S3.SS2.SSS2.p1"> In addition to the main processor and CNN accelerator, OmniBuds include specialised components designed for task-specific workloads, enhancing the platform’s ability to efficiently handle diverse data types while maintaining power efficiency. </div> <div class="ltx_para" id="S3.SS2.SSS2.p2"> One such component is the Audio DSP, which manages audio pipelines and processing tasks. It consists of two sub-cores: the Fast-DSP core for low-latency applications like ANC, and the Slow-DSP core for less time-critical tasks. The DSP allows efficient audio processing without burdening the main MCU, and its flexibility enables it to support a variety of tasks. </div> <div class="ltx_para" id="S3.SS2.SSS2.p3"> Another key component is the BioHub, a co-processor dedicated to biosignal processing. The BioHub manages the PPG sensor and its dedicated 6-axis IMU, computing the user’s vitals in real-time while minimising power consumption. By offloading tasks from the main MCU, the BioHub enables parallel execution, allowing the main processor to enter a low-power state when idle, optimising energy use. </div> </section> </section> <section class="ltx_subsection" id="S3.SS3"> <h3 class="ltx_title ltx_title_subsection"> 3.3. Wireless Subsystem</h3> <div class="ltx_para" id="S3.SS3.p1"> OmniBuds employ a dedicated Bluetooth System-on-Chip (SoC) for wireless communication. This SoC supports both Bluetooth Classic for audio streaming and Bluetooth Low Energy (BLE) for data communication, ensuring seamless connectivity with external devices. LE Audio can also be supported with software updates. In Bluetooth Classic mode, the SoC manages communication with the host device, notifying the main MCU of events like connections and audio stream changes. For BLE, it provides a transparent API for data transmission, allowing the main processor to focus on other tasks without handling communication protocols. </div> <div class="ltx_para" id="S3.SS3.p2"> This division of responsibilities between the Bluetooth SoC and main MCU ensures that OmniBuds can maintain efficient wireless communication while keeping overall power consumption low. We will delve further into the software management of this communication system in Section <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S4.SS1" title="4.1. Communication Subsystem ‣ 4. Software Architecture ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">4.1</a>. </div> </section> <section class="ltx_subsection" id="S3.SS4"> <h3 class="ltx_title ltx_title_subsection"> 3.4. Sensing Subsystem</h3> <section class="ltx_subsubsection" id="S3.SS4.SSS1"> <h4 class="ltx_title ltx_title_subsubsection"> 3.4.1. Photoplethysmography sensor (PPG)</h4> <figure class="ltx_figure" id="S3.F5"><img alt="Refer to caption" class="ltx_graphics ltx_centering ltx_img_landscape" height="406" id="S3.F5.g1" src="extracted/5905686/Images/ob_ppg_sample.png" width="509"/> <figcaption class="ltx_caption ltx_centering">Figure 5. PPG data sampled at 50Hz during rest.</figcaption> </figure> <div class="ltx_para" id="S3.SS4.SSS1.p1"> PPG is a non-invasive optical technique that detects blood volume changes by measuring light absorption. It uses an LED to emit light through the skin and a photodiode to measure the reflected light. As blood volume increases, more light is absorbed; as it decreases, less light is absorbed. This data provides insights into physiological parameters like heart rate, respiration rate, blood pressure, and oxygen saturation <cite class="ltx_cite ltx_citemacro_citep">(Ferlini et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib18" title="">2021c</a>; Balaji et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib6" title="">2023</a>; Romero et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib35" title="">2024</a>)</cite>. </div> <div class="ltx_para" id="S3.SS4.SSS1.p2"> One of the criteria that drove the choice of the specific PPG sensor used in the OmniBuds, is its flexibility and the number of different configurations/parameters that it supports. The sensor includes three LEDs which generate three different wavelengths, green (530nm), red (660nm) and infrared (880nm), and a single photodiode with peak sensitivity at 860nm and a spectral bandwidth range from 420nm to 1020nm. Figure <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S3.F5" title="Figure 5 ‣ 3.4.1. Photoplethysmography sensor (PPG) ‣ 3.4. Sensing Subsystem ‣ 3. Hardware Architecture ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">5</a> shows an example of 50Hz PPG data collected from the OmniBuds at the three available wavelengths where the vascular pulses and breathing-related modulations are clearly visible. </div> </section> <section class="ltx_subsubsection" id="S3.SS4.SSS2"> <h4 class="ltx_title ltx_title_subsubsection"> 3.4.2. Inertial Measurement Unit</h4> <figure class="ltx_figure" id="S3.F6"><img alt="Refer to caption" class="ltx_graphics ltx_centering ltx_img_landscape" height="405" id="S3.F6.g1" src="extracted/5905686/Images/imu_sample.png" width="509"/> <figcaption class="ltx_caption ltx_centering">Figure 6. IMU data sampled at 100Hz during head nodding.</figcaption> </figure> <div class="ltx_para" id="S3.SS4.SSS2.p1"> The OmniBuds feature two separate Inertial Measurement Units (IMUs). The first one is a 6-axis IMU (accelerometer and gyroscope) which is co-located on the same PCB that hosts the PPG sensor and is dedicated to PPG motion artefact removal. In addition to that, OmniBuds also features a 9-axis IMU (i.e., accelerometer, gyroscope, and magnetometer). Whilst this can be used to track macro motions such as walking and running, it can also be exploited by researchers to perform head motion tracking and 3D pose estimation <cite class="ltx_cite ltx_citemacro_citep">(Ferlini et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib17" title="">2019</a>; Lee et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib27" title="">2019</a>)</cite>. </div> <div class="ltx_para" id="S3.SS4.SSS2.p2"> The 9-axis IMU is a modern low-power component which offers natively several features like step detection, step counting, device orientation and single and double click detection. More importantly, the IMU allows the execution of simple on-chip ML models courtesy of its embedded Machine Learning Core module (MLC). The MLC works directly by taking as input the data streams coming from accelerometer, gyroscope and magnetometer, computing features on the data and passing these features to user-defined decision tree models. This enables users to run simple IMU-based inference tasks directly on the sensor waking up the main MCU only when an inference is available. This results in great power saving compared to waking up every time a new raw data sample is available and computing the inference on the MCU. Figure <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S3.F6" title="Figure 6 ‣ 3.4.2. Inertial Measurement Unit ‣ 3.4. Sensing Subsystem ‣ 3. Hardware Architecture ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">6</a> shows data collected from the main OmniBuds’ IMU while the user was nodding. </div> </section> <section class="ltx_subsubsection" id="S3.SS4.SSS3"> <h4 class="ltx_title ltx_title_subsubsection"> 3.4.3. Temperature</h4> <div class="ltx_para" id="S3.SS4.SSS3.p1"> Temperature sensing is achieved using an infrared sensor directed at the concha area of the ear, allowing for applications like body core temperature and fertility tracking. </div> <div class="ltx_para" id="S3.SS4.SSS3.p2"> The sensor was selected for its high accuracy of <math alttext="\pm 0.2^{\circ}" class="ltx_Math" display="inline" id="S3.SS4.SSS3.p2.1.m1.1"><semantics id="S3.SS4.SSS3.p2.1.m1.1a"><mrow id="S3.SS4.SSS3.p2.1.m1.1.1" xref="S3.SS4.SSS3.p2.1.m1.1.1.cmml"><mo id="S3.SS4.SSS3.p2.1.m1.1.1a" xref="S3.SS4.SSS3.p2.1.m1.1.1.cmml">±</mo><msup id="S3.SS4.SSS3.p2.1.m1.1.1.2" xref="S3.SS4.SSS3.p2.1.m1.1.1.2.cmml"><mn id="S3.SS4.SSS3.p2.1.m1.1.1.2.2" xref="S3.SS4.SSS3.p2.1.m1.1.1.2.2.cmml">0.2</mn><mo id="S3.SS4.SSS3.p2.1.m1.1.1.2.3" xref="S3.SS4.SSS3.p2.1.m1.1.1.2.3.cmml">∘</mo></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS4.SSS3.p2.1.m1.1b"><apply id="S3.SS4.SSS3.p2.1.m1.1.1.cmml" xref="S3.SS4.SSS3.p2.1.m1.1.1"><csymbol cd="latexml" id="S3.SS4.SSS3.p2.1.m1.1.1.1.cmml" xref="S3.SS4.SSS3.p2.1.m1.1.1">plus-or-minus</csymbol><apply id="S3.SS4.SSS3.p2.1.m1.1.1.2.cmml" xref="S3.SS4.SSS3.p2.1.m1.1.1.2"><csymbol cd="ambiguous" id="S3.SS4.SSS3.p2.1.m1.1.1.2.1.cmml" xref="S3.SS4.SSS3.p2.1.m1.1.1.2">superscript</csymbol><cn id="S3.SS4.SSS3.p2.1.m1.1.1.2.2.cmml" type="float" xref="S3.SS4.SSS3.p2.1.m1.1.1.2.2">0.2</cn><compose id="S3.SS4.SSS3.p2.1.m1.1.1.2.3.cmml" xref="S3.SS4.SSS3.p2.1.m1.1.1.2.3"></compose></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.SSS3.p2.1.m1.1c">\pm 0.2^{\circ}</annotation><annotation encoding="application/x-llamapun" id="S3.SS4.SSS3.p2.1.m1.1d">± 0.2 start_POSTSUPERSCRIPT ∘ end_POSTSUPERSCRIPT</annotation></semantics></math> C within the range of <math alttext="35^{\circ}" class="ltx_Math" display="inline" id="S3.SS4.SSS3.p2.2.m2.1"><semantics id="S3.SS4.SSS3.p2.2.m2.1a"><msup id="S3.SS4.SSS3.p2.2.m2.1.1" xref="S3.SS4.SSS3.p2.2.m2.1.1.cmml"><mn id="S3.SS4.SSS3.p2.2.m2.1.1.2" xref="S3.SS4.SSS3.p2.2.m2.1.1.2.cmml">35</mn><mo id="S3.SS4.SSS3.p2.2.m2.1.1.3" xref="S3.SS4.SSS3.p2.2.m2.1.1.3.cmml">∘</mo></msup><annotation-xml encoding="MathML-Content" id="S3.SS4.SSS3.p2.2.m2.1b"><apply id="S3.SS4.SSS3.p2.2.m2.1.1.cmml" xref="S3.SS4.SSS3.p2.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS4.SSS3.p2.2.m2.1.1.1.cmml" xref="S3.SS4.SSS3.p2.2.m2.1.1">superscript</csymbol><cn id="S3.SS4.SSS3.p2.2.m2.1.1.2.cmml" type="integer" xref="S3.SS4.SSS3.p2.2.m2.1.1.2">35</cn><compose id="S3.SS4.SSS3.p2.2.m2.1.1.3.cmml" xref="S3.SS4.SSS3.p2.2.m2.1.1.3"></compose></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.SSS3.p2.2.m2.1c">35^{\circ}</annotation><annotation encoding="application/x-llamapun" id="S3.SS4.SSS3.p2.2.m2.1d">35 start_POSTSUPERSCRIPT ∘ end_POSTSUPERSCRIPT</annotation></semantics></math> C to <math alttext="42^{\circ}" class="ltx_Math" display="inline" id="S3.SS4.SSS3.p2.3.m3.1"><semantics id="S3.SS4.SSS3.p2.3.m3.1a"><msup id="S3.SS4.SSS3.p2.3.m3.1.1" xref="S3.SS4.SSS3.p2.3.m3.1.1.cmml"><mn id="S3.SS4.SSS3.p2.3.m3.1.1.2" xref="S3.SS4.SSS3.p2.3.m3.1.1.2.cmml">42</mn><mo id="S3.SS4.SSS3.p2.3.m3.1.1.3" xref="S3.SS4.SSS3.p2.3.m3.1.1.3.cmml">∘</mo></msup><annotation-xml encoding="MathML-Content" id="S3.SS4.SSS3.p2.3.m3.1b"><apply id="S3.SS4.SSS3.p2.3.m3.1.1.cmml" xref="S3.SS4.SSS3.p2.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS4.SSS3.p2.3.m3.1.1.1.cmml" xref="S3.SS4.SSS3.p2.3.m3.1.1">superscript</csymbol><cn id="S3.SS4.SSS3.p2.3.m3.1.1.2.cmml" type="integer" xref="S3.SS4.SSS3.p2.3.m3.1.1.2">42</cn><compose id="S3.SS4.SSS3.p2.3.m3.1.1.3.cmml" xref="S3.SS4.SSS3.p2.3.m3.1.1.3"></compose></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.SSS3.p2.3.m3.1c">42^{\circ}</annotation><annotation encoding="application/x-llamapun" id="S3.SS4.SSS3.p2.3.m3.1d">42 start_POSTSUPERSCRIPT ∘ end_POSTSUPERSCRIPT</annotation></semantics></math> C111Ambient temperature between <math alttext="15^{\circ}" class="ltx_Math" display="inline" id="footnote1.m1.1"><semantics id="footnote1.m1.1b"><msup id="footnote1.m1.1.1" xref="footnote1.m1.1.1.cmml"><mn id="footnote1.m1.1.1.2" xref="footnote1.m1.1.1.2.cmml">15</mn><mo id="footnote1.m1.1.1.3" xref="footnote1.m1.1.1.3.cmml">∘</mo></msup><annotation-xml encoding="MathML-Content" id="footnote1.m1.1c"><apply id="footnote1.m1.1.1.cmml" xref="footnote1.m1.1.1"><csymbol cd="ambiguous" id="footnote1.m1.1.1.1.cmml" xref="footnote1.m1.1.1">superscript</csymbol><cn id="footnote1.m1.1.1.2.cmml" type="integer" xref="footnote1.m1.1.1.2">15</cn><compose id="footnote1.m1.1.1.3.cmml" xref="footnote1.m1.1.1.3"></compose></apply></annotation-xml><annotation encoding="application/x-tex" id="footnote1.m1.1d">15^{\circ}</annotation><annotation encoding="application/x-llamapun" id="footnote1.m1.1e">15 start_POSTSUPERSCRIPT ∘ end_POSTSUPERSCRIPT</annotation></semantics></math> C and <math alttext="40^{\circ}" class="ltx_Math" display="inline" id="footnote1.m2.1"><semantics id="footnote1.m2.1b"><msup id="footnote1.m2.1.1" xref="footnote1.m2.1.1.cmml"><mn id="footnote1.m2.1.1.2" xref="footnote1.m2.1.1.2.cmml">40</mn><mo id="footnote1.m2.1.1.3" xref="footnote1.m2.1.1.3.cmml">∘</mo></msup><annotation-xml encoding="MathML-Content" id="footnote1.m2.1c"><apply id="footnote1.m2.1.1.cmml" xref="footnote1.m2.1.1"><csymbol cd="ambiguous" id="footnote1.m2.1.1.1.cmml" xref="footnote1.m2.1.1">superscript</csymbol><cn id="footnote1.m2.1.1.2.cmml" type="integer" xref="footnote1.m2.1.1.2">40</cn><compose id="footnote1.m2.1.1.3.cmml" xref="footnote1.m2.1.1.3"></compose></apply></annotation-xml><annotation encoding="application/x-tex" id="footnote1.m2.1d">40^{\circ}</annotation><annotation encoding="application/x-llamapun" id="footnote1.m2.1e">40 start_POSTSUPERSCRIPT ∘ end_POSTSUPERSCRIPT</annotation></semantics></math> C., which is ideal for human sensing applications. The total measurement range spans from <math alttext="-20^{\circ}" class="ltx_Math" display="inline" id="S3.SS4.SSS3.p2.4.m4.1"><semantics id="S3.SS4.SSS3.p2.4.m4.1a"><mrow id="S3.SS4.SSS3.p2.4.m4.1.1" xref="S3.SS4.SSS3.p2.4.m4.1.1.cmml"><mo id="S3.SS4.SSS3.p2.4.m4.1.1a" xref="S3.SS4.SSS3.p2.4.m4.1.1.cmml">−</mo><msup id="S3.SS4.SSS3.p2.4.m4.1.1.2" xref="S3.SS4.SSS3.p2.4.m4.1.1.2.cmml"><mn id="S3.SS4.SSS3.p2.4.m4.1.1.2.2" xref="S3.SS4.SSS3.p2.4.m4.1.1.2.2.cmml">20</mn><mo id="S3.SS4.SSS3.p2.4.m4.1.1.2.3" xref="S3.SS4.SSS3.p2.4.m4.1.1.2.3.cmml">∘</mo></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS4.SSS3.p2.4.m4.1b"><apply id="S3.SS4.SSS3.p2.4.m4.1.1.cmml" xref="S3.SS4.SSS3.p2.4.m4.1.1"><minus id="S3.SS4.SSS3.p2.4.m4.1.1.1.cmml" xref="S3.SS4.SSS3.p2.4.m4.1.1"></minus><apply id="S3.SS4.SSS3.p2.4.m4.1.1.2.cmml" xref="S3.SS4.SSS3.p2.4.m4.1.1.2"><csymbol cd="ambiguous" id="S3.SS4.SSS3.p2.4.m4.1.1.2.1.cmml" xref="S3.SS4.SSS3.p2.4.m4.1.1.2">superscript</csymbol><cn id="S3.SS4.SSS3.p2.4.m4.1.1.2.2.cmml" type="integer" xref="S3.SS4.SSS3.p2.4.m4.1.1.2.2">20</cn><compose id="S3.SS4.SSS3.p2.4.m4.1.1.2.3.cmml" xref="S3.SS4.SSS3.p2.4.m4.1.1.2.3"></compose></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.SSS3.p2.4.m4.1c">-20^{\circ}</annotation><annotation encoding="application/x-llamapun" id="S3.SS4.SSS3.p2.4.m4.1d">- 20 start_POSTSUPERSCRIPT ∘ end_POSTSUPERSCRIPT</annotation></semantics></math> C to <math alttext="100^{\circ}" class="ltx_Math" display="inline" id="S3.SS4.SSS3.p2.5.m5.1"><semantics id="S3.SS4.SSS3.p2.5.m5.1a"><msup id="S3.SS4.SSS3.p2.5.m5.1.1" xref="S3.SS4.SSS3.p2.5.m5.1.1.cmml"><mn id="S3.SS4.SSS3.p2.5.m5.1.1.2" xref="S3.SS4.SSS3.p2.5.m5.1.1.2.cmml">100</mn><mo id="S3.SS4.SSS3.p2.5.m5.1.1.3" xref="S3.SS4.SSS3.p2.5.m5.1.1.3.cmml">∘</mo></msup><annotation-xml encoding="MathML-Content" id="S3.SS4.SSS3.p2.5.m5.1b"><apply id="S3.SS4.SSS3.p2.5.m5.1.1.cmml" xref="S3.SS4.SSS3.p2.5.m5.1.1"><csymbol cd="ambiguous" id="S3.SS4.SSS3.p2.5.m5.1.1.1.cmml" xref="S3.SS4.SSS3.p2.5.m5.1.1">superscript</csymbol><cn id="S3.SS4.SSS3.p2.5.m5.1.1.2.cmml" type="integer" xref="S3.SS4.SSS3.p2.5.m5.1.1.2">100</cn><compose id="S3.SS4.SSS3.p2.5.m5.1.1.3.cmml" xref="S3.SS4.SSS3.p2.5.m5.1.1.3"></compose></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.SSS3.p2.5.m5.1c">100^{\circ}</annotation><annotation encoding="application/x-llamapun" id="S3.SS4.SSS3.p2.5.m5.1d">100 start_POSTSUPERSCRIPT ∘ end_POSTSUPERSCRIPT</annotation></semantics></math> C, making it versatile for various conditions. Additionally, the sensor’s configurable sampling rate of up to <math alttext="64" class="ltx_Math" display="inline" id="S3.SS4.SSS3.p2.6.m6.1"><semantics id="S3.SS4.SSS3.p2.6.m6.1a"><mn id="S3.SS4.SSS3.p2.6.m6.1.1" xref="S3.SS4.SSS3.p2.6.m6.1.1.cmml">64</mn><annotation-xml encoding="MathML-Content" id="S3.SS4.SSS3.p2.6.m6.1b"><cn id="S3.SS4.SSS3.p2.6.m6.1.1.cmml" type="integer" xref="S3.SS4.SSS3.p2.6.m6.1.1">64</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.SSS3.p2.6.m6.1c">64</annotation><annotation encoding="application/x-llamapun" id="S3.SS4.SSS3.p2.6.m6.1d">64</annotation></semantics></math> Hz allows it to detect minute and quick temperature fluctuations, providing enhanced precision in continuous monitoring scenarios. </div> </section> <section class="ltx_subsubsection" id="S3.SS4.SSS4"> <h4 class="ltx_title ltx_title_subsubsection"> 3.4.4. Microphones</h4> <figure class="ltx_figure" id="S3.F7"><img alt="Refer to caption" class="ltx_graphics ltx_centering ltx_img_landscape" height="297" id="S3.F7.g1" src="extracted/5905686/Images/in_ear_mic_sample.png" width="598"/> <figcaption class="ltx_caption ltx_centering">Figure 7. Internal microphone data sampled at 48kHz during silence.</figcaption> </figure> <div class="ltx_para" id="S3.SS4.SSS4.p1"> The OmniBuds come equipped with three microphones. Two are facing outward while one is embedded in the speaker chamber and facing the user’s ear canal, as discussed in section <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S3.SS1" title="3.1. Form Factor and Sensor Placement ‣ 3. Hardware Architecture ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">3.1</a>. When selecting microphones for OmniBuds, we carefully balanced the requirements of standard acoustic features like active noise cancellation and transparency mode with the need for high-fidelity sensing performance. Microphones optimised for ANC and Transparency must capture environmental sounds accurately, while sensing applications require them to detect subtle internal body sounds, such as heartbeats. Ensuring that the microphones could excel at both tasks was critical to the design. We selected components that offered a sufficient dynamic range and sensitivity to perform well in both acoustic and sensing contexts, acknowledging the inherent trade-offs in optimising for dual functionality. </div> <div class="ltx_para" id="S3.SS4.SSS4.p2"> Figure <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S3.F7" title="Figure 7 ‣ 3.4.4. Microphones ‣ 3.4. Sensing Subsystem ‣ 3. Hardware Architecture ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">7</a> shows data captured from the OmniBuds’ internal microphone at 48kHz during silence. The signal clearly shows how the microphone is capable of capturing heart sounds which are used for diverse applications, like heart rate measurement and blood pressure estimation <cite class="ltx_cite ltx_citemacro_citep">(Truong et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib37" title="">2022</a>; Butkow et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib8" title="">2023</a>)</cite>. </div> </section> </section> </section> <section class="ltx_section" id="S4"> <h2 class="ltx_title ltx_title_section"> 4. Software Architecture</h2> <figure class="ltx_figure" id="S4.F8"><img alt="Refer to caption" class="ltx_graphics ltx_centering ltx_img_landscape" height="251" id="S4.F8.g1" src="x5.png" width="830"/> <figcaption class="ltx_caption ltx_centering">Figure 8. OmniBuds Ecosystem and SW Overview.</figcaption> </figure> <figure class="ltx_figure" id="S4.F9"><img alt="Refer to caption" class="ltx_graphics ltx_centering ltx_img_landscape" height="360" id="S4.F9.g1" src="x6.png" width="830"/> <figcaption class="ltx_caption ltx_centering">Figure 9. OmniBuds Software Architecture.</figcaption> </figure> <div class="ltx_para" id="S4.p1"> The OmniBuds ecosystem consists primarily of two earbuds with identical hardware, as discussed earlier, and is complemented by a charging case and an optional host device connected via Bluetooth Classic (BT) and/or Bluetooth Low Energy (BLE). Figure <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S4.F8" title="Figure 8 ‣ 4. Software Architecture ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">8</a> provides an overview of the OmniBuds ecosystem. The software architecture is designed to fully leverage this multi-device setup, enabling efficient coordination across devices. </div> <div class="ltx_para" id="S4.p2"> The software architecture of OmniBuds adopts a layered design that allows developers to extend the platform’s functionality while respecting the hardware’s constraints. As shown in Figure <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S4.F9" title="Figure 9 ‣ 4. Software Architecture ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">9</a>, the architecture consists of four key layers: </div> <div class="ltx_para" id="S4.p3"> <ul class="ltx_itemize" id="S4.I1"> <li class="ltx_item" id="S4.I1.i1" style="list-style-type:none;"> • <div class="ltx_para" id="S4.I1.i1.p1"> Hardware Abstraction Layer (HAL): Encapsulates device drivers and utility libraries, ensuring portability across different hardware configurations. </div> </li> <li class="ltx_item" id="S4.I1.i2" style="list-style-type:none;"> • <div class="ltx_para" id="S4.I1.i2.p1"> Middleware Layer: Provides essential APIs for applications to communicate with and efficiently use hardware resources. It ensures independent applications can run without conflicts by managing access to hardware in a modular manner. </div> </li> <li class="ltx_item" id="S4.I1.i3" style="list-style-type:none;"> • <div class="ltx_para" id="S4.I1.i3.p1"> Applications Layer: Hosts algorithms that implement functionalities such as vital signs monitoring, leveraging localised computing to optimise performance and conserve power. </div> </li> <li class="ltx_item" id="S4.I1.i4" style="list-style-type:none;"> • <div class="ltx_para" id="S4.I1.i4.p1"> Services Layer: Provides system-wide services such as file management, persistent storage, and task scheduling. The platform runs FreeRTOS which is packaged as part of the services. The applications are free to use FreeRTOS primitives for their own purposes, but should use the OmniBuds middleware APIs for the functionalities provided by the platform and to access the hardware infrastructure. </div> </li> </ul> </div> <div class="ltx_para" id="S4.p4"> This layered design ensures flexibility, scalability, and ease of development while maintaining tight integration between software and hardware. The following sections focus on three critical subsystems: communication, sense and compute, and load balancing, each playing a vital role in the platform’s efficient operation. </div> <section class="ltx_subsection" id="S4.SS1"> <h3 class="ltx_title ltx_title_subsection"> 4.1. Communication Subsystem</h3> <div class="ltx_para" id="S4.SS1.p1"> The Communication Subsystem in OmniBuds manages data exchange within the platform and with external devices. It consists of two main components: Inter-Module Communication (IMC) for internal communication between software modules and the Communication Interface, which uses the BLE protocol to connect with external peripherals (Figure <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S4.F9" title="Figure 9 ‣ 4. Software Architecture ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">9</a>). </div> <section class="ltx_subsubsection" id="S4.SS1.SSS1"> <h4 class="ltx_title ltx_title_subsubsection"> 4.1.1. Inter-Module Communication (IMC)</h4> <div class="ltx_para" id="S4.SS1.SSS1.p1"> IMC facilitates seamless and transparent communication between software modules, both locally and across devices in the OmniBuds ecosystem. This approach ensures that software modules can interact without needing to manage the complexities of local versus remote communication. </div> <div class="ltx_para" id="S4.SS1.SSS1.p2"> This system is based on the observer pattern <cite class="ltx_cite ltx_citemacro_citep">(Gamma, <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib20" title="">1995</a>)</cite>, where components register for updates from extendable triggers. When a trigger is activated, the registered modules receive notifications with the relevant data. IMC messages are asynchronous and can carry up to 64 bytes of data. </div> <div class="ltx_para" id="S4.SS1.SSS1.p3"> The IMC API offers flexibility in message dissemination, allowing modules to specify if a message is sent to the local device, a peer device (i.e. the other earbud), other devices in the OmniBuds ecosystem, or all of the above. The underlying IMC logic handles message queuing efficiently, even when devices use different physical layer connections. The APIs for sending and receiving IMC messages are shown below: </div> <div class="ltx_para" id="S4.SS1.SSS1.p4"> <div class="ltx_listing ltx_lst_language_C ltx_lstlisting ltx_framed ltx_framed_rectangle ltx_listing" id="S4.SS1.SSS1.p4.1"> <div class="ltx_listing_data"><a download="" href="data:text/plain;base64,dm9pZCBvYl9JTUNTZW5kTWVzc2FnZSh1aW50OF90IHRyaWdnZXJJRCwgdWludDhfdCAqZGF0YUJ1ZmZlciwgdWludDhfdCBkYXRhTGVuZ3RoLCB1aW50OF90IGRlc3RpbmF0aW9uKQoKaW50IG9iX0lNQ1JlZ2lzdGVyTWVzc2FnZUNhbGxiYWNrKHVpbnQ4X3QgdHJpZ2dlcklELCB0cmlnZ2VyRXZ0Q0JfdCAqdHJpZ2dlckNCKQ==">⬇</a></div> <div class="ltx_listingline" id="lstnumberx1"> void ob_IMCSendMessage(uint8_t triggerID, uint8_t *dataBuffer, uint8_t dataLength, uint8_t destination) </div> <div class="ltx_listingline" id="lstnumberx2"> </div> <div class="ltx_listingline" id="lstnumberx3"> int ob_IMCRegisterMessageCallback(uint8_t triggerID, triggerEvtCB_t *triggerCB) </div> </div> </div> </section> <section class="ltx_subsubsection" id="S4.SS1.SSS2"> <h4 class="ltx_title ltx_title_subsubsection"> 4.1.2. Communication Interface</h4> <div class="ltx_para" id="S4.SS1.SSS2.p1"> The Communication Interface module in OmniBuds manages data exchange with external devices via Bluetooth Low Energy (BLE) and handles audio playback and phone calls through Bluetooth Classic. </div> <div class="ltx_para" id="S4.SS1.SSS2.p2"> To external devices, the two OmniBuds earbuds appear as a single device, with the primary earbud managing the connection and forwarding messages to the secondary earbud. This architecture simplifies communication while maintaining seamless interaction between both earbuds. Details on how this supports load balancing and battery optimisation are discussed in Section <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S4.SS3" title="4.3. Load Balancing Subsystem ‣ 4. Software Architecture ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">4.3</a>. </div> <div class="ltx_para" id="S4.SS1.SSS2.p3"> For data exchange, OmniBuds introduces addressable peripherals, which can be either physical sensors (e.g., IMU or PPG) or virtual peripherals (e.g., heart rate derived from PPG). These allow third-party devices to send commands, collect data, and interact with specific peripherals via a consistent interface. BLE communication is encrypted to ensure data privacy and security. </div> <div class="ltx_para" id="S4.SS1.SSS2.p4"> BLE communication is based on three message types: <ul class="ltx_itemize" id="S4.I2"> <li class="ltx_item" id="S4.I2.i1" style="list-style-type:none;"> • <div class="ltx_para" id="S4.I2.i1.p1"> Data messages: For large or continuous data streams. </div> </li> <li class="ltx_item" id="S4.I2.i2" style="list-style-type:none;"> • <div class="ltx_para" id="S4.I2.i2.p1"> Event messages: For handling sporadic peripheral triggers. </div> </li> <li class="ltx_item" id="S4.I2.i3" style="list-style-type:none;"> • <div class="ltx_para" id="S4.I2.i3.p1"> Configuration messages: For configuring peripheral settings such as sampling rate or power mode (using different endpoints associated with each peripheral). </div> </li> </ul> </div> <div class="ltx_para" id="S4.SS1.SSS2.p5"> The main APIs for managing BLE communication are as follows: <div class="ltx_listing ltx_lst_language_C ltx_lstlisting ltx_framed ltx_framed_rectangle ltx_listing" id="S4.SS1.SSS2.p5.2"> <div class="ltx_listing_data"><a download="" href="data:text/plain;base64,Ly8gUmVnaXN0ZXIgY2FsbGJhY2sgdG8gcmVjZWl2ZSBkYXRhL2V2ZW50L2NvbmZpZyBtZXNzYWdlcwppbnQgb2JfcmVnaXN0ZXJQZXJpcGhlcmFsQ2FsbGJhY2sob2JfcGVyaXBoSURfdCBwZXJpcGhlcmFsRCwgcnhDb21tRXZ0Q0IgKmNhbGxiYWMpOwoKLy8gU2VuZCBhIGRhdGEvZXZlbnQgbWVzc2FnZQp2b2lkIG9iX3NlbmREYXRhTWVzc2FnZShvYl9wZXJpcGhJRF90IHBlcmlwaGVyYWxELCB1aW50OF90ICpkYXRhX2J1ZiwgdWludDE2X3QgZGF0YV9sZW4pOwp2b2lkIG9iX3NlbmRFdmVudE1lc3NhZ2Uob2JfcGVyaXBoSURfdCBwZXJpcGhlcmFsRCwgdWludDhfdCAqZGF0YV9idWYsIHVpbnQxNl90IGRhdGFfbGVuKTsKCi8vIFNlbmQgYSBjb25maWd1cmF0aW9uIG1lc3NhZ2UKdm9pZCBvYl9zZW5kQ29uZmlnTWVzc2FnZShvYl9wZXJpcGhJRF90IHBlcmlwaGVyYWxELCB1aW50OF90IGNvbmZpZ0VuZHBvaW50LCB1aW50OF90ICpkYXRhX2J1ZiwgdWludDE2X3QgZGF0YV9sZW4pOwoKLy8gU2VuZCBhIGNvbmZpZ3VyYXRpb24gbWVzc2FnZSByZXNwb25zZQp2b2lkIG9iX3NlbmRDb25maWdNZXNzYWdlUmVzcG9uc2Uob2JfcGVyaXBoSURfdCBwZXJpcGhlcmFsRCwgdWludDhfdCBjb25maWdFbmRwb2ludCwgb2JfbXNnRXJyb3JDb2RlX3QgZXJyQ29kZSwgdWludDhfdCAqZGF0YV9idWYsIHVpbnQxNl90IGRhdGFfbGVuKTs=">⬇</a></div> <div class="ltx_listingline" id="lstnumberx4"> // Register callback to receive data/event/config messages </div> <div class="ltx_listingline" id="lstnumberx5"> int ob_registerPeripheralCallback(ob_periphID_t peripheralD, rxCommEvtCB *callbac); </div> <div class="ltx_listingline" id="lstnumberx6"> </div> <div class="ltx_listingline" id="lstnumberx7"> // Send a data/event message </div> <div class="ltx_listingline" id="lstnumberx8"> void ob_sendDataMessage(ob_periphID_t peripheralD, uint8_t *data_buf, uint16_t data_len); </div> <div class="ltx_listingline" id="lstnumberx9"> void ob_sendEventMessage(ob_periphID_t peripheralD, uint8_t *data_buf, uint16_t data_len); </div> <div class="ltx_listingline" id="lstnumberx10"> </div> <div class="ltx_listingline" id="lstnumberx11"> // Send a configuration message </div> <div class="ltx_listingline" id="lstnumberx12"> void ob_sendConfigMessage(ob_periphID_t peripheralD, uint8_t configEndpoint, uint8_t *data_buf, uint16_t data_len); </div> <div class="ltx_listingline" id="lstnumberx13"> </div> <div class="ltx_listingline" id="lstnumberx14"> // Send a configuration message response </div> <div class="ltx_listingline" id="lstnumberx15"> void ob_sendConfigMessageResponse(ob_periphID_t peripheralD, uint8_t configEndpoint, ob_msgErrorCode_t errCode, uint8_t *data_buf, uint16_t data_len); </div> </div> </div> <div class="ltx_para" id="S4.SS1.SSS2.p6"> OmniBuds also manages audio playback and phone calls via Bluetooth Classic, using the Hands-Free Profile (HFP) for calls and Audio-Video Remote Control Profile (A2DP) for playback, coordinated by the communication subsystem and wireless chipset. </div> </section> </section> <section class="ltx_subsection" id="S4.SS2"> <h3 class="ltx_title ltx_title_subsection"> 4.2. Sense and Compute Subsystem</h3> <div class="ltx_para" id="S4.SS2.p1"> The Sense and Compute Subsystem in OmniBuds provides applications with primitives to manage the platform’s computational units and sensors efficiently. Three critical modules define this subsystem: the Sensor Distribution Module, which manages sensor data access for multiple applications; the Machine Learning Engine, responsible for executing models on the CNN accelerator; and the Audio Manager, which oversees audio pipelines within the dedicated Audio DSP. The following sections provide an overview of these key modules. </div> <section class="ltx_subsubsection" id="S4.SS2.SSS1"> <h4 class="ltx_title ltx_title_subsubsection"> 4.2.1. Sensor Distribution Module</h4> <div class="ltx_para" id="S4.SS2.SSS1.p1"> The Sensor Distribution Module simplifies access to sensor data for multiple applications, particularly for sensors like the IMU, PPG and temperature. It allows application developers to easily integrate sensor data without managing conflicts or sensor resource issues. </div> <div class="ltx_para" id="S4.SS2.SSS1.p2"> Different Business Logic modules are provided for the available sensors (Figure <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S4.F9" title="Figure 9 ‣ 4. Software Architecture ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">9</a>) which are used as singleton <cite class="ltx_cite ltx_citemacro_citep">(Gamma, <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib20" title="">1995</a>)</cite> from multiple applications. Applications register with the module by specifying their requirements for the sensor data, such as sampling rate, window length, and other parameters. The module then configures the sensor accordingly and efficiently delivers the requested data to each application. To further enhance efficiency, the module leverages on-chip computational features, such as the BioHub and the machine learning core in the IMU, to process data locally when necessary. This reduces the computational load on the main MCU, conserves power, and ensures that sensor data is processed and delivered in a timely manner. </div> </section> <section class="ltx_subsubsection" id="S4.SS2.SSS2"> <h4 class="ltx_title ltx_title_subsubsection"> 4.2.2. Machine Learning Engine</h4> <div class="ltx_para" id="S4.SS2.SSS2.p1"> The Machine Learning Engine manages the execution of machine learning models on the CNN accelerator, ensuring efficient on-device inference. The use of this module involves a complete pipeline, from offline model training and synthesis to deployment and execution, ensuring compatibility with the accelerator’s architecture. </div> <div class="ltx_para" id="S4.SS2.SSS2.p2"> Models are trained and optimised offline before being transferred to the OmniBuds via BLE. The engine then loads and executes the model on the CNN accelerator, making inference results available to other applications. To conserve power, the engine dynamically configures the accelerator, shutting down unused hardware sections during inference. </div> <div class="ltx_para" id="S4.SS2.SSS2.p3"> Looking ahead, the Machine Learning Engine is designed to support future capabilities such as splitting inference tasks across both earbuds, enabling distributed computation, and running multiple models concurrently. These features will extend the flexibility of the system, enhancing performance and energy efficiency. </div> </section> <section class="ltx_subsubsection" id="S4.SS2.SSS3"> <h4 class="ltx_title ltx_title_subsubsection"> 4.2.3. Audio Manager</h4> <figure class="ltx_figure" id="S4.F10"><img alt="Refer to caption" class="ltx_graphics ltx_centering ltx_img_landscape" height="451" id="S4.F10.g1" src="x7.png" width="830"/> <figcaption class="ltx_caption ltx_centering">Figure 10. Fast DSP algorithms for ANC/Pass-through and music playback and calls.</figcaption> </figure> <figure class="ltx_figure" id="S4.F11"><img alt="Refer to caption" class="ltx_graphics ltx_centering ltx_img_landscape" height="456" id="S4.F11.g1" src="x8.png" width="830"/> <figcaption class="ltx_caption ltx_centering">Figure 11. Slow DSP algorithms to support sensing applications like spoken keyword spotting and multi-modal blood pressure estimation.</figcaption> </figure> <div class="ltx_para" id="S4.SS2.SSS3.p1"> The Audio Manager handles the audio processing pipelines on the dedicated audio DSP, divided into time-critical and non-time-critical categories. </div> <section class="ltx_paragraph" id="S4.SS2.SSS3.Px1"> <h5 class="ltx_title ltx_title_paragraph">Time-Critical Pipelines</h5> <div class="ltx_para" id="S4.SS2.SSS3.Px1.p1"> Time-critical pipelines include processes such as Active Noise Cancellation (ANC) and pass-through functionality. These pipelines run on the Fast-DSP core, which is optimised for low-latency processing, operating at a high sampling frequency of <math alttext="F_{s}=384" class="ltx_Math" display="inline" id="S4.SS2.SSS3.Px1.p1.1.m1.1"><semantics id="S4.SS2.SSS3.Px1.p1.1.m1.1a"><mrow id="S4.SS2.SSS3.Px1.p1.1.m1.1.1" xref="S4.SS2.SSS3.Px1.p1.1.m1.1.1.cmml"><msub id="S4.SS2.SSS3.Px1.p1.1.m1.1.1.2" xref="S4.SS2.SSS3.Px1.p1.1.m1.1.1.2.cmml"><mi id="S4.SS2.SSS3.Px1.p1.1.m1.1.1.2.2" xref="S4.SS2.SSS3.Px1.p1.1.m1.1.1.2.2.cmml">F</mi><mi id="S4.SS2.SSS3.Px1.p1.1.m1.1.1.2.3" xref="S4.SS2.SSS3.Px1.p1.1.m1.1.1.2.3.cmml">s</mi></msub><mo id="S4.SS2.SSS3.Px1.p1.1.m1.1.1.1" xref="S4.SS2.SSS3.Px1.p1.1.m1.1.1.1.cmml">=</mo><mn id="S4.SS2.SSS3.Px1.p1.1.m1.1.1.3" xref="S4.SS2.SSS3.Px1.p1.1.m1.1.1.3.cmml">384</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS3.Px1.p1.1.m1.1b"><apply id="S4.SS2.SSS3.Px1.p1.1.m1.1.1.cmml" xref="S4.SS2.SSS3.Px1.p1.1.m1.1.1"><eq id="S4.SS2.SSS3.Px1.p1.1.m1.1.1.1.cmml" xref="S4.SS2.SSS3.Px1.p1.1.m1.1.1.1"></eq><apply id="S4.SS2.SSS3.Px1.p1.1.m1.1.1.2.cmml" xref="S4.SS2.SSS3.Px1.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="S4.SS2.SSS3.Px1.p1.1.m1.1.1.2.1.cmml" xref="S4.SS2.SSS3.Px1.p1.1.m1.1.1.2">subscript</csymbol><ci id="S4.SS2.SSS3.Px1.p1.1.m1.1.1.2.2.cmml" xref="S4.SS2.SSS3.Px1.p1.1.m1.1.1.2.2">𝐹</ci><ci id="S4.SS2.SSS3.Px1.p1.1.m1.1.1.2.3.cmml" xref="S4.SS2.SSS3.Px1.p1.1.m1.1.1.2.3">𝑠</ci></apply><cn id="S4.SS2.SSS3.Px1.p1.1.m1.1.1.3.cmml" type="integer" xref="S4.SS2.SSS3.Px1.p1.1.m1.1.1.3">384</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS3.Px1.p1.1.m1.1c">F_{s}=384</annotation><annotation encoding="application/x-llamapun" id="S4.SS2.SSS3.Px1.p1.1.m1.1d">italic_F start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT = 384</annotation></semantics></math> KHz. As shown in Figure <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S4.F10" title="Figure 10 ‣ 4.2.3. Audio Manager ‣ 4.2. Sense and Compute Subsystem ‣ 4. Software Architecture ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">10</a>, the Fast-DSP processes both ANC and pass-through in a shared structure, with differences in the parameters loaded into the feed-forward and feedback filter banks. For safety, when the pass-through functionality is enabled, the limiter ensures that no sound pressure levels exceed 85 dB-SPL. The Fast-DSP also manages music playback and phone calls, which are processed in a similar pipeline to ANC and pass-through but differ in the filter parameters used. In both music and phone calls, audio content coming from the Bluetooth SOC needs to be routed to the earbud’s speaker to be played back. </div> <div class="ltx_para" id="S4.SS2.SSS3.Px1.p2"> While ANC and pass-through are mutually exclusive, they can coexist with music playback or calls, where the outputs from both pipelines are mixed before being sent to the speaker. In this dual-processing configuration, different filter parameters are loaded to compensate for the effect of ANC or pass-through on the ear canal’s acoustics. </div> </section> <section class="ltx_paragraph" id="S4.SS2.SSS3.Px2"> <h5 class="ltx_title ltx_title_paragraph">Non-Time-Critical Pipelines</h5> <div class="ltx_para" id="S4.SS2.SSS3.Px2.p1"> Non-time-critical pipelines (see Figure <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S4.F11" title="Figure 11 ‣ 4.2.3. Audio Manager ‣ 4.2. Sense and Compute Subsystem ‣ 4. Software Architecture ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">11</a>) handle tasks that are less latency-sensitive, such as keyword spotting, blood pressure monitoring, and heart rate detection, running on the Slow-DSP core at a lower sampling frequency of <math alttext="F_{s}=48" class="ltx_Math" display="inline" id="S4.SS2.SSS3.Px2.p1.1.m1.1"><semantics id="S4.SS2.SSS3.Px2.p1.1.m1.1a"><mrow id="S4.SS2.SSS3.Px2.p1.1.m1.1.1" xref="S4.SS2.SSS3.Px2.p1.1.m1.1.1.cmml"><msub id="S4.SS2.SSS3.Px2.p1.1.m1.1.1.2" xref="S4.SS2.SSS3.Px2.p1.1.m1.1.1.2.cmml"><mi id="S4.SS2.SSS3.Px2.p1.1.m1.1.1.2.2" xref="S4.SS2.SSS3.Px2.p1.1.m1.1.1.2.2.cmml">F</mi><mi id="S4.SS2.SSS3.Px2.p1.1.m1.1.1.2.3" xref="S4.SS2.SSS3.Px2.p1.1.m1.1.1.2.3.cmml">s</mi></msub><mo id="S4.SS2.SSS3.Px2.p1.1.m1.1.1.1" xref="S4.SS2.SSS3.Px2.p1.1.m1.1.1.1.cmml">=</mo><mn id="S4.SS2.SSS3.Px2.p1.1.m1.1.1.3" xref="S4.SS2.SSS3.Px2.p1.1.m1.1.1.3.cmml">48</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS3.Px2.p1.1.m1.1b"><apply id="S4.SS2.SSS3.Px2.p1.1.m1.1.1.cmml" xref="S4.SS2.SSS3.Px2.p1.1.m1.1.1"><eq id="S4.SS2.SSS3.Px2.p1.1.m1.1.1.1.cmml" xref="S4.SS2.SSS3.Px2.p1.1.m1.1.1.1"></eq><apply id="S4.SS2.SSS3.Px2.p1.1.m1.1.1.2.cmml" xref="S4.SS2.SSS3.Px2.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="S4.SS2.SSS3.Px2.p1.1.m1.1.1.2.1.cmml" xref="S4.SS2.SSS3.Px2.p1.1.m1.1.1.2">subscript</csymbol><ci id="S4.SS2.SSS3.Px2.p1.1.m1.1.1.2.2.cmml" xref="S4.SS2.SSS3.Px2.p1.1.m1.1.1.2.2">𝐹</ci><ci id="S4.SS2.SSS3.Px2.p1.1.m1.1.1.2.3.cmml" xref="S4.SS2.SSS3.Px2.p1.1.m1.1.1.2.3">𝑠</ci></apply><cn id="S4.SS2.SSS3.Px2.p1.1.m1.1.1.3.cmml" type="integer" xref="S4.SS2.SSS3.Px2.p1.1.m1.1.1.3">48</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS3.Px2.p1.1.m1.1c">F_{s}=48</annotation><annotation encoding="application/x-llamapun" id="S4.SS2.SSS3.Px2.p1.1.m1.1d">italic_F start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT = 48</annotation></semantics></math> KHz. To achieve this, MEMS microphone signals—originally sampled at higher frequencies—are decimated. </div> <div class="ltx_para" id="S4.SS2.SSS3.Px2.p2"> The Slow-DSP offloads audio-related tasks, such as sensor sampling, filtering, and signal pre-conditioning, reducing the load on the MCU. For tasks like keyword spotting or blood pressure monitoring, the DSP routes MEMS microphone signals to the microcontroller via the <math alttext="I2S_{0}" class="ltx_Math" display="inline" id="S4.SS2.SSS3.Px2.p2.1.m1.1"><semantics id="S4.SS2.SSS3.Px2.p2.1.m1.1a"><mrow id="S4.SS2.SSS3.Px2.p2.1.m1.1.1" xref="S4.SS2.SSS3.Px2.p2.1.m1.1.1.cmml"><mi id="S4.SS2.SSS3.Px2.p2.1.m1.1.1.2" xref="S4.SS2.SSS3.Px2.p2.1.m1.1.1.2.cmml">I</mi><mo id="S4.SS2.SSS3.Px2.p2.1.m1.1.1.1" xref="S4.SS2.SSS3.Px2.p2.1.m1.1.1.1.cmml">⁢</mo><mn id="S4.SS2.SSS3.Px2.p2.1.m1.1.1.3" xref="S4.SS2.SSS3.Px2.p2.1.m1.1.1.3.cmml">2</mn><mo id="S4.SS2.SSS3.Px2.p2.1.m1.1.1.1a" xref="S4.SS2.SSS3.Px2.p2.1.m1.1.1.1.cmml">⁢</mo><msub id="S4.SS2.SSS3.Px2.p2.1.m1.1.1.4" xref="S4.SS2.SSS3.Px2.p2.1.m1.1.1.4.cmml"><mi id="S4.SS2.SSS3.Px2.p2.1.m1.1.1.4.2" xref="S4.SS2.SSS3.Px2.p2.1.m1.1.1.4.2.cmml">S</mi><mn id="S4.SS2.SSS3.Px2.p2.1.m1.1.1.4.3" xref="S4.SS2.SSS3.Px2.p2.1.m1.1.1.4.3.cmml">0</mn></msub></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS3.Px2.p2.1.m1.1b"><apply id="S4.SS2.SSS3.Px2.p2.1.m1.1.1.cmml" xref="S4.SS2.SSS3.Px2.p2.1.m1.1.1"><times id="S4.SS2.SSS3.Px2.p2.1.m1.1.1.1.cmml" xref="S4.SS2.SSS3.Px2.p2.1.m1.1.1.1"></times><ci id="S4.SS2.SSS3.Px2.p2.1.m1.1.1.2.cmml" xref="S4.SS2.SSS3.Px2.p2.1.m1.1.1.2">𝐼</ci><cn id="S4.SS2.SSS3.Px2.p2.1.m1.1.1.3.cmml" type="integer" xref="S4.SS2.SSS3.Px2.p2.1.m1.1.1.3">2</cn><apply id="S4.SS2.SSS3.Px2.p2.1.m1.1.1.4.cmml" xref="S4.SS2.SSS3.Px2.p2.1.m1.1.1.4"><csymbol cd="ambiguous" id="S4.SS2.SSS3.Px2.p2.1.m1.1.1.4.1.cmml" xref="S4.SS2.SSS3.Px2.p2.1.m1.1.1.4">subscript</csymbol><ci id="S4.SS2.SSS3.Px2.p2.1.m1.1.1.4.2.cmml" xref="S4.SS2.SSS3.Px2.p2.1.m1.1.1.4.2">𝑆</ci><cn id="S4.SS2.SSS3.Px2.p2.1.m1.1.1.4.3.cmml" type="integer" xref="S4.SS2.SSS3.Px2.p2.1.m1.1.1.4.3">0</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS3.Px2.p2.1.m1.1c">I2S_{0}</annotation><annotation encoding="application/x-llamapun" id="S4.SS2.SSS3.Px2.p2.1.m1.1d">italic_I 2 italic_S start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT</annotation></semantics></math> bus, where further processing or inference (using the CNN accelerator) takes place. During calls, the DSP similarly routes MEMS microphone signals to the Bluetooth SoC, ensuring efficient transmission to the host device while the MCU remains focused on higher-level application logic. </div> </section> </section> </section> <section class="ltx_subsection" id="S4.SS3"> <h3 class="ltx_title ltx_title_subsection"> 4.3. Load Balancing Subsystem</h3> <div class="ltx_para" id="S4.SS3.p1"> OmniBuds prioritise efficient battery management and balanced system performance given the limited battery capacity. The Load Balancing Subsystem ensures that both communication and computational tasks are distributed optimally across the two earbuds, enhancing battery life. </div> <div class="ltx_para" id="S4.SS3.p2"> In terms of communication, OmniBuds present themselves as a single device to external devices (Section <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S4.SS1.SSS2" title="4.1.2. Communication Interface ‣ 4.1. Communication Subsystem ‣ 4. Software Architecture ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">4.1.2</a>). The primary earbud maintains the connection with the external device, routing any necessary information to and from the secondary earbud. To prevent one earbud from being overly burdened, the system periodically alternates the roles of primary and secondary between the two earbuds, balancing the communication load and distributing battery usage evenly. </div> <div class="ltx_para" id="S4.SS3.p3"> For computational load balancing, OmniBuds include a dedicated Load Balancing Module. This module autonomously determines, at runtime, which earbud should execute a given task. The Load Balancer operates as a system-wide component, running on both earbuds and maintaining a shared database of peripherals and their states. When a peripheral is enabled, the Load Balancer determines its operational state based on the active policies, ensuring that tasks are executed efficiently across the earbuds. </div> </section> </section> <section class="ltx_section" id="S5"> <h2 class="ltx_title ltx_title_section"> 5. Smartphone Application</h2> <figure class="ltx_figure" id="S5.F12"><img alt="Refer to caption" class="ltx_graphics ltx_centering ltx_img_portrait" height="519" id="S5.F12.g1" src="extracted/5905686/Images/ob_app_dashboard_crop_transparent.png" width="299"/> <figcaption class="ltx_caption ltx_centering">Figure 12. Dashboard of the OmniBuds companion app.</figcaption> </figure> <div class="ltx_para" id="S5.p1"> The OmniBuds come with a companion app for both iOS and Android, providing real-time monitoring and visualisation of physiological data. Through a dashboard (Figure <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#S5.F12" title="Figure 12 ‣ 5. Smartphone Application ‣ OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning">12</a>), users can view six vital signs: heart rate, heart rate variability, respiratory rate, blood oxygen saturation, body temperature, and blood pressure. Historical data is accessible with a simple tap, enabling health tracking over daily, monthly, and yearly intervals. The app also allows users to control acoustic settings, switch between transparency and noise-cancelling modes, and initiate blood pressure measurements directly from the dashboard. </div> <div class="ltx_para" id="S5.p2"> For advanced users and developers, the app includes a Developer Mode, which allows the collection of raw, unprocessed sensor data. The data can be saved as .csv files for further analysis, offering invaluable opportunities for experimentation and research. Additionally, the Developer Mode enables modifications to parameters such as sensor sampling rates, LED current for the PPG sensor, and the angular rate for the accelerometer, among others, as well as the choice of which sensor to enable/disable. In addition, when data is being recorded, users can visualise real-time plots of raw sensor data from a dedicated Plot View. This allows for immediate feedback on parameter fine-tuning. </div> </section> <section class="ltx_section" id="S6"> <h2 class="ltx_title ltx_title_section"> 6. Applications of OmniBuds</h2> <div class="ltx_para" id="S6.p1"> OmniBuds’ unique combination of sensors, computational capabilities, and compact form factor opens the door to a wide range of applications across multiple fields. While some of these applications have already been implemented, others remain potential future use cases. The following examples, although not exhaustive, showcase the versatility of OmniBuds, highlighting how the platform’s advanced features can be leveraged for innovative research and practical solutions in earable computing. </div> <section class="ltx_subsection" id="S6.SS1"> <h3 class="ltx_title ltx_title_subsection"> 6.1. Existing Applications</h3> <section class="ltx_subsubsection" id="S6.SS1.SSS1"> <h4 class="ltx_title ltx_title_subsubsection"> 6.1.1. Vital Signs Monitoring</h4> <div class="ltx_para" id="S6.SS1.SSS1.p1"> The advanced sensing capabilities of OmniBuds, coupled with on-device processing and efficient use of computational resources, make it a versatile platform for continuous vital signs monitoring. By integrating multiple sensor modalities in a compact form factor, OmniBuds enable precise, real-time health tracking, supporting a wide range of research and healthcare applications. Out-of-the-box OmniBuds can monitor heart rate (HR), heart rate variability (HRV), respiration rate (RR), blood oxygen saturation (SpO2), blood pressure, and skin temperature. While skin temperature is measured directly, the other vitals are computed on-device, utilising either the main processor or the dedicated BioHub. </div> <div class="ltx_para" id="S6.SS1.SSS1.p2"> Heart Rate and Heart Rate Variability OmniBuds extract HR and HRV using the PPG signal, where HR is determined from peaks in the signal, and HRV is derived from inter-beat intervals. These computations occur on the BioHub, which also mitigates motion artefacts by combining PPG data with IMU readings. </div> <div class="ltx_para" id="S6.SS1.SSS1.p3"> Blood Oxygen Saturation Blood oxygen saturation (SpO2) is calculated by analysing the ratio of the pulsatile and non-pulsatile components of red and infrared PPG signals. This computation is also handled by the BioHub, ensuring efficient processing while freeing the main MCU for other tasks. </div> <div class="ltx_para" id="S6.SS1.SSS1.p4"> Respiration Rate OmniBuds estimate RR through PPG-based respiratory-induced intensity variation (RIIV). The signal is filtered to remove cardiac frequencies, and FFT is applied to determine the respiratory rate <cite class="ltx_cite ltx_citemacro_citep">(Romero et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib35" title="">2024</a>)</cite>. This process is performed by the main MCU using the PPG raw data through the Sensor Distribution module which manages the access to BioHub and PPG for both processed data (HR, HRV and SpO2) and raw data. </div> <div class="ltx_para" id="S6.SS1.SSS1.p5"> Blood Pressure OmniBuds support cuff-less blood pressure measurement using a multi-modal technique that captures the time difference between the S1 heart sound (detected by the in-ear microphone) and the PPG upstroke, known as vascular transit time (VTT) <cite class="ltx_cite ltx_citemacro_citep">(Truong et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib37" title="">2022</a>)</cite>. This, combined with ejection time (ET), enables systolic and diastolic blood pressure estimation through a personalised model. Future extensions could further enhance accuracy by leveraging OmniBuds’ symmetrical hardware for multi-location sensing <cite class="ltx_cite ltx_citemacro_citep">(Balaji et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib6" title="">2023</a>)</cite>. </div> </section> <section class="ltx_subsubsection" id="S6.SS1.SSS2"> <h4 class="ltx_title ltx_title_subsubsection"> 6.1.2. Multi-modal Contextual Recognition</h4> <div class="ltx_para" id="S6.SS1.SSS2.p1"> In addition to monitoring all five vital signs and their derivatives, the sensor suite in OmniBuds enables the detection of typical motion-based contexts commonly tracked by earable devices. These include physical activity <cite class="ltx_cite ltx_citemacro_citep">(Kawsar et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib25" title="">2018b</a>; Ma et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib28" title="">2021</a>)</cite>, head tracking <cite class="ltx_cite ltx_citemacro_citep">(Ferlini et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib17" title="">2019</a>)</cite>, head gestures <cite class="ltx_cite ltx_citemacro_citep">(Ma et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib28" title="">2021</a>)</cite>, and facial expressions <cite class="ltx_cite ltx_citemacro_citep">(Montanari et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib31" title="">2023</a>; Lee et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib27" title="">2019</a>)</cite>. While primarily driven by the integrated 9-axis IMU, more complex activities such as dietary monitoring <cite class="ltx_cite ltx_citemacro_citep">(Kawsar et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib25" title="">2018b</a>)</cite>, energy expenditure <cite class="ltx_cite ltx_citemacro_citep">(Gashi et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib21" title="">2022</a>)</cite>, and mental fatigue assessment <cite class="ltx_cite ltx_citemacro_citep">(Kalanadhabhatta et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib23" title="">2021</a>)</cite> require combining data from multiple sensors, such as PPG and microphones. </div> <div class="ltx_para" id="S6.SS1.SSS2.p2"> What sets OmniBuds apart from previous platforms is its capability to run these diverse pipelines directly on-device, selecting the hardware component that is most suited to the task. For instance, low-power motion-based pipelines, such as activity recognition (e.g., walking, running) and head gestures detection (e.g., nodding, shaking), are efficiently processed directly on the IMU, minimising power consumption and reducing the load on the main MCU. Conversely, more complex tasks, such as dietary monitoring and fatigue estimation, can leverage the CNN accelerator to execute larger machine learning models, enabling the device to handle higher-capacity computations on-device with reduced latency <cite class="ltx_cite ltx_citemacro_citep">(Ardis and Muchsel, <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib4" title="">[n. d.]</a>; Moss et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib32" title="">2022</a>)</cite>. </div> </section> </section> <section class="ltx_subsection" id="S6.SS2"> <h3 class="ltx_title ltx_title_subsection"> 6.2. Future Applications</h3> <section class="ltx_subsubsection" id="S6.SS2.SSS1"> <h4 class="ltx_title ltx_title_subsubsection"> 6.2.1. Emotion Recognition and Augmented Feedback</h4> <div class="ltx_para" id="S6.SS2.SSS1.p1"> OmniBuds could potentially integrate emotion recognition based on physiological changes, such as heart rate, body temperature, voice tone and facial expressions. For example, the device could monitor stress, anxiety and fatigue and adjust audio feedback accordingly, offering calming sounds or adjusting music tempo to match the user’s emotional state <cite class="ltx_cite ltx_citemacro_citep">(Butkow et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.04775v1#bib.bib9" title="">2024</a>)</cite>. This speculative application leverages multi-modal sensor data available on OmniBuds and their privacy-preserving design. </div> </section> <section class="ltx_subsubsection" id="S6.SS2.SSS2"> <h4 class="ltx_title ltx_title_subsubsection"> 6.2.2. Gesture-Based Interface for Cognitive Augmentation</h4> <div class="ltx_para" id="S6.SS2.SSS2.p1"> Using the IMU, OmniBuds could detect subtle head and facial movements, enabling gesture-based interfaces to control devices or interact with digital content hands-free. This could enhance AR/VR experiences or provide accessibility options for users with limited mobility. </div> </section> <section class="ltx_subsubsection" id="S6.SS2.SSS3"> <h4 class="ltx_title ltx_title_subsubsection"> 6.2.3. Personalised Fitness Coaching with Motion Analysis</h4> <div class="ltx_para" id="S6.SS2.SSS3.p1"> OmniBuds’ 9-axis IMU and CNN accelerator offer the potential for real-time motion analysis during workouts. By processing head movements and posture data directly on the device, OmniBuds could enable personalised fitness coaching, offering feedback on form and performance without needing external computation. </div> </section> <section class="ltx_subsubsection" id="S6.SS2.SSS4"> <h4 class="ltx_title ltx_title_subsubsection"> 6.2.4. Adaptive Personal Assistant Based on Physical and Cognitive State</h4> <div class="ltx_para" id="S6.SS2.SSS4.p1"> By monitoring a user’s physical and cognitive state, OmniBuds could adapt the behaviour of a personal assistant based on the user’s current conditions. For instance, when detecting fatigue or cognitive overload, the assistant could suggest breaks or adjust task complexity. This speculative application highlights the potential for a more responsive and adaptive interaction enabled by the OmniBuds’ computational units. </div> </section> <section class="ltx_subsubsection" id="S6.SS2.SSS5"> <h4 class="ltx_title ltx_title_subsubsection"> 6.2.5. Acoustic Augmented Reality for Situational Awareness</h4> <div class="ltx_para" id="S6.SS2.SSS5.p1"> Using its dual microphones and Audio DSP, OmniBuds could enhance situational awareness by amplifying critical environmental sounds, such as approaching vehicles or alerts, while reducing background noise. This speculative application could augment safety in urban environments or outdoor activities. This shows the potential for real-time auditory augmentation offered by OmniBuds. </div> </section> </section> </section> <section class="ltx_section" id="S7"> <h2 class="ltx_title ltx_title_section"> 7. Conclusion</h2> <div class="ltx_para" id="S7.p1"> OmniBuds represent a significant advancement in the field of earable sensing and computing, offering a versatile platform that combines cutting-edge hardware with a flexible software architecture. With their integrated health monitoring, energy-efficient computing, and privacy-preserving design, OmniBuds open up new possibilities for researchers exploring diverse applications in wearable technology. As the platform continues to evolve, it has the potential to support groundbreaking research in areas such as health monitoring, cognitive interaction, and beyond. For updates and more information, visit <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://www.omnibuds.tech/" title="">https://www.omnibuds.tech/</a>. </div> </section> <section class="ltx_bibliography" id="bib"> <h2 class="ltx_title ltx_title_bibliography" style="font-size:90%;">References</h2> <ul class="ltx_biblist"> <li class="ltx_bibitem" id="bib.bib1"> (1) </li> <li class="ltx_bibitem" id="bib.bib2"> app ([n. d.]) [n. d.]. <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://support.apple.com/en-mide/111851" style="font-size:90%;" title="">https://support.apple.com/en-mide/111851</a>. [Accessed 09-09-2024]. </li> <li class="ltx_bibitem" id="bib.bib3"> bos ([n. d.]) [n. d.]. Bose QuietComfort Earbuds II — Bose — boseindia.com. <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://www.boseindia.com/en_in/products/headphones/earbuds/quietcomfort-earbuds-ii.html" style="font-size:90%;" title="">https://www.boseindia.com/en_in/products/headphones/earbuds/quietcomfort-earbuds-ii.html</a>. [Accessed 08-09-2024]. </li> <li class="ltx_bibitem" id="bib.bib4"> Ardis and Muchsel ([n. d.]) Kristopher Ardis and Robert Muchsel. [n. d.]. Cutting the AI Power Cord: Technology to Enable True Edge Inference. <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://cms.tinyml.org/wp-content/uploads/talks2020/tinyML_Talks_Kris_Ardis_and_Robert_Muchsel_-201027.pdf" style="font-size:90%;" title="">https://cms.tinyml.org/wp-content/uploads/talks2020/tinyML_Talks_Kris_Ardis_and_Robert_Muchsel_-201027.pdf</a>. [Accessed 09-09-2024]. </li> <li class="ltx_bibitem" id="bib.bib5"> Atallah et al. (2014) L Atallah, A Wiik, B Lo, JP Cobb, AA Amis, and GZ Yang. 2014. Gait asymmetry detection in older adults using a light ear-worn sensor. Physiological measurement 35, 5 (2014), N29. </li> <li class="ltx_bibitem" id="bib.bib6"> Balaji et al. (2023) Ananta Narayanan Balaji, Andrea Ferlini, Fahim Kawsar, and Alessandro Montanari. 2023. Stereo-bp: Non-invasive blood pressure sensing with earables. In Proceedings of the 24th International Workshop on Mobile Computing Systems and Applications. 96–102. </li> <li class="ltx_bibitem" id="bib.bib7"> Bleichner and Debener (2017) Martin G Bleichner and Stefan Debener. 2017. Concealed, unobtrusive ear-centered EEG acquisition: cEEGrids for transparent EEG. Frontiers in human neuroscience 11 (2017), 163. </li> <li class="ltx_bibitem" id="bib.bib8"> Butkow et al. (2023) Kayla-Jade Butkow, Ting Dang, Andrea Ferlini, Dong Ma, and Cecilia Mascolo. 2023. heart: Motion-resilient heart rate monitoring with in-ear microphones. In 2023 IEEE International Conference on Pervasive Computing and Communications (PerCom). IEEE, 200–209. </li> <li class="ltx_bibitem" id="bib.bib9"> Butkow et al. (2024) Kayla-Jade Butkow, Andrea Ferlini, Fahim Kawsar, Cecilia Mascolo, and Alessandro Montanari. 2024. EarTune: Exploring the Physiology of Music Listening. In Companion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing. 644–649. </li> <li class="ltx_bibitem" id="bib.bib10"> Chan et al. (2023) Justin Chan, Antonio Glenn, Malek Itani, Lisa R Mancl, Emily Gallagher, Randall Bly, Shwetak Patel, and Shyamnath Gollakota. 2023. Wireless earbuds for low-cost hearing screening. In Proceedings of the 21st Annual International Conference on Mobile Systems, Applications and Services. 84–95. </li> <li class="ltx_bibitem" id="bib.bib11"> Choi et al. (2022) Seokmin Choi, Yang Gao, Yincheng Jin, Se Jun Kim, Jiyang Li, Wenyao Xu, and Zhanpeng Jin. 2022. PPGface: Like what you are watching? Earphones can” feel” your facial expressions. Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies 6, 2 (2022), 1–32. </li> <li class="ltx_bibitem" id="bib.bib12"> Choudhury (2021) Romit Roy Choudhury. 2021. Earable computing: A new area to think about. In Proceedings of the 22nd International Workshop on Mobile Computing Systems and Applications. 147–153. </li> <li class="ltx_bibitem" id="bib.bib13"> Demirel et al. (2023) Berken Utku Demirel, Khaldoon Al-Naimi, Fahim Kawsar, and Alessandro Montanari. 2023. Cancelling Intermodulation Distortions for Otoacoustic Emission Measurements with Earbuds. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 1–5. </li> <li class="ltx_bibitem" id="bib.bib14"> Demirel et al. (2024) Berken Utku Demirel, Ting Dang, Khaldoon Al-Naimi, Fahim Kawsar, and Alessandro Montanari. 2024. Unobtrusive air leakage estimation for earables with in-ear microphones. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 7, 4 (2024), 1–29. </li> <li class="ltx_bibitem" id="bib.bib15"> Ferlini et al. (2021a) Andrea Ferlini, Dong Ma, Robert Harle, and Cecilia Mascolo. 2021a. EarGate: gait-based user identification with in-ear microphones. In Proceedings of the 27th Annual International Conference on Mobile Computing and Networking. 337–349. </li> <li class="ltx_bibitem" id="bib.bib16"> Ferlini et al. (2021b) Andrea Ferlini, Alessandro Montanari, Andreas Grammenos, Robert Harle, and Cecilia Mascolo. 2021b. Enabling in-ear magnetic sensing: Automatic and user transparent magnetometer calibration. In 2021 IEEE International Conference on Pervasive Computing and Communications (PerCom). IEEE, 1–8. </li> <li class="ltx_bibitem" id="bib.bib17"> Ferlini et al. (2019) Andrea Ferlini, Alessandro Montanari, Cecilia Mascolo, and Robert Harle. 2019. Head motion tracking through in-ear wearables. In Proceedings of the 1st International Workshop on Earable Computing. 8–13. </li> <li class="ltx_bibitem" id="bib.bib18"> Ferlini et al. (2021c) Andrea Ferlini, Alessandro Montanari, Chulhong Min, Hongwei Li, Ugo Sassi, and Fahim Kawsar. 2021c. In-ear ppg for vital signs. IEEE Pervasive Computing 21, 1 (2021), 65–74. </li> <li class="ltx_bibitem" id="bib.bib19"> Franklin et al. (2021) Matija Franklin, David Lagnado, Chulhong Min, Akhil Mathur, and Fahim Kawsar. 2021. Designing memory aids for dementia patients using earables. In Adjunct Proceedings of the 2021 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2021 ACM International Symposium on Wearable Computers. 152–157. </li> <li class="ltx_bibitem" id="bib.bib20"> Gamma (1995) Erich Gamma. 1995. Design patterns: elements of reusable object-oriented software. </li> <li class="ltx_bibitem" id="bib.bib21"> Gashi et al. (2022) Shkurta Gashi, Chulhong Min, Alessandro Montanari, Silvia Santini, and Fahim Kawsar. 2022. A multidevice and multimodal dataset for human energy expenditure estimation using wearable devices. Scientific Data 9, 1 (2022), 537. </li> <li class="ltx_bibitem" id="bib.bib22"> Goverdovsky et al. (2017) Valentin Goverdovsky, Wilhelm Von Rosenberg, Takashi Nakamura, David Looney, David J Sharp, Christos Papavassiliou, Mary J Morrell, and Danilo P Mandic. 2017. Hearables: Multimodal physiological in-ear sensing. Scientific reports 7, 1 (2017), 6948. </li> <li class="ltx_bibitem" id="bib.bib23"> Kalanadhabhatta et al. (2021) Manasa Kalanadhabhatta, Chulhong Min, Alessandro Montanari, and Fahim Kawsar. 2021. FatigueSet: A multi-modal dataset for modeling mental fatigue and fatigability. In International Conference on Pervasive Computing Technologies for Healthcare. Springer, 204–217. </li> <li class="ltx_bibitem" id="bib.bib24"> Kawsar et al. (2018a) Fahim Kawsar, Chulhong Min, Akhil Mathur, and Alessandro Montanari. 2018a. Earables for personal-scale behavior analytics. IEEE Pervasive Computing 17, 3 (2018), 83–89. </li> <li class="ltx_bibitem" id="bib.bib25"> Kawsar et al. (2018b) Fahim Kawsar, Chulhong Min, Akhil Mathur, Alessandro Montanari, Utku Günay Acer, and Marc Van den Broeck. 2018b. eSense: Open earable platform for human sensing. In Proceedings of the 16th ACM Conference on Embedded Networked Sensor Systems. 371–372. </li> <li class="ltx_bibitem" id="bib.bib26"> Kidmose et al. (2013) Preben Kidmose, David Looney, Michael Ungstrup, Mike Lind Rank, and Danilo P Mandic. 2013. A study of evoked potentials from ear-EEG. IEEE Transactions on Biomedical Engineering 60, 10 (2013), 2824–2830. </li> <li class="ltx_bibitem" id="bib.bib27"> Lee et al. (2019) Seungchul Lee, Chulhong Min, Alessandro Montanari, Akhil Mathur, Youngjae Chang, Junehwa Song, and Fahim Kawsar. 2019. Automatic smile and frown recognition with kinetic earables. In Proceedings of the 10th Augmented Human International Conference 2019. 1–4. </li> <li class="ltx_bibitem" id="bib.bib28"> Ma et al. (2021) Dong Ma, Andrea Ferlini, and Cecilia Mascolo. 2021. Oesense: employing occlusion effect for in-ear human sensing. In Proceedings of the 19th Annual International Conference on Mobile Systems, Applications, and Services. 175–187. </li> <li class="ltx_bibitem" id="bib.bib29"> Min et al. (2023) Chulhong Min, Akhil Mathur, Utku Günay Acer, Alessandro Montanari, and Fahim Kawsar. 2023. SensiX++: Bringing mlops and multi-tenant model serving to sensory edge devices. ACM Transactions on Embedded Computing Systems 22, 6 (2023), 1–27. </li> <li class="ltx_bibitem" id="bib.bib30"> Min et al. (2022) Chulhong Min, Akhil Mathur, Alessandro Montanari, and Fahim Kawsar. 2022. SensiX: A system for best-effort inference of machine learning models in multi-device environments. IEEE Transactions on Mobile Computing 22, 9 (2022), 5525–5538. </li> <li class="ltx_bibitem" id="bib.bib31"> Montanari et al. (2023) Alessandro Montanari, Andrea Ferlini, Ananta Narayanan Balaji, Cecilia Mascolo, and Fahim Kawsar. 2023. EarSet: A Multi-Modal Dataset for Studying the Impact of Head and Facial Movements on In-Ear PPG Signals. Scientific Data 10, 1 (2023), 850. </li> <li class="ltx_bibitem" id="bib.bib32"> Moss et al. (2022) Arthur Moss, Hyunjong Lee, Lei Xun, Chulhong Min, Fahim Kawsar, and Alessandro Montanari. 2022. Ultra-low power DNN accelerators for IoT: Resource characterization of the MAX78000. In Proceedings of the 20th ACM Conference on Embedded Networked Sensor Systems. 934–940. </li> <li class="ltx_bibitem" id="bib.bib33"> Prakash et al. (2019) Jay Prakash, Zhijian Yang, Yu-Lin Wei, and Romit Roy Choudhury. 2019. Stear: Robust step counting from earables. In Proceedings of the 1st International Workshop on Earable Computing. 36–41. </li> <li class="ltx_bibitem" id="bib.bib34"> Röddiger et al. (2022) Tobias Röddiger, Christopher Clarke, Paula Breitling, Tim Schneegans, Haibin Zhao, Hans Gellersen, and Michael Beigl. 2022. Sensing with earables: A systematic literature review and taxonomy of phenomena. Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies 6, 3 (2022), 1–57. </li> <li class="ltx_bibitem" id="bib.bib35"> Romero et al. (2024) Julia Romero, Andrea Ferlini, Dimitris Spathis, Ting Dang, Katayoun Farrahi, Fahim Kawsar, and Alessandro Montanari. 2024. OptiBreathe: An Earable-based PPG System for Continuous Respiration Rate, Breathing Phase, and Tidal Volume Monitoring. In Proceedings of the 25th International Workshop on Mobile Computing Systems and Applications. 99–106. </li> <li class="ltx_bibitem" id="bib.bib36"> Shahid et al. (2024) Irtaza Shahid, Khaldoon Al-Naimi, Ting Dang, Yang Liu, Fahim Kawsar, and Alessandro Montanari. 2024. Towards Enabling DPOAE Estimation on Single-Speaker Earbuds. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 246–250. </li> <li class="ltx_bibitem" id="bib.bib37"> Truong et al. (2022) Hoang Truong, Alessandro Montanari, and Fahim Kawsar. 2022. Non-invasive blood pressure monitoring with multi-modal in-ear sensing. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 6–10. </li> <li class="ltx_bibitem" id="bib.bib38"> Veluri et al. (2023) Bandhav Veluri, Malek Itani, Justin Chan, Takuya Yoshioka, and Shyamnath Gollakota. 2023. Semantic hearing: Programming acoustic scenes with binaural hearables. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. 1–15. </li> <li class="ltx_bibitem" id="bib.bib39"> Veluri et al. (2024a) Bandhav Veluri, Malek Itani, Tuochao Chen, and Shyamnath Gollakota. 2024a. AI-based headphones for augmenting human hearing. The Journal of the Acoustical Society of America 155, 3_Supplement (2024), A298–A298. </li> <li class="ltx_bibitem" id="bib.bib40"> Veluri et al. (2024b) Bandhav Veluri, Malek Itani, Tuochao Chen, Takuya Yoshioka, and Shyamnath Gollakota. 2024b. Look Once to Hear: Target Speech Hearing with Noisy Examples. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1–16. </li> <li class="ltx_bibitem" id="bib.bib41"> Yadav et al. (2018) Umang Yadav, Sherif N Abbas, and Dimitrios Hatzinakos. 2018. Evaluation of PPG biometrics for authentication in different states. In 2018 International Conference on Biometrics (ICB). IEEE, 277–282. </li> <li class="ltx_bibitem" id="bib.bib42"> Yang and Choudhury (2021) Zhijian Yang and Romit Roy Choudhury. 2021. Personalizing head related transfer functions for earables. In Proceedings of the 2021 ACM SIGCOMM 2021 Conference. 137–150. </li> <li class="ltx_bibitem" id="bib.bib43"> Yang et al. (2020) Zhijian Yang, Yu-Lin Wei, Sheng Shen, and Romit Roy Choudhury. 2020. Ear-ar: indoor acoustic augmented reality on earphones. In Proceedings of the 26th Annual International Conference on Mobile Computing and Networking. 1–14. </li> </ul> </section> </article> </div> <footer class="ltx_page_footer"> <div class="ltx_page_logo">Generated on Mon Oct 7 06:31:24 2024 by <a class="ltx_LaTeXML_logo" href="http://dlmf.nist.gov/LaTeXML/">LaTeXML<img alt="Mascot Sammy" src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg=="/></a> </div></footer> </div> </body> </html>