publications

2025

1. NeurIPS

   ContextAgent: Context-Aware Proactive LLM Agents with Open-world Sensory Perceptions

   Bufang Yang, Lilin Xu, Liekang Zeng, Kaiwei Liu, Siyang Jiang, Wenrui Lu, Hongkai Chen, Xiaofan Jiang, Guoliang Xing, and Zhenyu Yan^*

   In The 39th Annual Conference on Neural Information Processing Systems, Acceptance ratio: 5290/21575=24.5% , Dec 2025

   Abs Bib PDF

   Recent advances in Large Language Models (LLMs) have propelled intelligent agents from reactive responses to proactive support. While promising, existing proactive agents either rely exclusively on observations from enclosed environments (e.g., desktop UIs) with direct LLM inference or employ rule-based proactive notifications, leading to suboptimal user intent understanding and limited functionality for proactive service. In this paper, we introduce ContextAgent, the first context-aware proactive agent that incorporates extensive sensory contexts to enhance the proactive capabilities of LLM agents. ContextAgent first extracts multi-dimensional contexts from massive sensory perceptions on wearables (e.g., video and audio) to understand user intentions. ContextAgent then leverages the sensory contexts and the persona contexts from historical data to predict the necessity for proactive services. When proactive assistance is needed, ContextAgent further automatically calls the necessary tools to assist users unobtrusively. To evaluate this new task, we curate ContextAgentBench, the first benchmark for evaluating context-aware proactive LLM agents, covering 1,000 samples across nine daily scenarios and twenty tools. Experiments on ContextAgentBench show that ContextAgent outperforms baselines by achieving up to 8.5% and 6.0% higher accuracy in proactive predictions and tool calling, respectively.

   @inproceedings{yang2025contextagent,
     title = {ContextAgent: Context-Aware Proactive LLM Agents with Open-world Sensory Perceptions},
     author = {Yang, Bufang and Xu, Lilin and Zeng, Liekang and Liu, Kaiwei and Jiang, Siyang and Lu, Wenrui and Chen, Hongkai and Jiang, Xiaofan and Xing, Guoliang and Yan, Zhenyu},
     booktitle = {The 39th Annual Conference on Neural Information Processing Systems},
     year = {2025},
     month = dec,
     keywords = {LLM Agents, Human-centric AI, Personal Assistant},
     publisher = {NeurIPS},
   }

2. MobiCom

   AquaScan: A Sonar-based Underwater Sensing System for Human Activity Monitoring

   Haozheng Hou, Bowen Zheng, Sitong Cheng, Xiaoguang Zhao, Peiheng Wu, Lixing He, Yunqi Guo, Guoliang Xing, and Zhenyu Yan^*

   In The 31st Annual International Conference on Mobile Computing and Networking, Nov 2025

   Gold Medal@Geneva Abs Bib PDF

   Gold Medal, the 50th International Exhibition of Inventions Geneva

   Human activity monitoring in the water is essential for pool management and drowning prevention. Existing camera-based solutions pose significant concerns about privacy and extra installation costs. Although sonars have been widely used for underwater sensing in open aquatic environments such as oceans and lakes, monitoring human activities with sonars in a pool setup is still challenging. In this work, we propose AquaScan, the first scanning sonar-based underwater sensing system for human activity monitoring. To overcome the low frame rate due to the sonar’s physical limitation, we propose a novel scanning strategy and apply an image reconstruction method to accelerate the scanning speed without compromising the performance of motion detection. To overcome the dynamic interferences in the underwater scenario, we develop a novel signal processing pipeline based on a physical model to remove noises and localize human subjects. We further extract features like motion, time, and spatial information from sonar images and develop a state-transfer-based activity recognition system to recognize five common water activities, i.e., swimming, motionless, splashing, struggling, and drowning. We have deployed AquaScan on three public swimming pools for a total period of 94 hours. The evaluation results show that AquaScan can successfully recognize the five activities in the water with around 91.5%.

   @inproceedings{aquascan-mobicom,
     author = {Hou, Haozheng and Zheng, Bowen and Cheng, Sitong and Zhao, Xiaoguang and Wu, Peiheng and He, Lixing and Guo, Yunqi and Xing, Guoliang and Yan, Zhenyu},
     title = {AquaScan: A Sonar-based Underwater Sensing System for Human Activity Monitoring},
     year = {2025},
     month = nov,
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     booktitle = {The 31st Annual International Conference on Mobile Computing and Networking},
     series = {MobiCom},
   }

3. SenSys

   TaskSense: A Translation-like Approach for Tasking Heterogeneous Sensor Systems with LLMs

   Kaiwei Liu, Bufang Yang, Lilin Xu, Yunqi Guo, Guoliang Xing, Xian Shuai, Xiaozhe Ren, Xin Jiang, and Zhenyu Yan^*

   In Proceedings of the 23rd ACM Conference on Embedded Networked Sensor Systems, Acceptance ratio: 46/235=19.6% , May 2025

   Bib PDF

   @inproceedings{liu2025tasksense,
     title = {TaskSense: A Translation-like Approach for Tasking Heterogeneous Sensor Systems with LLMs},
     author = {Liu, Kaiwei and Yang, Bufang and Xu, Lilin and Guo, Yunqi and Xing, Guoliang and Shuai, Xian and Ren, Xiaozhe and Jiang, Xin and Yan, Zhenyu},
     booktitle = {Proceedings of the 23rd ACM Conference on Embedded Networked Sensor Systems},
     month = may,
     year = {2025},
   }

4. Digital Medicine

   Alzheimer’s disease digital biomarkers multidimensional landscape and AI model scoping review

   Wenhao Qi, Xiaohong Zhu, Bin Wang, Yankai Shi, Chaoqun Dong, Shiying Shen, Jiaqi Li, Kun Zhang, Yunfan He, Mengjiao Zhao, Shiyan Yao, Yongze Dong, Huajuan Shen, Junling Kang, Xiaodong Lu, Guowei Jiang, Lizzy M. M. Boots, Heming Fu, Li Pan, Hongkai Chen, Zhenyu Yan, Guoliang Xing^*, and Shihua Cao^*

   npj Digital Medicine, May 2025

   Abs DOI Bib HTML PDF

   As digital biomarkers gain traction in Alzheimer’s disease (AD) diagnosis, understanding recent advancements is crucial. This review conducts a bibliometric analysis of 431 studies from five online databases: Web of Science, PubMed, Embase, IEEE Xplore, and CINAHL, and provides a scoping review of 86 artificial intelligence (AI) models. Research in this field is supported by 224 grants across 54 disciplines and 1403 institutions in 44 countries, with 2571 contributing researchers. Key focuses include motor activity, neurocognitive tests, eye tracking, and speech analysis. Classical machine learning models dominate AI research, though many lack performance reporting. Of 21 AD-focused models, the average AUC is 0.887, while 45 models for mild cognitive impairment show an average AUC of 0.821. Notably, only 2 studies incorporated external validation, and 3 studies performed model calibration. This review highlights the progress and challenges of integrating digital biomarkers into clinical practice.

   @article{qi2025alzheimer,
     title = {Alzheimer's disease digital biomarkers multidimensional landscape and AI model scoping review},
     author = {Qi, Wenhao and Zhu, Xiaohong and Wang, Bin and Shi, Yankai and Dong, Chaoqun and Shen, Shiying and Li, Jiaqi and Zhang, Kun and He, Yunfan and Zhao, Mengjiao and Yao, Shiyan and Dong, Yongze and Shen, Huajuan and Kang, Junling and Lu, Xiaodong and Jiang, Guowei and Boots, Lizzy M. M. and Fu, Heming and Pan, Li and Chen, Hongkai and Yan, Zhenyu and Xing, Guoliang and Cao, Shihua},
     journal = {npj Digital Medicine},
     volume = {8},
     number = {1},
     pages = {1--33},
     year = {2025},
     publisher = {Nature Publishing Group},
     doi = {10.1038/s41746-025-01640-z},
   }

5. IMWUT

   SocialMind: LLM-based Proactive AR Social Assistive System with Human-like Perception for In-situ Live Interactions

   Bufang Yang, Yunqi Guo, Lilin Xu, Zhenyu Yan^*, Hongkai Chen, Guoliang Xing, and Xiaofan Jiang

   Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., May 2025

   Abs Bib PDF

   Social interactions are fundamental to human life. The recent emergence of large language models (LLMs)-based virtual assistants has demonstrated their potential to revolutionize human interactions and lifestyles. However, existing assistive systems mainly provide reactive services to individual users, rather than offering in-situ assistance during live social interactions with conversational partners. In this study, we introduce SocialMind, the first LLM-based proactive AR social assistive system that provides users with in-situ social assistance. SocialMind employs human-like perception leveraging multi-modal sensors to extract both verbal and nonverbal cues, social factors, and implicit personas, incorporating these social cues into LLM reasoning for social suggestion generation. Additionally, SocialMind employs a multi-tier collaborative generation strategy and proactive update mechanism to display social suggestions on Augmented Reality (AR) glasses, ensuring that suggestions are timely provided to users without disrupting the natural flow of conversation. Evaluations on three public datasets and a user study with 20 participants show that SocialMind achieves 38.3% higher engagement compared to baselines, and 95% of participants are willing to use SocialMind in their live social interactions.

   @article{yang2025imwut,
     author = {Yang, Bufang and Guo, Yunqi and Xu, Lilin and Yan, Zhenyu and Chen, Hongkai and Xing, Guoliang and Jiang, Xiaofan},
     title = {SocialMind: LLM-based Proactive AR Social Assistive System with Human-like Perception for In-situ Live Interactions},
     year = {2025},
     journal = {Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.},
     volume = {9},
     number = {1},
     pages = {1--30},
   }

2024

1. IMWUT

   DrHouse: An LLM-empowered Diagnostic Reasoning System through Harnessing Outcomes from Sensor Data and Expert Knowledge

   Bufang Yang, Siyang Jiang, Lilin Xu, Kaiwei Liu, Hai Li, Guoliang Xing, Hongkai Chen, Xiaofan Jiang, and Zhenyu Yan^*

   Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., May 2024

   Abs DOI Bib PDF

   Large language models (LLMs) have the potential to transform digital healthcare, as evidenced by recent advances in LLM-based virtual doctors. However, current approaches rely on patient’s subjective descriptions of symptoms, causing increased misdiagnosis. Recognizing the value of daily data from smart devices, we introduce a novel LLM-based multi-turn consultation virtual doctor system, DrHouse, which incorporates three significant contributions: 1) It utilizes sensor data from smart devices in the diagnosis process, enhancing accuracy and reliability. 2) DrHouse leverages continuously updating medical knowledge bases to ensure its model remains at diagnostic standard’s forefront. 3) DrHouse introduces a novel diagnostic algorithm that concurrently evaluates potential diseases and their likelihood, facilitating more nuanced and informed medical assessments. Through multi-turn interactions, DrHouse determines the next steps, such as accessing daily data from smart devices or requesting in-lab tests, and progressively refines its diagnoses. Evaluations on three public datasets and our self-collected datasets show that DrHouse can achieve up to an 31.5% increase in diagnosis accuracy over the state-of-the-art baselines. The results of a 32-participant user study show that 75% medical experts and 91.7% test subjects are willing to use DrHouse.

   @article{yang2024imwut,
     author = {Yang, Bufang and Jiang, Siyang and Xu, Lilin and Liu, Kaiwei and Li, Hai and Xing, Guoliang and Chen, Hongkai and Jiang, Xiaofan and Yan, Zhenyu},
     title = {DrHouse: An LLM-empowered Diagnostic Reasoning System through Harnessing Outcomes from Sensor Data and Expert Knowledge},
     year = {2024},
     issue_date = {December 2024},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     volume = {8},
     number = {4},
     url = {https://doi.org/10.1145/3699765},
     doi = {10.1145/3699765},
     journal = {Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.},
   }

2. MobiCom

   Soar: Design and Deployment of A Smart Roadside Infrastructure System for Autonomous Driving

   Shuyao Shi, Neiwen Ling, Zhehao Jiang, Xuan Huang, Yuze He, Xiaoguang Zhao, Bufang Yang, Chen Bian, Jingfei Xia, Zhenyu Yan, Raymond W. Yeung, and Guoliang Xing^*

   In The 30th Annual International Conference on Mobile Computing and Networking, Acceptance ratio: 48/207=23.2% , Nov 2024

   Best Artifact Awards Runner-up Abs Bib PDF

   Best Artifact Awards Runner-up, ACM MobiCom 2024

   Recently, smart roadside infrastructure (SRI) has demonstrated the potential of achieving fully autonomous driving systems. To explore the potential of infrastructure-assisted autonomous driving, this paper presents the design and deployment of Soar, the first end-to-end SRI system specifically designed to support autonomous driving systems. Soar consists of both software and hardware components carefully designed to overcome various system and physical challenges. Soar can leverage the existing operational infrastructure like street lampposts for a lower barrier of adoption. Soar adopts a new communication architecture that comprises a bi-directional multi-hop I2I network and a downlink I2V broadcast service, which are designed based on off-the-shelf 802.11ac interfaces in an integrated manner. Soar also features a hierarchical DL task management framework to achieve desirable load balancing among nodes and enable them to collaborate efficiently to run multiple data-intensive autonomous driving applications. We deployed a total of 18 Soar nodes on existing lampposts on campus, which have been operational for over two years. Our real-world evaluation shows that Soar can support a diverse set of autonomous driving applications and achieve desirable real-time performance and high communication reliability. Our findings and experiences in this work offer key insights into the development and deployment of next-generation smart roadside infrastructure and autonomous driving systems.

   @inproceedings{shi2024mobicom,
     author = {Shi, Shuyao and Ling, Neiwen and Jiang, Zhehao and Huang, Xuan and He, Yuze and Zhao, Xiaoguang and Yang, Bufang and Bian, Chen and Xia, Jingfei and Yan, Zhenyu and Yeung, Raymond W. and Xing, Guoliang},
     title = {Soar: Design and Deployment of A Smart Roadside Infrastructure System for Autonomous Driving},
     booktitle = {The 30th Annual International Conference on Mobile Computing and Networking},
     year = {2024},
     month = nov,
     address = {Washington, D.C., USA},
     pages = {14},
   }

3. MobiCom

   ADMarker: A Multi-Modal Federated Learning System for Monitoring Digital Biomarkers of Alzheimer’s Disease

   Xiaomin Ouyang, Xian Shuai, Yang Li, Li Pan, Xifan Zhang, Heming Fu, Sitong Cheng, Xinyan Wang, Shihua Cao, Jiang Xin, Hazel Mok, Zhenyu Yan, Doris Sau Fung Yu, Timothy Kwok, and Guoliang Xing^*

   In The 30th Annual International Conference on Mobile Computing and Networking, Washington, D.C., USA, Acceptance rate: 48/207=23.2% , Nov 2024

   Abs Bib PDF

   Alzheimer’s Disease (AD) and related dementia are a growing global health challenge due to the aging population. In this paper, we present ADMarker, the first end-to-end system that integrates multi-modal sensors and new federated learning algorithms for detecting multidimensional AD digital biomarkers in natural living environments. ADMarker features a novel three-stage multi-modal federated learning architecture that can accurately detect digital biomarkers in a privacy-preserving manner. Our approach collectively addresses several major real-world challenges, such as limited data labels, data heterogeneity, and limited computing resources. We built a compact multi-modality hardware system and deployed it in a four-week clinical trial involving 91 elderly participants. The results indicate that ADMarker can accurately detect a comprehensive set of digital biomarkers with up to 93.8% accuracy and identify early AD with an average of 88.9% accuracy. ADMarker offers a new platform that can allow AD clinicians to characterize and track the complex correlation between multidimensional interpretable digital biomarkers, demographic factors of patients, and AD diagnosis in a longitudinal manner.

   @inproceedings{admarker-mobicom,
     author = {Ouyang, Xiaomin and Shuai, Xian and Li, Yang and Pan, Li and Zhang, Xifan and Fu, Heming and Cheng, Sitong and Wang, Xinyan and Cao, Shihua and Xin, Jiang and Mok, Hazel and Yan, Zhenyu and Yu, Doris Sau Fung and Kwok, Timothy and Xing, Guoliang},
     title = {ADMarker: A Multi-Modal Federated Learning System for Monitoring Digital Biomarkers of Alzheimer's Disease},
     year = {2024},
     month = nov,
     booktitle = {The 30th Annual International Conference on Mobile Computing and Networking},
     location = {Washington, D.C., USA},
     pages = {14},
   }

2023

1. SenSys

   EdgeFM: Leveraging Foundation Model for Open-set Learning on the Edge

   Bufang Yang, Lixing He, Neiwen Ling, Zhenyu Yan^*, Guoliang Xing, Xian Shuai, Xiaozhe Ren, and Xin Jiang

   In The 21st ACM Conference on Embedded Networked Sensor Systems, Acceptance ratio: 35/179=19.6% , Nov 2023

   Abs Bib PDF

   Deep Learning (DL) models have been widely deployed on IoT devices with the help of advancements in DL algorithms and chips. However, the limited resources of edge devices make these on-device DL models hard to be generalizable to diverse environments and tasks. Although the recently emerged foundation models (FMs) show impressive generalization power, how to effectively leverage the rich knowledge of FMs on resource-limited edge devices is still not explored. In this paper, we propose EdgeFM, a novel edge-cloud cooperative system with open-set recognition capability. EdgeFM selectively uploads unlabeled data to query the FM on the cloud and customizes the specific knowledge and architectures for edge models. Meanwhile, EdgeFM conducts dynamic model switching at run-time taking into account both data uncertainty and dynamic network variations, which ensures the accuracy always close to the original FM. We implement EdgeFM using two FMs on two edge platforms. We evaluate EdgeFM on three public datasets and two self-collected datasets. Results show that EdgeFM can reduce the end-to-end latency up to 3.2x and achieve 34.3% accuracy increase compared with the baseline.

   @inproceedings{edgefm-sensys,
     author = {Yang, Bufang and He, Lixing and Ling, Neiwen and Yan, Zhenyu and Xing, Guoliang and Shuai, Xian and Ren, Xiaozhe and Jiang, Xin},
     title = {EdgeFM: Leveraging Foundation Model for Open-set Learning on the Edge},
     year = {2023},
     month = nov,
     booktitle = {The 21st ACM Conference on Embedded Networked Sensor Systems},
   }

2. MobiCom

   VI-Map: Infrastructure-Assisted Real-Time HD Mapping for Autonomous Driving

   Yuze He, Chen Bian, Jingfei Xia, Shuyao Shi, Zhenyu Yan^*, Qun Song, and Guoliang Xing

   In The 29th Annual International Conference on Mobile Computing and Networking, Madrid, Spain, Acceptance ratio: 92/377=24.4% , Oct 2023

   Best Community Contributions Award; Gold Medal@Geneva Abs Bib PDF

   Best Community Contributions Award, ACM MobiCom 2023; Gold Modal, the 49th International Exhibition of Inventions Geneva

   HD map is a key enabling technology towards fully autonomous driving. We propose VI-Map, the first system that leverages roadside infrastructure to enhance real-time HD mapping for autonomous driving. The core concept of VI-Map is to exploit the unique cumulative observations made by roadside infrastructure to build and maintain an accurate and current HD map. This HD map is then fused with on-vehicle HD maps in real time, resulting in a more comprehensive and up-to-date HD map. By extracting concise bird-eye-view features from infrastructure observations and utilizing vectorized map representations, VI-Map incurs low compute and communication overhead. We conducted end-to-end evaluations of VI-Map on a real-world testbed and a simulator. Experiment results show that VI-Map can construct decentimeter-level (up to 0.3 m) HD maps and achieve real-time (up to a delay of 42 ms) map fusion between driv- ing vehicles and roadside infrastructure. This represents a significant improvement of 2.8x and 3x in map accuracy and coverage compared to the state-of-the-art online HD map- ping approaches. A video demo of VI-Map on our real-world testbed is available at https://youtu.be/p2RO65R5Ezg.

   @inproceedings{vimap-mobicom,
     author = {He, Yuze and Bian, Chen and Xia, Jingfei and Shi, Shuyao and Yan, Zhenyu and Song, Qun and Xing, Guoliang},
     title = {VI-Map: Infrastructure-Assisted Real-Time HD Mapping for Autonomous Driving},
     year = {2023},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     booktitle = {The 29th Annual International Conference on Mobile Computing and Networking},
     location = {Madrid, Spain},
     series = {MobiCom '23},
     month = oct,
   }

3. IMWUT

   TouchKey: Touch to Generate Symmetric Keys by Skin Electric Potentials Induced by Powerline Radiation

   Yuchen Miao, Chaojie Gu, Zhenyu Yan, Sze Yiu Chau, Rui Tan, Qi Lin, Wen Hu, Shibo He, and Jiming Chen

   In Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Oct 2023

   Abs Bib HTML PDF

   Secure device pairing is important to wearables. Existing solutions either degrade usability due to the need of specific actions like shaking, or they lack universality due to the need of dedicated hardware like electrocardiogram sensors. This paper proposes TouchKey, a symmetric key generation scheme that exploits the skin electric potential (SEP) induced by powerline electromagnetic radiation. The SEP is ubiquitously accessible indoors with analog-to-digital converters widely available on Internet of Things devices. Our measurements show that the SEP has high randomness and the SEPs measured at two close locations on the same human body are similar. Extensive experiments show that TouchKey achieves a high key generation rate of 345 bit/s and an average success rate of 99.29%. Under a range of adversary models including active and passive attacks, TouchKey shows a low false acceptance rate of 0.86%, which outperforms existing solutions. Besides, the overall execution time and energy usage are 0.44 s and 2.716 mJ, which make it suitable for resource-constrained devices.

   @inproceedings{touchkey-imwut,
     title = {TouchKey: Touch to Generate Symmetric Keys by Skin Electric Potentials Induced by Powerline Radiation},
     author = {Miao, Yuchen and Gu, Chaojie and Yan, Zhenyu and Chau, Sze Yiu and Tan, Rui and Lin, Qi and Hu, Wen and He, Shibo and Chen, Jiming},
     booktitle = {Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies},
     year = {2023},
   }

4. IEEE S&P

   Uncovering User Interactions on Smartphones via Contactless Wireless Charging Side Channels

   Tao Ni, Xiaokuan Zhang, Chaoshun Zuo, Jianfeng Li, Zhenyu Yan, Wubing Wang, Weitao Xu, Xiapu Luo, and Qingchuan Zhao

   In The 44th IEEE Symposium on Security and Privacy, May 2023

   Abs Bib PDF

   Today, there is an increasing number of smartphones supporting wireless charging that leverages electromagnetic induction to transmit power from a wireless charger to the charging smartphone. In this paper, we report a new contactless and context-aware wireless-charging side-channel attack, which captures two physical phenomena (i.e., the coil whine and the magnetic field perturbation) generated during this wireless charging process and further infers the user interactions on the charging smartphone. We design and implement a three-stage attack framework, dubbed WISERS, to demonstrate the practicality of this new side channel. WISERS first captures the coil whine and the magnetic field perturbation emitted by the wireless charger, then infers (i) inter-interface switches (e.g., switching from the home screen to an app interface) and (ii) intra-interface activities (e.g., keyboard inputs inside an app) to build user interaction contexts, and further reveals sensitive information. We extensively evaluate the effectiveness of WISERS with popular smartphones and commercial-off-the-shelf (COTS) wireless chargers. Our evaluation results suggest that WISERS can achieve over 90.4% accuracy in inferring sensitive information, such as screen-unlocking passcode and app launch. In addition, our study also shows that WISERS is resilient to a list of impact factors.

   @inproceedings{wireless-charge-sp,
     title = {Uncovering User Interactions on Smartphones via Contactless Wireless Charging Side Channels},
     author = {Ni, Tao and Zhang, Xiaokuan and Zuo, Chaoshun and Li, Jianfeng and Yan, Zhenyu and Wang, Wubing and Xu, Weitao and Luo, Xiapu and Zhao, Qingchuan},
     booktitle = {The 44th IEEE Symposium on Security and Privacy},
     month = may,
     year = {2023},
   }

5. MobiSys

   Towards Bone-Conducted Vibration Speech Enhancement on Head-Mounted Wearables

   Lixing He, Haozheng Hou, Shuyao Shi, Xian Shuai, and Zhenyu Yan^*

   In The 21st ACM International Conference on Mobile Systems, Applications, and Services , Acceptance ratio: 41/198=20.7% , Dec 2023

   Abs Bib PDF

   Head-mounted wearables are rapidly growing in popularity. However, a gap exists in providing robust voice-related applications like conversation or command control in complex environments such as competing speakers and strong noises. The compact design of HMWs introduces non-trivial challenges to existing speech enhancement systems that use microphone recording only. In this paper, we handle this problem by using bone vibration conducted through the head skull. The principle is that the accelerometer is widely installed on head-mounted wearables and can capture the clean user’s voice. Hence, we develop VibVoice, a lightweight multi-modal speech enhancement system for head-mounted wearables. We design a two-branch encoder-decoder deep neural network to fuse the high-level features of the two modalities and reconstruct clean speech. To address the insufficient training data of paired data, we extensively measure the bone conduction effect from a limited dataset to extract the physical impulse function for cross-modal data augmentation. We evaluate VibVoice on a real-world dataset and compare it with two state-of-the-art baselines. Results show that VibVoice yields up to 21% better performance in PESQ and up to 26% better performance in SNR compared with the baseline with 72 times less paired data required. We also validate VibVoice’s performance through a user study with 35 participants, where 87% participants prefer VibVoice compared with the baseline. In addition, VibVoice requires 4 to 31 times less execution time compared with baselines on mobile devices. The demo audio of VibVoice is available at https://www.youtube.com/watch?v=8_-s_C_NGRI.

   @inproceedings{vibvoice-mobisys,
     title = {Towards Bone-Conducted Vibration Speech Enhancement on Head-Mounted Wearables},
     author = {He, Lixing and Hou, Haozheng and Shi, Shuyao and Shuai, Xian and Yan, Zhenyu},
     booktitle = {The 21st ACM International Conference on Mobile Systems, Applications, and Services },
     year = {2023},
     month = dec,
   }

6. IPSN

   CoEdge: A Cooperative Edge System for Distributed Real-Time Deep Learning Tasks

   Zhehao Jiang, Neiwen Ling, Xuan Huang, Shuyao Shi, Chenhao Wu, Xiaoguang Zhao, Zhenyu Yan, and Guoliang Xing^*

   In The 22nd International Conference on Information Processing in Sensor Networks, May 2023

   Abs Bib HTML PDF

   Recent years have witnessed the emergence of a new class of cooperative edge systems in which a large number of edge nodes can collaborate through local peer-to-peer connectivity. In this paper, we propose CoEdge, a novel cooperative edge system that can support concurrent data/compute-intensive deep learning (DL) models for distributed real-time applications such as city-scale traffic monitoring and autonomous driving. First, CoEdge includes a hierarchical DL task scheduling framework that dispatches DL tasks to edge nodes based on their computational profiles, communication overhead, and real-time requirements. Second, CoEdge can dramatically increase the execution efficiency of DL models by batching sensor data and aggregating the inferences of the same model. Finally, we propose a new edge containerization approach that enables an edge node to execute concurrent DL tasks by partitioning the CPU and GPU workloads into different containers. We extensively evaluate CoEdge on a self-deployed smart lamppost testbed on a university campus. Our results show that CoEdge can achieve up to reduction on deadline missing rate compared to baselines.

   @inproceedings{jiang2023coedge,
     title = {CoEdge: A Cooperative Edge System for Distributed Real-Time Deep Learning Tasks},
     author = {Jiang, Zhehao and Ling, Neiwen and Huang, Xuan and Shi, Shuyao and Wu, Chenhao and Zhao, Xiaoguang and Yan, Zhenyu and Xing, Guoliang},
     booktitle = {The 22nd International Conference on Information Processing in Sensor Networks},
     year = {2023},
     month = may,
   }

7. HotMobile

   Moses: Efficient Exploitation of Cross-device Transferable Features for Tensor Program Optimization

   Zhihe Zhao, Xian Shuai, Neiwen Ling, Nan Guan, Zhenyu Yan, and Guoliang Xing^*

   In The 24th International Workshop on Mobile Computing Systems and Applications, Acceptance ratio: 19/46=41% , Feb 2023

   Abs Bib PDF

   Achieving efficient execution of machine learning models on mobile/edge devices has attracted significant attention recently. A key challenge is to generate high-performance tensor programs for each operator inside a DNN model efficiently. To this end, deep learning compilers have adopted auto-tuning approaches such as Ansor. However, it is challenging to optimize tensor codes for mobile/edge devices by auto-tuning due to limited time budgets and on-device resources. A key component of DNN compilers is the cost model that can predict the performance of each configuration on specific devices. However, current design of cost models cannot provide transferable features between different hardware accelerators efficiently and effectively. In this paper, we propose Moses, a simple yet efficient design based on the lottery ticket hypothesis, which fully takes advantage of the hardware-agnostic features transferable to the target device via domain adaptation to optimize the time-consuming auto-tuning process of DNN compiling on a new hardware platform. Compared with state-of-the-art approaches, Moses achieves up to 1.53X efficiency gain in the search stage and 1.41X inference speedup on challenging DNN benchmarks.

   @inproceedings{moses-hotmobile,
     title = {Moses: Efficient Exploitation of Cross-device Transferable Features for Tensor Program Optimization},
     author = {Zhao, Zhihe and Shuai, Xian and Ling, Neiwen and Guan, Nan and Yan, Zhenyu and Xing, Guoliang},
     booktitle = {The 24th International Workshop on Mobile Computing Systems and Applications},
     year = {2023},
     month = feb,
   }

2022

1. SenSys

   Telesonar: Robocall Alarm System by Detecting Echo Channel and Breath Timing

   Zhenyu Yan, Qun Song, Rui Tan, and Chris Xiaoxuan Lu

   In The 20th ACM Conference on Embedded Networked Sensor Systems, Acceptance ratio: 52/208=25% , Nov 2022

   Abs Bib PDF

   Massive fraudulent and phishing robocalls present threats to societies. The integration of artificial intelligence technologies, including dialogue and voice generation systems, renders the robocalls more deceptive. Existing countermeasures such as caller ID, call provenance, voiceprint, and fake voice detection have respective limitations and are heavyweight for end users’ smartphones. This paper studies detecting the acoustic echo channel on the remote end of a call based on the received voice. The positive detection result evidencing the physical setup of an audio system is indicative of a human caller. However, the acoustic echo cancellation mechanisms of most audio systems and the use of earphone/headset diminish echoes significantly. To address these issues, the proposed Telesonar transmits short chirps during the vulnerable time of echo cancellation, detects the tiny echo remnants from the received voice, and passively analyzes the timing of caller’s breath sounds to confirm a human caller. Extensive real experiments under a wide range of settings show that Telesonar correctly recognizes human callers with a rate of over 95%, while wrongly recognizing voice robots as human with a rate of 3.8%.

   @inproceedings{telesonar-sensys,
     author = {Yan, Zhenyu and Song, Qun and Tan, Rui and Lu, Chris Xiaoxuan},
     title = {Telesonar: Robocall Alarm System by Detecting Echo Channel and Breath Timing},
     year = {2022},
     booktitle = {The 20th ACM Conference on Embedded Networked Sensor Systems},
     month = nov,
   }

2. SenSys

   BlastNet: Exploiting Duo-Blocks for Cross-Processor Real-Time DNN Inference

   Neiwen Ling, Xuan Huang, Zhihe Zhao, Nan Guan, Zhenyu Yan, and Guoliang Xing

   In The 20th ACM Conference on Embedded Networked Sensor Systems, Acceptance ratio: 52/208=25% , Nov 2022

   Best Paper Award Finalist Abs Bib PDF

   Best Paper Award Finalist, ACM SenSys 2022

   In recent years, Deep Neural Network (DNN) has been increasingly adopted by a wide range of time-critical applications running on edge platforms with heterogeneous multiprocessors. To meet the stringent timing requirements of these applications, heterogeneous CPU and GPU resources must be efficiently utilized for the inference of multiple DNN models. Such a cross-processor real-time DNN inference paradigm poses major challenges due to the inherent performance imbalance among different processors and the lack of real-time support for cross-processor inference from existing deep learning frameworks. In this work, we propose a new system named BlastNet that exploits duo-block - a new model inference abstraction to support highly efficient cross-processor real-time DNN inference. Each duo-block has a dual model structure, enabling efficient fine-grained inference alternatively across different processors. BlastNet employs a novel block-level Neural Architecture Search (NAS) technique to generate duo-blocks, which accounts for computing characteristics and communication overhead. The duo-blocks are optimized at design time and then dynamically scheduled to achieve high resource utilization of heterogeneous CPU and GPU at runtime. BlastNet is implemented on an indoor autonomous driving platform and three popular edge platforms. Extensive results show that BlastNetachieves 45.70% less deadline missing rate with a mere 1.63% of model accuracy loss.

   @inproceedings{blastnet-sensys,
     author = {Ling, Neiwen and Huang, Xuan and Zhao, Zhihe and Guan, Nan and Yan, Zhenyu and Xing, Guoliang},
     title = {BlastNet: Exploiting Duo-Blocks for Cross-Processor Real-Time DNN Inference},
     year = {2022},
     booktitle = {The 20th ACM Conference on Embedded Networked Sensor Systems},
   }

3. SenSys

   AutoMatch: Leveraging Traffic Camera to Improve Perception and Localization of Autonomous Vehicles

   Yuze He, Li Ma, Jiahe Cui, Zhenyu Yan, Guoliang Xing, Sen Wang, Qintao Hu, and Chen Pan

   In The 20th ACM Conference on Embedded Networked Sensor Systems, Acceptance ratio: 52/208=25% , Nov 2022

   Abs Bib PDF

   Traffic camera is one of the most ubiquitous traffic facilities, providing high coverage of complex, accident-prone road sections such as intersections. This work leverages traffic camera to improve the perception and localization performance of autonomous vehicles at intersections. In particular, vehicles can expand their range of perception by matching the images captured by both the traffic cameras and on-vehicle cameras. Moreover, a traffic camera can match its images to an existing high-definition map (HD map) to derive centimeter-level location of the vehicles in its field of view. To this end, we propose AutoMatch - a novel system for real-time image registration, which is a key enabling technology for traffic camera-assisted perception and localization of autonomous vehicles. Our key idea is to leverage distinctive structures such as ground signs at intersections as the domain knowledge to facilitate image registration between traffic camera and HD map or vehicle. Moreover, by leveraging the strong structural characteristics of ground signs, AutoMatch can extract very few but precise keypoints for registration, which effectively reduces the communication/compute overhead. We implement AutoMatch on a testbed consisting of a self-built autonomous car, drones for surveying and mapping, and real traffic cameras. In addition, we collect two new multi-view traffic image datasets at intersections, which contain images from 220 real operational traffic cameras in 22 cities. Experimental results show that AutoMatch achieves pixel-level image registration accuracy within 88 milliseconds, and delivers an 11.65x improvement in accuracy and 1.42x speedup in compute time over state-of-the-art baselines.

   @inproceedings{automatch-sensys,
     author = {He, Yuze and Ma, Li and Cui, Jiahe and Yan, Zhenyu and Xing, Guoliang and Wang, Sen and Hu, Qintao and Pan, Chen},
     title = {AutoMatch: Leveraging Traffic Camera to Improve Perception and Localization of Autonomous Vehicles},
     year = {2022},
     month = nov,
     booktitle = {The 20th ACM Conference on Embedded Networked Sensor Systems},
   }

4. SenSys

   Indoor Smartphone SLAM with Learned Echoic Location Features

   Wenjie Luo, Qun Song, Zhenyu Yan, Rui Tan, and Guosheng Lin

   In The 20th ACM Conference on Embedded Networked Sensor Systems, Acceptance ratio: 52/208=25% , Nov 2022

   Abs Bib PDF

   Indoor self-localization is a highly demanded system function for smartphones. The current solutions based on inertial, radio frequency, and geomagnetic sensing may have degraded performance when their limiting factors take effect. In this paper, we present a new indoor simultaneous localization and mapping (SLAM) system that utilizes the smartphone’s built-in audio hardware and inertial measurement unit (IMU). Our system uses a smartphone’s loudspeaker to emit near-inaudible chirps and then the microphone to record the acoustic echoes from the indoor environment. Our profiling measurements show that the echoes carry location information with sub-meter granularity. To enable SLAM, we apply contrastive learning to construct an echoic location feature (ELF) extractor, such that the loop closures on the smartphone’s trajectory can be accurately detected from the associated ELF trace. The detection results effectively regulate the IMU-based trajectory reconstruction. Extensive experiments show that our ELF-based SLAM achieves median localization errors of 0.1m, 0.53m, and 0.4m in a living room, an office, and a shopping mall, and outperforms the Wi-Fi and geomagnetic SLAM systems.

   @inproceedings{elf-sensys,
     author = {Luo, Wenjie and Song, Qun and Yan, Zhenyu and Tan, Rui and Lin, Guosheng},
     title = {Indoor Smartphone SLAM with Learned Echoic Location Features},
     year = {2022},
     month = nov,
     booktitle = {The 20th ACM Conference on Embedded Networked Sensor Systems},
   }

5. EWSN

   Sardino: Ultra-Fast Dynamic Ensemble for Secure Visual Sensing at Mobile Edge

   Qun Song, Zhenyu Yan, Wenjie Luo, and Rui Tan

   In The 19th International Conference on Embedded Wireless Systems and Networks (EWSN), Acceptance ratio: 14/46=30% , Nov 2022

   Abs Bib PDF

   Adversarial example attack endangers the mobile edge systems such as vehicles and drones that adopt deep neural networks for visual sensing. This paper presents Sardino, an active and dynamic defense approach that renews the inference ensemble at run time to develop security against the adaptive adversary who tries to exfiltrate the ensemble and construct the corresponding effective adversarial examples. By applying consistency check and data fusion on the ensemble’s predictions, Sardino can detect and thwart adversarial inputs. Compared with the training-based ensemble renewal, we use HyperNet to achieve one million times acceleration and per-frame ensemble renewal that presents the highest level of difficulty to the prerequisite exfiltration attacks. Moreover, the robustness of the renewed ensembles against adversarial examples is enhanced with adversarial learning for the HyperNet. We design a run-time planner that maximizes the ensemble size in favor of security while maintaining the processing frame rate. Beyond adversarial examples, Sardino can also address the issue of out-of-distribution inputs effectively. This paper presents extensive evaluation of Sardino’s performance in counteracting adversarial examples and applies it to build a real-time car-borne traffic sign recognition system. Live on-road tests show the built system’s effectiveness in maintaining frame rate and detecting out-of-distribution inputs due to the false positives of a preceding YOLO-based traffic sign detector.

   @inproceedings{song2022sardino,
     title = {Sardino: Ultra-Fast Dynamic Ensemble for Secure Visual Sensing at Mobile Edge},
     author = {Song, Qun and Yan, Zhenyu and Luo, Wenjie and Tan, Rui},
     booktitle = {The 19th International Conference on Embedded Wireless Systems and Networks (EWSN)},
     year = {2022},
   }

6. MobiCom

   VIPS: Real-Time Perception Fusion for Infrastructure-Assisted Autonomous Driving

   Shuyao Shi, Jiahe Cui, Zhehao Jiang, Zhenyu Yan, Guoliang Xing, Jianwei Niu, and Zhenchao Ouyang

   In The 28th Annual International Conference on Mobile Computing and Networking, Syndey, Australia, Acceptance ratio: 56/314=17.8% , Nov 2022

   Best Paper Award Runner-Up Abs Bib HTML PDF

   Best Paper Award Runner-Up, ACM MobiCom 2022

   Infrastructure-assisted autonomous driving is an emerging paradigm that expects to significantly improve the driving safety of autonomous vehicles. The key enabling technology for this vision is to fuse LiDAR results from the roadside infrastructure and the vehicle to improve the vehicle’s perception in real time. In this work, we propose VIPS, a novel lightweight system that can achieve decimeter-level and real-time (up to 100 ms) perception fusion between driving vehicles and roadside infrastructure. The key idea of VIPS is to exploit highly efficient matching of graph structures that encode objects’ lean representations as well as their relationships, such as locations, semantics, sizes, and spatial distribution. Moreover, by leveraging the tracked motion trajectories, VIPS can maintain the spatial and temporal consistency of the scene, which effectively mitigates the impact of asynchronous data frames and unpredictable communication/compute delays. We implement VIPS end-to-end based on a campus smart lamppost testbed. To evaluate the performance of VIPS under diverse situations, we also collect two new multi-view point cloud datasets using the smart lamppost testbed and an autonomous driving simulator, respectively. Experiment results show that VIPS can extend the vehicle’s perception range by 140% within 58 ms on average, and delivers a 4X improvement in perception fusion accuracy and 47X data transmission saving over existing approaches. A video demo of VIPS based on the lamppost dataset is available at https://youtu.be/zW4oi_EWOu0.

   @inproceedings{mobicom22,
     author = {Shi, Shuyao and Cui, Jiahe and Jiang, Zhehao and Yan, Zhenyu and Xing, Guoliang and Niu, Jianwei and Ouyang, Zhenchao},
     title = {VIPS: Real-Time Perception Fusion for Infrastructure-Assisted Autonomous Driving},
     year = {2022},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     booktitle = {The 28th Annual International Conference on Mobile Computing and Networking},
     location = {Syndey, Australia},
     series = {MobiCom '22},
   }

7. TOSN

   Physics-Directed Data Augmentation for Deep Model Transfer to Specific Sensor

   Wenjie Luo, Zhenyu Yan, Qun Song, and Rui Tan

   ACM Transactions on Sensor Networks, Jul 2022

   Just Accepted

   Abs DOI Bib HTML PDF

   Run-time domain shifts from the training phase caused by sensor characteristic variation incur performance drops of the deep learning-based sensing systems. To address this problem, existing transfer learning techniques require substantial target-domain data and incur high post-deployment overhead. Differently, we propose to exploit the first principle governing the domain shift to reduce the demand for target-domain data. Specifically, our proposed approach called PhyAug uses the first principle fitted with few labeled or unlabeled data pairs collected by the source sensor and the target sensor to transform the existing source-domain training data into the augmented target-domain data for calibrating the deep neural networks. In two audio sensing case studies of keyword spotting and automatic speech recognition, PhyAug recovers the recognition accuracy losses due to microphones’ characteristic variations by 37% to 72% with 5-second unlabeled data collected from the target microphones. In a case study of acoustics-based room recognition, PhyAug recovers the recognition accuracy loss caused by smartphone microphone variation by 33% to 80%. In the last case study of fisheye image recognition, PhyAug reduces the image recognition error due to the camera-induced distortions by 72%.

   @article{phyaug-tosn,
     author = {Luo, Wenjie and Yan, Zhenyu and Song, Qun and Tan, Rui},
     title = {Physics-Directed Data Augmentation for Deep Model Transfer to Specific Sensor},
     year = {2022},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     issn = {1550-4859},
     url = {https://doi.org/10.1145/3549076},
     doi = {10.1145/3549076},
     note = {Just Accepted},
     month = jul,
     journal = {ACM Transactions on Sensor Networks},
   }

8. IPSN

   BalanceFL: Addressing Class Imbalance in Long-Tail Federated Learning

   Xian Shuai, Yulin Shen, Siyang Jiang, Zhihe Zhao, Zhenyu Yan, and Guoliang Xing

   In The 21st ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN), Acceptance ratio: 38/126=30.2% , May 2022

   Abs DOI Bib HTML PDF

   Federated Learning (FL) is an emerging learning paradigm that enables the collaborative learning of different nodes without ex-posing the raw data. However, a critical challenge faced by the current federated learning algorithms in real-world applications is the long-tailed data distribution, i.e., in both local and global views, the numbers of classes are often highly imbalanced. This would lead to poor model accuracy on some rare but vital classes, e.g., those related to safety in health and autonomous driving applications. In this paper, we propose BalanceFL, a federated learning frame-work that can robustly learn both common and rare classes from a long-tailed real-world dataset, addressing both the global and local data imbalance at the same time. Specifically, instead of letting nodes upload a class-drifted model trained on imbalanced private data, we design a novel local update scheme that rectifies the class imbalance, forcing the local model to behave as if it were trained on ideal uniform distributed data. To evaluate the performance of BalanceFL, we first adapt two public datasets to the long-tailed federated learning setting, and then collect a real-life IMU dataset for action recognition, which includes more than 10,000 data sam-ples and naturally exhibits the global long tail effect and the local imbalance. On all of these three datasets, BalanceFL outperforms state-of-the-art federated learning approaches by a large margin.

   @inproceedings{balancefl-ipsn,
     author = {Shuai, Xian and Shen, Yulin and Jiang, Siyang and Zhao, Zhihe and Yan, Zhenyu and Xing, Guoliang},
     booktitle = {The 21st ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN)},
     title = {BalanceFL: Addressing Class Imbalance in Long-Tail Federated Learning},
     year = {2022},
     volume = {},
     number = {},
     pages = {271-284},
     keywords = {},
     doi = {10.1109/IPSN54338.2022.00029},
     issn = {},
     month = may,
   }

9. IPSN

   Demo Abstract: An Underwater Sonar-Based Drowning Detection System

   Lixing He, Haozheng Hou, Zhenyu Yan, and Guoliang Xing

   In The 21st International Conference on Information Processing in Sensor Networks (Demo), May 2022

   Abs Bib PDF

   Drowning is a major cause of unintentional deaths in swimming pools. Most swimming pools hire lifeguards for continuous surveillance, which is labor-intensive and hence unfeasible for small private pools. The existing unmanned surveillance solutions like camera array requires non-trivial installations, only work in certain conditions (e.g., with adequate ambient lighting), or raise privacy concerns. This demo presents SwimSonar, the first practical drowning detection system based on underwater sonar. SwimSonar employs an active ultrasonic sonar and features a novel sonar scanning strategy that balances the time and accuracy. Lastly, SwimSonar leverages a deep neural network for accurate drowning detection. Our experiments in real swimming pools show that the system achieves 88% classification accuracy with a scan time of 1.5 seconds.

   @inproceedings{ipsn2022-3d-emr,
     author = {He, Lixing and Hou, Haozheng and Yan, Zhenyu and Xing, Guoliang},
     title = {Demo Abstract: An Underwater Sonar-Based Drowning Detection System},
     year = {2022},
     publisher = {IEEE},
     booktitle = {The 21st International Conference on Information Processing in Sensor Networks (Demo)},
   }

10. IPSN

    Demo Abstract: 3D Simultaneous Localization and Mapping with Power Network Electromagnetic Radiation

    Rongrong Wang, Zhenyu Yan, Rui Tan, and Chris Xiaoxuan Lu

    In The 21st International Conference on Information Processing in Sensor Networks (Demo), May 2022

    Abs Bib PDF

    Indoor localization by leveraging the existing residential instruments has been widely explored. Given the properties of temporal stability and spatial distinctness, the electromagnetic radiation (EMR) from the powerline network is a promising signal for location sensing. In this demo, we present a three-sensor setup to capture the powerline EMR signal from the three-dimensional (3D) space and formulate a new powerline EMR feature to implement the simultaneous localization and mapping (SLAM). Compared with the single sensor setup, our proposed approach can improve the localization accuracy to decimeter level.

    @inproceedings{ipsn2022-3d-ems,
      author = {Wang, Rongrong and Yan, Zhenyu and Tan, Rui and Lu, Chris Xiaoxuan},
      title = {Demo Abstract: 3D Simultaneous Localization and Mapping with Power Network Electromagnetic Radiation},
      year = {2022},
      publisher = {IEEE},
      booktitle = {The 21st International Conference on Information Processing in Sensor Networks (Demo)},
    }

2021

1. IPSN

   PhyAug: Physics-Directed Data Augmentation for Deep Sensing Model Transfer in Cyber-Physical Systems

   Wenjie Luo, Zhenyu Yan, Qun Song, and Rui Tan

   In The 20th International Conference on Information Processing in Sensor Networks, Acceptance ratio: 26/105=24.8% , May 2021

   Best Artifact Award Runner-Up Abs Bib HTML PDF

   Best Artifact Award Runner-Up, ACM/IEEE IPSN 2021

   Run-time domain shifts from training-phase domains are common in sensing systems designed with deep learning. The shifts can be caused by sensor characteristic variations and/or discrepancies between the design-phase model and the actual model of the sensed physical process. To address these issues, existing transfer learning techniques require substantial target-domain data and thus incur high post-deployment overhead. This paper proposes to exploit the first principle governing the domain shift to reduce the demand on target-domain data. Specifically, our proposed approach called PhyAug, uses the first principle fitted with few labeled or unlabeled source/target-domain data pairs to transform the existing source-domain training data into augmented data for updating the deep neural networks. In two case studies of keyword spotting and DeepSpeech2-based automatic speech recognition, with 5-second unlabeled data collected from the target microphones, PhyAug recovers the recognition accuracy losses due to microphone characteristic variations by 37% to 72%. In a case study of seismic source localization with TDoA fingerprints, by exploiting the first principle of signal propagation in uneven media, PhyAug only requires 3% to 8% of labeled TDoA measurements required by the vanilla fingerprinting approach in achieving the same localization accuracy.

   @inproceedings{ipsn2021,
     author = {Luo, Wenjie and Yan, Zhenyu and Song, Qun and Tan, Rui},
     title = {PhyAug: Physics-Directed Data Augmentation for Deep Sensing Model Transfer in Cyber-Physical Systems},
     year = {2021},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     booktitle = {The 20th International Conference on Information Processing in Sensor Networks},
   }

2. TMC

   Touch-to-Access Device Authentication for Indoor Smart Objects

   Zhenyu Yan, Qun Song, and Rui Tan

   IEEE Transactions on Mobile Computing (in press), May 2021

   Abs Bib HTML PDF

   This paper presents TouchAuth, a new touch-to-access device authentication approach using induced body electric potentials (iBEPs) caused by the indoor ambient electric field that is mainly emitted from the building’s electrical network. The design of TouchAuth is based on the electrostatics of iBEP generation and a resulting property, i.e., the iBEPs at two close locations on the same human body are similar, whereas those from different human bodies are distinct. Extensive experiments verify the above property and show that TouchAuth achieves high-profile receiver operating characteristics in implementing the touch-to-access policy. Our experiments also show that a range of possible interfering sources including appliances’ electromagnetic emanations and noise injections into the power network do not affect the performance of TouchAuth. A key advantage of TouchAuth is that the iBEP sensing requires a simple analog-to-digital converter only, which is widely available on microcontrollers. Compared with the existing approaches including intra-body communication and physiological sensing, TouchAuth is a low-cost, faster, and easy-to-use approach for authorized users to access the smart objects found in indoor environments.

   @article{touchauth-tmc,
     author = {Yan, Zhenyu and Song, Qun and Tan, Rui},
     title = {Touch-to-Access Device Authentication for Indoor Smart Objects},
     journal = {IEEE Transactions on Mobile Computing (in press)},
     year = {2021},
   }

3. TOSN

   DeepMTD: Moving Target Defense for Deep Visual Sensing against Adversarial Examples

   Qun Song, Zhenyu Yan, and Rui Tan

   ACM Transactions on Sensor Networks, Oct 2021

   Abs DOI Bib HTML PDF

   Deep learning-based visual sensing has achieved attractive accuracy but is shown vulnerable to adversarial attacks. Specifically, once the attackers obtain the deep model, they can construct adversarial examples to mislead the model to yield wrong classification results. Deployable adversarial examples such as small stickers pasted on the road signs and lanes have been shown effective in misleading advanced driver-assistance systems. Most existing countermeasures against adversarial examples build their security on the attackers’ ignorance of the defense mechanisms. Thus, they fall short of following Kerckhoffs’s principle and can be subverted once the attackers know the details of the defense. This article applies the strategy of moving target defense (MTD) to generate multiple new deep models after system deployment that will collaboratively detect and thwart adversarial examples. Our MTD design is based on the adversarial examples’ minor transferability across different models. The post-deployment of dynamically generated models significantly increase the bar of successful attacks. We also apply serial data fusion with early stopping to reduce the inference time by a factor of up to 5, as well as exploit hardware inference accelerators’ characteristics to strike better tradeoffs between inference time and power consumption. Evaluation based on three datasets including a road sign dataset and two GPU-equipped embedded computing boards shows the effectiveness and efficiency of our approach in counteracting the attack.

   @article{deepmtd-tosn,
     author = {Song, Qun and Yan, Zhenyu and Tan, Rui},
     title = {DeepMTD: Moving Target Defense for Deep Visual Sensing against Adversarial Examples},
     year = {2021},
     issue_date = {February 2022},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     volume = {18},
     number = {1},
     issn = {1550-4859},
     doi = {10.1145/3469032},
     journal = {ACM Transactions on Sensor Networks},
     month = oct,
     articleno = {5},
     numpages = {32},
     keywords = {moving target defense, Deep neural networks, adversarial examples, embedded computer vision},
   }

4. Book Chapter

   Clock Synchronization for Wide-Area Applications

   Zhenyu Yan, Yang Li, and Rui Tan

   Oct 2021
5. SenSys

   Infrastructure-Free Smartphone Indoor Localization Using Room Acoustic Responses

   Dongfang Guo, Wenjie Luo, Chaojie Gu, Yuting Wu, Qun Song, Zhenyu Yan, and Rui Tan

   In The 19th ACM Conference on Embedded Networked Sensor Systems (Demo), New York, New York, Oct 2021

   Abs DOI Bib HTML PDF

   Smartphone indoor location awareness is increasingly demanded by a variety of mobile applications. The existing solutions for accurate smartphone indoor localization rely on additional devices or preinstalled infrastructure (e.g., dense WiFi access points, Bluetooth beacons). In this demo, we present EchoLoc, an infrastructure free smartphone indoor localization system using room acoustic response to a chirp emitted by the phone. EchoLoc consists of a mobile client for echo data collection and a cloud server hosting a deep neural network for location inference. EchoLoc achieves 95% accuracy in recognizing 101 locations in a large public indoor space and a median localization error of 0.5 m in a typical lab area. Demo video is available at https://youtu.be/5si0Cq6LzT4.

   @inproceedings{10.1145/3356250.3360026,
     author = {Guo, Dongfang and Luo, Wenjie and Gu, Chaojie and Wu, Yuting and Song, Qun and Yan, Zhenyu and Tan, Rui},
     title = {Infrastructure-Free Smartphone Indoor
     Localization Using Room Acoustic Responses},
     year = {2021},
     isbn = {9781450390972},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {https://doi.org/10.1145/3485730.3492877},
     doi = {10.1145/3485730.3492877},
     booktitle = {The 19th ACM Conference on Embedded Networked Sensor Systems (Demo)},
     location = {New York, New York},
     series = {SenSys '21 Demo},
   }

2020

1. PhD Thesis

   Exploiting Induced skin electric potential for body-area IoT system functions

   Zhenyu Yan

   Oct 2020

   DOI Bib HTML PDF

   @book{thesis,
     author = {Yan, Zhenyu},
     title = {Exploiting Induced skin electric potential for body-area IoT system functions},
     year = {2020},
     doi = {10.32657/10356/137802},
     publisher = {Nanyang Technological University, Singapore},
   }

2019

1. MobiCom

   Towards Touch-to-Access Device Authentication Using Induced Body Electric Potentials

   Zhenyu Yan, Qun Song, Rui Tan, Yang Li, and Adams Wai Kin Kong

   In The 25th Annual International Conference on Mobile Computing and Networking, Los Cabos, Mexico, Acceptance ratio: 55/290=18.9% , Oct 2019

   Abs DOI Bib HTML PDF

   This paper presents TouchAuth, a new touch-to-access device authentication approach using induced body electric potentials (iBEPs) caused by the indoor ambient electric field that is mainly emitted from the building’s electrical cabling. The design of TouchAuth is based on the electrostatics of iBEP generation and a resulting property, i.e., the iBEPs at two close locations on the same human body are similar, whereas those from different human bodies are distinct. Extensive experiments verify the above property and show that TouchAuth achieves high-profile receiver operating characteristics in implementing the touch-to-access policy. Our experiments also show that a range of possible interfering sources including appliances’ electromagnetic emanations and noise injections into the power network do not affect the performance of TouchAuth. A key advantage of TouchAuth is that the iBEP sensing requires a simple analog-to-digital converter only, which is widely available on microcontrollers. Compared with existing approaches including intra-body communication and physiological sensing, TouchAuth is a low-cost, lightweight, and convenient approach for authorized users to access the smart objects found in indoor environments.

   @inproceedings{10.1145/3300061.3300118,
     author = {Yan, Zhenyu and Song, Qun and Tan, Rui and Li, Yang and Kong, Adams Wai Kin},
     title = {Towards Touch-to-Access Device Authentication Using Induced Body Electric Potentials},
     year = {2019},
     isbn = {9781450361699},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {https://doi.org/10.1145/3300061.3300118},
     doi = {10.1145/3300061.3300118},
     booktitle = {The 25th Annual International Conference on Mobile Computing and Networking},
     articleno = {23},
     numpages = {16},
     keywords = {device authentication, wearables, induced body electric potential},
     location = {Los Cabos, Mexico},
     series = {MobiCom '19},
   }

2. SenSys

   Moving Target Defense for Embedded Deep Visual Sensing against Adversarial Examples

   Qun Song, Zhenyu Yan, and Rui Tan

   In The 17th ACM Conference on Embedded Networked Sensor Systems, New York, New York, Acceptance ratio: 28/144=19% , Oct 2019

   Abs DOI Bib HTML PDF

   Deep learning-based visual sensing has achieved attractive accuracy but is shown vulnerable to adversarial example attacks. Specifically, once the attackers obtain the deep model, they can construct adversarial examples to mislead the model to yield wrong classification results. Deployable adversarial examples such as small stickers pasted on the road signs and lanes have been shown effective in misleading advanced driver-assistance systems. Many existing countermeasures against adversarial examples build their security on the attackers’ ignorance of the defense mechanisms. Thus, they fall short of following Kerckhoffs’s principle and can be subverted once the attackers know the details of the defense. This paper applies the strategy of moving target defense (MTD) to generate multiple new deep models after system deployment, that will collaboratively detect and thwart adversarial examples. Our MTD design is based on the adversarial examples’ minor transferability across different models. The post-deployment dynamically generated models significantly increase the bar of successful attacks. We also apply serial data fusion with early stopping to reduce the inference time by a factor of up to 5. Evaluation based on four datasets including a road sign dataset and two GPU-equipped Jetson embedded computing platforms shows the effectiveness of our approach.

   @inproceedings{10.1145/3356250.3360025,
     author = {Song, Qun and Yan, Zhenyu and Tan, Rui},
     title = {Moving Target Defense for Embedded Deep Visual Sensing against Adversarial Examples},
     year = {2019},
     isbn = {9781450369503},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {https://doi.org/10.1145/3356250.3360025},
     doi = {10.1145/3356250.3360025},
     booktitle = {The 17th ACM Conference on Embedded Networked Sensor Systems},
     pages = {124–137},
     numpages = {14},
     keywords = {adversarial examples, neural networks, moving target defense},
     location = {New York, New York},
     series = {SenSys '19},
   }

2018

1. TMC

   Wearables Clock Synchronization Using Skin Electric Potentials

   Zhenyu Yan, Rui Tan, Yang Li, and Jun Huang

   IEEE Transactions on Mobile Computing, Dec 2018

   Abs DOI Bib HTML PDF

   Design of clock synchronization for networked nodes faces a fundamental trade-off between synchronization accuracy and universality of heterogeneous platforms, because a high synchronization accuracy generally requires platform-dependent hardware-level network packet timestamping. This paper presents TouchSync, a new indoor clock synchronization approach for wearables that achieves millisecond accuracy while preserving universality in that it uses standard system calls only, such as reading system clock, sampling sensors, and sending/receiving network messages. The design of TouchSync is driven by a key finding from our extensive measurements that the skin electric potentials (SEPs) induced by powerline radiation are salient, periodic, and synchronous on a same wearer and even across different wearers. TouchSync integrates the SEP signal into the universal principle of Network Time Protocol and solves an integer ambiguity problem by fusing the ambiguous results in multiple synchronization rounds to conclude an accurate clock offset between two synchronizing wearables. With our shared code, TouchSync can be readily integrated into any wearable applications. Extensive evaluation based on our Arduino and TinyOS implementations shows that TouchSync’s synchronization errors are below 3 and 7 milliseconds on the same wearer and between two wearers 10 kilometers apart, respectively.

   @article{432,
     author = {Yan, Zhenyu and Tan, Rui and Li, Yang and Huang, Jun},
     journal = {IEEE Transactions on Mobile Computing},
     title = {Wearables Clock Synchronization Using Skin Electric Potentials},
     year = {2018},
     volume = {18},
     number = {12},
     pages = {2984-2998},
     keywords = {bioelectric potentials;body sensor networks;clocks;protocols;synchronisation;telecommunication network reliability;wearables clock synchronization;skin electric potentials;networked nodes;synchronization accuracy;heterogeneous platforms;high synchronization accuracy;platform-dependent hardware-level network packet timestamping;indoor clock synchronization approach;system clock;network messages;SEP signal;universal principle;Network Time Protocol;integer ambiguity problem;multiple synchronization rounds;wearable applications;TouchSync's synchronization errors;time 3.0 ms;time 7.0 ms;Synchronization;Sensors;Calibration;Mobile computing;Hardware;Clock synchronization;skin electric potential;wearables},
     doi = {10.1109/TMC.2018.2884897},
     issn = {1558-0660},
     month = dec,
   }

2017

1. SenSys

   Application-Layer Clock Synchronization for Wearables Using Skin Electric Potentials Induced by Powerline Radiation

   Zhenyu Yan, Yang Li, Rui Tan, and Jun Huang

   In The 15th ACM Conference on Embedded Network Sensor Systems, Delft, Netherlands, Acceptance ratio: 26/151=17% , Dec 2017

   Abs DOI Bib HTML PDF

   Design of clock synchronization for networked nodes faces a fundamental trade-off between synchronization accuracy and universality for heterogeneous platforms, because a high synchronization accuracy generally requires platform-dependent hardware-level network packet timestamping. This paper presents TouchSync, a new indoor clock synchronization approach for wearables that achieves millisecond accuracy while preserving universality in that it uses standard system calls only, such as reading system clock, sampling sensors, and sending/receiving network messages. The design of TouchSync is driven by a key finding from our extensive measurements that the skin electric potentials (SEPs) induced by powerline radiation are salient, periodic, and synchronous on a same wearer and even across different wearers. TouchSync integrates the SEP signal into the universal principle of Network Time Protocol and solves an integer ambiguity problem by fusing the ambiguous results in multiple synchronization rounds to conclude an accurate clock offset between two synchronizing wearables. With our shared code, TouchSync can be readily integrated into any wearable applications. Extensive evaluation based on our Arduino and TinyOS implementations shows that TouchSync’s synchronization errors are below 3 and 7 milliseconds on the same wearer and between two wearers 10 kilometers apart, respectively.

   @inproceedings{touchsyncsensys,
     author = {Yan, Zhenyu and Li, Yang and Tan, Rui and Huang, Jun},
     title = {Application-Layer Clock Synchronization for Wearables Using Skin Electric Potentials Induced by Powerline Radiation},
     year = {2017},
     isbn = {9781450354592},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {https://doi.org/10.1145/3131672.3131681},
     doi = {10.1145/3131672.3131681},
     booktitle = {The 15th ACM Conference on Embedded Network Sensor Systems},
     articleno = {10},
     numpages = {14},
     keywords = {Clock synchronization, skin electric potential, wearables},
     location = {Delft, Netherlands},
     series = {SenSys '17},
   }