Most recent smartphones are equipped with various advanced sensors, such as cameras, accelerometer, gyroscope, GPS, barometer, magnetometer, microphones, ambient light, infrared and proximity sensors that can be used for different application areas. For security and device authentication, many modern smartphones use biometric sensors including fingerprint scanner, iris scanner and face camera.

Some of the smartphone embedded sensors can provide measurements of the user’s movement. Since many people carry their phones almost all of the time, there is a question as to whether these sensors can be used to model a person’s movements and identify them uniquely, which could then potentially lead to applications in a border control scenario.

Why use smartphone sensors?

The current common person verification methods are passwords, PINs, and biometric recognition (e.g., fingerprint, face, iris, and voice recognition). These approaches are one-time verification methods in that the verification process is only performed once at the beginning of a session (e.g., to unlock the phone, log into mobile banking, etc.).

In contrast, a continuous person authentication process will enforce access control during the entire work session, constantly verifying the user’s identity at a selected frequency based on the system requirements. This can be useful, for example, for preventing an unauthorised user from gaining access to the system by temporarily taking possession of a device from an already authenticated user. Smartphone multisensory readings can provide continuous authentication whenever the person is with the phone – even if it’s not actively in use and is merely left in their jacket pocket during the whole session. A change in user’s motion patterns can help identify unauthorised use of the mobile device. It is unintrusive and also naturally spoofing-resistant as human motion or activities are very difficult to be copied due to the many degrees of freedom and differences in body flexibility, etc.

Continuous person verification for border control

Continuous person authentication is a relatively new research topic. It can be applied in domains that require higher security such as mobile banking, however, there has been little exploration in utilizing the techniques for border security scenarios.

There are many challenges with applying this approach to border security. Firstly, there are not many public datasets or other resources available to train or validate the approach – this was the motivation for organising a D4FLY data collection event that took place in the UK in October 2020. In addition, sensor data is generally very noisy (in signal processing, noise is a general term for unwanted modifications that a signal may suffer during capture, storage, transmission, processing or conversion[1]) which can make it difficult to learn the underlying motion patterns distinct from the sensor noise.

Despite these challenges, the advantages of smartphone sensor-based person verification, particularly that people carry smartphones all the time and that the verification result is extremely difficult to spoof, make it an attractive possibility for identity verification. In D4FLY, we are exploring the potential of using smartphone multisensory data to assist the on-the-move biometric verification process, and hope to complement the traditional biometric methods, giving more confidence to the overall verification result.

This technology could be envisioned to be integrated in the D4FLY Automated On-The-Move Border Check scenario. In this scenario, travellers will first consent to enrol in the D4FLY system and voluntarily install the D4FLY app. As they disembark from an airplane, the D4FLY TSSV (Traveller Smartphone Sensor Verification) app will be initiated by the D4FLY main smartphone app to begin the process of continuous identity verification. A machine learning model embedded in the app will compare the multi-sensor data gathered by the smartphone as the traveller walks through the border crossing area with the traveller’s previously enrolled patterns. This process will be passive from the traveller’s perspective once the app is activated and will require no further input from them as they proceed through a biometric verification corridor installation equipped with smart sensors.

Once the traveller arrives at the crossing point, the results of the model will be used as an independent biometric trait and combined (or “fused”) with other biometric traits captured during the crossing to produce a final biometric verification result. The D4FLY TSSV app is currently under development and will be demonstrated and validated in a real border crossing environment during the final project period.

At the same time, the traveller’s enrolment data is encrypted and stored on their own device, meaning that the process is privacy-enhancing and inherently GDPR compliant. Furthermore, travellers can cancel their enrolment and delete the app and stored data from their device at any time. Consequently, continuous authentication via mobile phone sensors promotes traveller’s autonomy while safeguarding their privacy [2].

This technology is, as mentioned above, a relatively new research area and the goal of the activity is to explore the potential of using this technology in a border crossing context. The technical feasibility is being investigated alongside the data privacy and potential ethical aspects described in the PIA (Privacy Impact Assessment) and EIA (Ethical Impact Assessment) that were produced within the frame of the project, ensuring that privacy and ethics are considered from the earliest stages of the experiments and research.

[1] Vyacheslav Tuzlukov (2010), Signal Processing Noise, Electrical Engineering and Applied Signal Processing Series, CRC Press. 688 pages. ISBN 9781420041118

[2] To ensure the ethical and legal compliance of this and all D4Fly tools, the project has undertaken privacy and ethical impact assessments.