Surrey University, Author at Ģ��ý.

iLIDS Research – Part 3.

Surrey University — Fri, 07 Dec 2018 09:44:41 +0000

Revealing your true colours with facial recognition

‘iLIDS’ stands for ‘Interactive Locations and Intelligent Digital Signage’. It is part of the Department for Transport’s £9m Innovate UK ‘Accelerating Innovation in Rail’ programme under which Enable iD, The Rail Safety and Standards Board (RSSB), Arriva UK Trains and research partners the University of Surrey are researching, developing and testing new technology to improve information, safety and accessibility on railway station platforms.

Overview

The original iLIDS project intended to provide personalised information to passengers on the platform/train. The way to achieve this level of personalisation could be achieved with a number of different technologies including Bluetooth beacon/WIFI sensors, which would detect a person’s mobile handset or supplied Bluetooth beacon, or by using biometrics like finger/palm print or facial recognition.

This summer project looked at the feasibility of using facial recognition to identify passengers so that relevant information could be supplied to them when standing in front of a “smart” display fitted with a webcam. Alternatively, this technology could be combined with a smart ticketing solution that provides additional information at the ticket gate (eg it states the platform the next appropriate train departs from)

Approach

The project main brief was to investigate how best to manage the privacy of users using facial recognition within a transport systems context. As part of the project, a prototype was developed that would use a local camera to send an encrypted video/image stream to a server which would carry out the actual facial recognition and return appropriate user details potentially including personalised information about their journey. To ensure the privacy of the users, the server-side computation should be carried out in such a way that the image data was ideally not accessible to the cloud platform on which the code was executed.

The architecture that was chosen is presented in Figure 7. In order to minimise the bandwidth requirements, the IoT device is assumed to be capable of running a simple face detection (different from recognition!) algorithm. For example, a Raspberry Pi 3B+ is capable of carrying out face detection in about 400ms while an Intel Core i5-6600 CPU @3.3GHz takes about 25-40ms. Note that the communication between the webcam and the server utilised the code developed as part of the object-recognition project.

Once a face has been detected, it is extracted from the image and sent via an encrypted channel over the internet to an application server hosted by a (cloud) service provider. This has been done to reduce the bandwidth requirements drastically. Instead of transmitting a video stream of say 25 frames of VGA (640×480 pixels) video, now only a few small images of roughly 200×200 pixels get transferred for further processing by the application server.

The application server then passes the image to the face recognition algorithm which in turn will return a label identifying the passenger together with a confidence score which can be used to look up the customer details in the encrypted customer database provided the confidence score is above a certain threshold, eg >85%.

Note that the facial recognition algorithm relies on a trained model which is loaded on start-up. Since the algorithm (MobileNetV2 [4] provided as part of the TensorFlow library [5]) used has a very small footprint, the amount of data stored is in the range of 200kb. No image data used to train the model is ever stored on the application server.

New faces are added to this model using an off-line server controlled by the train operator or the organisation responsible for running the service. This is a daily batch process that adds new passengers who have signed up to the service to the facial recognition model using Transfer Learning which uses the existing pre-trained model and retrains it to recognise the new faces instead of training it again from scratch. This has the advantage that training time is reduced from days to hours. The new model can then be uploaded to the cloud server in an encrypted format and the facial recognition app will reload the model during the night.

The customer database also is also encrypted when at rest and only decrypts its contents on start-up.

Consequently, all data (customer database and facial recognition model) is encrypted at rest and only held in unencrypted form in memory.

Image data is only ever held in memory and never persisted. Once the facial recognition algorithm has returned a label5.3.

Figure 7: Facial Recognition System Architecture

For the prototype, the application server also hosted a web application that displayed a simple web page which displayed the uploaded image and the corresponding data extracted for the label associated with the image. This application effectively represented the potential processing that could be done given the passenger information which could be as simple as just relaying the users season ticket details to open the gate or a detailed travel plan including train information for the next train for a passenger standing in front of a smart display.

The application server architecture is shown in Figure 8.

Figure 8: Application Server Design

A docker image [6] was used to avoid the installation of a significant number of third-party libraries on the server. The Tensorflow docker image contains an implementation of the MobileNetV2 algorithm. For simplicity, the re-training of the model is done on the same machine but in a proper deployment, this would usually take place on a separate server which the train operating company would control to avoid storing any images of passengers on a cloud server.

Prototype Evaluation

The implementation faced a number of challenges. The initial approaches for facial recognition were based on Machine Learning approaches using key feature extraction algorithms including David Lowe’s SIFT [7], SURF [8], HoG [9] but their performance was less than satisfactory. One critical aspect was the lighting conditions under which the algorithms were able to perform. For the best performance, a well-lit subject is important but more work needs to be done to determine how the performance degrades given different levels of light.

Given these empirical results, it was decided to use a Deep Neural Network (DNN) approach in the form of the MobileNetV2 implementation within the TensorFlow project which performed much better but again more work needs to be done to establish the exact operational constraints under which the false positive (recognising someone as someone else) and false negative(not recognising someone when it should) rates detection rate are acceptably low. Note that from a revenue protection point a false positive is probably worse than a false negative. However, a false negative has a negative impact on the customer experience and might reduce trust in the system resulting in reputational damage.

Conclusion and Future Work

The facial recognition summer project showed that the performance of the face recognition algorithm was sufficiently fast for use with a smart display but probably needs more work to operate in a ticket gate scenario where throughput is critical.

The other outstanding piece of work is to look at moving the server-side into a trusted environment like Intel’s SGX enclaves to ensure that all sensitive information like images is only ever held in an encrypted format even in memory.

��ܳ��

The 3 summer projects have looked at potential technologies and approaches to:

provide functionality to the iLIDS project that got descoped during the initial requirements gathering phase as too ambitious for the timescales of the project (facial and object recognition)
fill information gaps which are critical to the successful roll-of the iLIDS solution (providing train consist information)

The prototypes and their evaluation produced as part of these internships are intended to inform any corresponding future enhancements and their requirements to the iLIDS project going forward.

About – Surry Centre for Cybersecurity

Surry centre of cybersecurity is one of the 17 Academic Centres of Excellence in Cyber Security Research (ACEs-CSR) recognised by the UK National Cyber Security Centre (NCSC) in partnership with the Engineering and Physical Sciences Research Council (EPSRC).

Surry focuses on “Security by Design”, with particular expertise in trusted systems, privacy and authentication, secure communications, and multimedia security and forensics, with human dimensions in security as a cross-cutting theme. We have a strong ethos of applying theory into practice, and the application domains we work in include transportation (automotive, rail), democracy (e-voting), telecommunications, digital economy, and law enforcement.

The post iLIDS Research – Part 3. appeared first on Ģ��ý..

iLIDS Research Part 2

Surrey University — Thu, 30 Aug 2018 15:42:25 +0000

Exploiting the power of smart object recognition

Overview

The main motivation for the final project was the original iLIDS requirement to detect the presence of wheelchair users, bicycles, people with luggage/prams to alert staff of potential passengers who might have assistance needs as well as offering intelligent lighting or display solutions which would provide personalised information to these groups of passengers.

Another aspect was to carry out the processing of this information in as much a privacy-preserving manner as possible.

Approach

The first part of the project was to ensure that any data from the camera was sent over an encrypted channel. To limit the bandwidth required the receiver could control how much data was sent by the data feed, eg it could request just a single frame or the whole video stream. The client-server architecture of the image transfer application is shown in Figure 4.

Figure 4: Client-Server Architecture of the Video feed

Once the server has received the image, it is passed to the Object Recognition Application. The algorithm was developed to be run such that it could be run on a CPU-only server as well as take advantage of GPU acceleration if a supported graphics card was available.

The object recognition algorithm used was YOLO v3 [3]which was chosen for its claimed speed and accuracy as shown in Figure 5

Figure 5: YOLOv3 performance (from [3])

The application then reported the detected objects and associated confidence levels on a website to illustrate the output generated. The following image is an example output that the program generated:

Figure 6: Sample output of Object Recognition Algorithm

The final part of the project was to use Intel’s Software Guard Extensions (SGX) to protect the processing of the images on an untrusted host (eg a server provided by a cloud provider).SGX is a technology which allows programs to be executed in encrypted areas of memory (enclaves) to protect their data (and code) from the underlying operating system. It promises to provide a secure environment even if the underlying platform is untrusted.

Prototype Evaluation

The following table shows the performance (measured and interpolated) of the object detection algorithm (values with a question mark are estimations):

	CPU-ONLY (SINGLE THREAD)	CPU-ONLY (MULTIPLE THREADS)	CPU-WITH SGX ENABLED (SINGLE THREAD)	CPU WITH SGX AND MULTIPLE READING	GPU ACCELERATED
Full version of YOLOv3	8000ms	2400ms	87500ms?	26000ms?	30ms
Tiny version of YOLOv3	750ms	300ms	8200ms	3250ms?	5ms

Table 1: Performance of Object Recognition Algorithm (CPU i7-4790@4GHz, GPU: Nvidia GTX 980ti)

The tiny version of YOLOv3 is a simpler model which is less accurate but can be run in memory and processor-constrained environments like mobile phones. The reason why it was included was due to the current memory restrictions on SGX enclaves which limit them to 128Mb of memory but can use more using paging. The tiny version of YOLOv3 requires only 38Mb of data to initialise the model while the full version requires over 240Mb. The actual memory consumption is even higher so that even the tiny version of YOLOv3 required paging to run in the SGX enclave. Note that this explains the poor performance using SGX as the processor constantly needed to swap memory in and out of the enclave during the running of the program.

Nevertheless, the prototype proved that object recognition can be carried out on at reasonable speeds using the tiny version of YOLOv3 even on CPU only servers while a GPU accelerated version is several magnitudes faster.

For the problem of detecting specific objects or number of people on a platform, the algorithm only needs to process an image every few seconds as the objects in questions are likely to be reasonably stationary for a considerable amount of time. As such even the CPU the only version of YOLOv3 could be sufficient (especially the tiny version).