Microsoft Project15 & University of Oxford Capstone Project with Elephant Listening Project Team 2

by Contributed | Apr 8, 2021 | Technology

This article is contributed. See the original author and article here.

Team 3. University of Oxford Microsoft Project 15 and Elephant Listening Project Capstone

Project Title. Gunshot Detection in Tropical African Forests

Introduction

This project is a group exercise undertaken as part of the University of Oxford Artificial Intelligence: Cloud and Edge Implementations course, as a learning challenge from Microsoft Project 15 and in association with the Elephant Listening Project. The objective is to devise solutions against illegal elephant hunting in tropical African forests by enabling sensors for instant prediction of gunshot events and thus mitigate poaching attempts.

Project Resources

•  Project Report


•  Project Repository

Meet the Team

Our team is an interesting mix of technologists with a wide range of experience from software engineers, data scientists, dev-ops engineers and solution architects.

Sachin Mathew Varghese	Prasad Deshpande	Loc Nguyen	Menelaos Malaxianakis
Kennedy Mumba	Eden Bodkin	Ashwin Mylavarapu

Elephant Listening Project Challenge

As proposed by Dr. Peter Wrege, Director Elephant Listening Project, Center for Conservation Bioacoustics, Cornell Lab of Ornithology, gunshot detection is an issue in African National Parks and environmental audio data is being collected into a public database at Congo Soundscapes as part of the Elephant Listening Project.

The current model (for gunshot detection) is very inefficient – less than .2% of tagged signals are gunshots and we typically get 10K- 15K tagged signals in a four-month deployment at just one of the 50 recording sites. The issue is we have about 200 good gunshots annotated, but because poaching is way too high, gunshots are still extremely rare in the sounds and it is extremely time-consuming to create the “truth” logs where we can say that every gunshot in a 24hr file has been tagged. From our understanding, this makes developing a detector more difficult.

How you approached the challenge

We approached the challenge by individual research about the problem in general and we gathered ideas on converting the gunshot detection problem statement into a machine learning problem. The main issue as per the challenge is the lack of tagged data for model training and also mostly the gunshot audio data for such use cases is proprietary. So we collected free gunshot samples and other environmental audio samples from random internet sources. There was some basic audio cleaning done to remove noise and clip exact audio data points post converting all formats to .wav files into the dataset. The further audio analysis was carried out using Azure Machine Learning Cloud Services. For brevity, only a part of the dataset is uploaded in the repo but all examples, notebooks, and the dataset are structured in a way to accommodate more data and scale the model training process as we move ahead.

Many features were analyzed across two different classes namely gunshot and environmental audio. The environmental audio contains audio data of elephant noises and other sounds of the tropical African forests. Some of the features analyzed are as follows,

• Spectral Centroid


• 13 MFCCs


• Zero-Crossing Rate


• Onset Detection Frequency

There could be more features relevant to this classification problem that can be analyzed to improve the model in the future and for the scope of this exercise the first thirteen Mel-Frequency Cepstral Coefficients suited best as input features for audio classification.

The input to the machine learning model will be a set of audio features and the model has to classify each input set into two classes, i.e gunshot or environmental audio. This is a binary classification problem and a suitable Dataset was created for training purposes. A feature extraction script was created to convert the raw audio .wav files into relevant feature data and corresponding true values for classification.

Azure Machine Learning cloud service has been leveraged for the classification model supervised training. In this case, two different model architectures were tested by running scripts from the Makefile and the following results were observed,

Note: These accuracy metrics depend directly on the limited data used for model training and testing. Further improvements in contextual data collection, feature extraction and analysis, and experimentation with model architectures is needed to validate the reliability of these model metrics for practical applications.

Solution Architecture – Microsoft Project 15 – Azure Cloud Services

The Microsoft Project 15 Architecture is leveraged to follow the best practices in the deployment of scalable IoT solutions. One of the core goals of the Project 15 Open Platform is to boost innovation with a ready-made platform, allowing the scientific developer to expand into specific use cases.

Data Scientists can train the model using Azure Machine Learning cloud service and deploy the model to edge devices using IoT services. This process can be automated using Azure cloud services for upgrading the edge devices models at runtime through a strategic rollout.

Event processing for Gunshot detection

• Edge devices continuously read audio data, extracts features, and makes ML inference to classify events as gunshot or not.


• Edge devices at multiple locations in the National park then, sends this telemetry data to Event Hub through the IoT gateway


• If an alert is required to be sent then it will be sent to Azure Notifications Hub


• We could also write the events to a structured database for long-term persistence and audit purposes later.


• Azure apps or Power BI monitoring dashboard reads and displays live telemetry data from multiple sensors at different locations.


• Rangers can monitor the notifications and live dashboard for threats.

Edge Device Simulation

For demo purposes, we have created an edge device prototype that simulates predictions from audio data and sends telemetry data to IoT Hub. This is a nodejs based application that uses tensorflow as a backend to make predictions using the model created during the training phase.

The model created during the training phase needs conversion to a tensorflow js graph model format to be used in a node js application. So we convert the models to tfjs graph format for the client to predict

Register an edge device to IoT Hub and copy IoT device connection string from the microsoft portal into .env file in the edge device source folder. Finally, run the edge device prototype to simulate gunshot predictions randomly and send telemetry data to IoT Hub

We found the challenge very interesting to begin with but finally to see the end to end prototype working as intended was extremely satisfying. Through this journey we learnt many new capabilities offered by the Azure Cloud for Model development, deployment and monitoring or the full MLOps lifecycle.

Live Monitoring and Demo

• Monitor gunshot detection from the Project 15 app
  
   
  
  

   

  
  

   

  
  

Future Work

Gunshot detection is still an unsolved problem today and there is ongoing research on this front for both military and surveillance purposes around the world. The machine learning classification model in this context can be improved with better data collection from the tropical forests in Africa. The lack of sufficient data remains the major issue with this challenge. Accurate gunshot audio should be recorded using the apposite set of hunting rifles for improved accuracy and further feature analysis can be conducted comparing gunshots with other surrounding sounds in the elephant habitat.

Further, many more model architectures can be tested and compared to understand what hyper-parameters work best for such remote environments. This can surely improve the model training process. Also, better deployment practices, telemetry, alert systems, and model upgrade processes can be explored with cloud-based IoT solutions to improve the overall efficiency of the solution.

Project GitHub Repo
https://github.com/Oxford-ContEd/project15-elp 

Microsoft Project15 & University of Oxford Capstone Project with Elephant Listening Project Team 2

by Contributed | Apr 8, 2021 | Technology

This article is contributed. See the original author and article here.

MICROSOFT AI FOR GOOD

PROJECT TITLE: A Baseline Elephant Infrasound Detection architecture using IoT Edge, Cloud and Machine learning frameworks for elephant behavioural monitoring.

Group 2:

Authors: Shawn Deggans, Aneeq Ur Rehman, Tommaso Pappagallo, Vivek Shaw, Giulia Ciardi, Dnyaneshwar Kulkarni

Shawn Deggans	Aneeq Ur Rehman	Tommaso Pappagallo
Vivek Shaw	Giulia Ciardi	Dnyaneshwar Kulkarni

INTRODUCTION

The goal of this project is to create a platform that will allow researchers to continuously build and refine machine learning models that can recognize elephant behaviour from infrasound. This platform is based on Microsoft’s Project15 architecture- an open-source solution platform built to aid conservation and sustainability projects using IoT devices.

We aim to apply this technology stack to the tracking and understanding of elephant behaviour to contribute to an effective monitoring program. Our solution aims to achieve this goal by:

1. Systematically collecting data related to elephant vocalization via IoT devices.


2. Detect threats and infer behavioural patterns based on elephant vocalization by using custom machine learning and signal processing modules deployed on the IoT device using Azure IoT stack.                                                            

Furthermore, Elephants’ low-frequency vocalizations (infrasound) are produced by flow-induced self-sustaining oscillations of laryngeal tissue [1]. This unique anatomy allows elephants to communicate in a sonic range that is below the audible human hearing range (14-24hz). This means that a standard microphone will not be an adequate detect for elephant vocalization. The normal range for standard microphones is 20hz-20khz. Specialized microphones can capture sound in the range of 1hz and above. Acquiring these specialized microphones and infrasound detection tools are beyond the scope of this project, in our approach we will outline how such a tool could be created and used in the field by rangers and researchers in our proposed architecture below.

The design of this architecture can be built upon in the future, to deploy more advanced custom modules and do more advanced analytics based on these collated vocalizations. We propose an end-to-end architecture that manages these needs for the researchers and domain experts to analyse and monitor elephant behaviour and identify potential threats and take timely action.

PROPOSED  APPROACH

The above follows the overall proposed flow of the device-to-cloud communication, as well as the IoT Edge’s internal device logic.

1. Capture Infrasound using infrasound detection arrays on elephants:

The infrasound capture device continuously monitors for infrasound. These infrasound signals are captured and sent to the device for analysis. We are interested in exploring one type of infrasound capture device:

• Infrasound detection array

The infrastructure detection array is the capstone project of a Spring 2020 project by team members: James Berger, Tiffany Graham, Josh Wewerka, and Tyler Wyse. Project details can be explored here, “[2]“

This project was designed to detect infrasound direction using an array of infrasound microphones, but we would use the infrasound microphones in a low-powered device that could communicate over a long-range radio network, such as LoRaWAN. This network of devices would serve as the primary means for tracking elephants and recording their vocalization.

We envision this device as an IoT Edge device capable of supporting Azure Storage Account Blob Storage and custom IoT Edge Modules that are containerized machine learning modules capable to converting audio into spectrograms, and spectrograms into bounding box marked images, and delivering telemetry to IoT Hub.

2. Send infrasound data to IoT Device:

The 64-bit Raspberry Pi model captures the infrasound audio signals, breaks these into segments based on detected beginning-of-signal and end-of-signal indicators. Initially this limitation could mean we miss “conversations,” because this is the equivalent of capturing a word from each sentence in human speech. Elephants do not necessarily speak in words and sentences, so this likely will not keep us from meeting the goals of the system.

3. Storing data via Azure Blob Storage: 

These audio files will be saved to an Azure Blob Storage device module in an audio clips file [3].

4. Azure Functions and Spectrogram conversion via the FileWatcher Module. 

We develop a custom IoT Edge module known as the filewatcher module. This module uses an Azure Function to serve as a file watcher for the file storage. [4]

When it detects the file has been received in the blob storage account, it will convert the audio file to a spectrogram.

A MATLAB script is used to convert the raw audio files to spectrograms. The details of this script are available on our github repository. MATLAB offers better resolution images and a higher fidelity power spectrum that improves our elephant rumble detection process.

These spectrograms are stored in an image inbox in the blob storage.

5. Elephant Rumble Detection Module: 

Another Azure function listens to the image inbox for new images and keeps a track of the images uploaded here.

When new images appear, the function feeds the images to the rumble detection model.

The rumble detection model is trained via the custom vision ai using the spectrogram images from the raw audio files.

The custom vision AI is a good place to prototype and to build object detection and image classification models.

An example of steps to build a custom computer vision model and deploy to IoT Edge can be found here via DevOps (https://github.com/aneeqr/DevOpsCICD_final)

The rumble detection model returns the image label (rumble or no rumble) and the bounding box coordinates. We use OpenCV to draw the coordinates on the image. Images are then saved to the Bounding Box Images folder for more backend processing within Azure ML to help feed the ML loop.

We implement this logic flow as part of a DevOps pipeline. For more detail on the pipeline components, please see technical design.

6. Elephant Spectrogram Feature Extraction and Analysis:

In addition to the above, we also extract features from the created spectrograms in our Azure storage accounts.

For our feature extraction we calculated 19 different variables with relation to flow statistic on the audio (mean, variance, etc.) and common signal characteristics (rms, peak2peak, etc.) making use of the MATLAB software here which enabled a wider array of spectral features to be extracted.

In summary for each sample, we found the frequency of the peak magnitude across the overall power spectrum, the 1st formant, the 2nd formant, the max, min, mean and finish of the fundamental. In addition, we computed the power in the low frequency range 0-30 Hz, mid frequency range 30-120 Hz and high frequency range >120 Hz. We located the frequency of the spectral kurtosis peak and the flow cumulative sum range.

7. Analysis in the ML Workspace using stored data: 

Our results offer some starting points for further research particularly in relation to trying to learn any underlying structure, with regards to age groups, in our population, given unlabelled data.

By unlabelled data, we mean that the raw audio files have no annotations or additional information associated with it to classify or distinguish elephant rumbles into maturity groups.

We classify elephant rumbles into maturity groups by

1. Leveraging the extracted features such as the 1^st  and 2^nd formant frequencies, rms, peak to peak envelopes, max, min, finish, frequencies for each sample- We then aim to classify the samples into Maturity Groups 1 or 2 We achieve this by observing the thresholds of these features extracted by setting these thresholds as proposed in this paper [5]. We also do a visual inspection by plotting these features out doing a visual inspection using various data visualisation techniques. An example of a Violin plot is shown below.

The violin plot clearly shows that at certain thresholds, we can cluster elephants into two different maturity groups.

2. We use unsupervised methods such as PCA to generate principal components and perform K-means clustering. This method showed clear signs of clustering when the points were projected into the PCA dimensions as seen in the output below. 

When comparing method 1 and 2 the classified maturity groups overlapped for over 75% of the data, however, to check whether this is statistically significant we would need to perform a t-test.

PROJECT OVERVIEW VIDEO

MLOPS TECHNICAL DESIGN

The above is the MLOPs pipeline as part of our workflow. We use the continuous integration and continuous deployment feature of azure DevOps to manage our workflows.

The following are some of the steps in this pipeline.

1. Creation of Azure Resources:

Resource Group:

Needed to create all our resources in azure.

IoT Hub:

IoT Hub is used as the primary IoT gateway. From here we manage device provisioning, IoT edge device creation, telemetry routing, and security. We will use the SAS token security and employ a regular SAS key rotation policy to keep devices secure.

IoT Edge device:

This is created in the IoT Hub and acts as our IoT device.

IoT Edge RunTime Agent:

This will be deployed on a raspberry pi OS VM container.

Azure Blob Storage:

Blob Storage holds images saved on the IoT Edge Module. This means that it also has folders that represent audio clips, image inbox, and bounding box images. This represents cold storage of data that could be studied later or needs to be archived.

Azure Container Registry: 

Container Registry is used to hold the Dockerfile images that represent the IoT Edge Modules. Modules are pulled by [1] IoT Hub to automatically deploy to devices in the field. All our custom modules mentioned previously contain docker files which are pushed to azure container registry as part of the pipeline.

Azure DevOps: 

Azure DevOps is one of the most important pieces of our MLOPs process. From here code is checked in and pushed out to the Container Registry. This also accounts for version control for our code base and manages pipelines. All docker files for both modules are located here and DevOps offers a continuous integration and continuous deployment scenario, as builds the code for the two custom modules, and generates a deployment manifest which is deployed on the IoT Edge device in the IoT Hub.

2. Import source code in DevOps:

We import our code files and repos in DevOps from our source repository.

3. Create a CI pipeline in DevOps:

The pipeline is then created with all environment variables defined. Environment variables include details of container registry, IoT Hub to build and push our modules to the azure container registry. We enable continuous integration.

4. Enable CD pipeline in DevOps:

We then create a release pipeline and push our deployment to the IoT Hub and hence the IoT Edge device [6].

The device is now ready to work in production.

TECHNICAL STACK AND CORE CONCEPTS APPLIED:

• Azure Custom Vision AI Portal


• Azure DevOps.


• Azure Machine learning.


• Azure Blob Storage


• Azure IoT Edge


• Azure IoT Hub


• Azure Container Registry


• Docker


• Matlab


• Python


• Jupyter notebooks


• Digital Signal Processing


• Data visualisation

CHALLENGES FACED: 

• Procurement of physical device/microphone to detect elephant rumbles.


• Unlabelled data- Lack of ground truth available to classify elephant rumble type.


• Spectrogram resolutions
  
  
  
  •  Need to be good enough for the training of the object detection model in custom vision ai portal.
  
  
  • We applied a variety of data augmentation methods using the Augmentor software package [7]. Some transformations yielded better results than the other but this needs to be explored in more detail.
  
  
  
  


• Noise Separation- Separating elephant rumbles from background noise.


• After research, we understand that our approach of analysing infrasound is of a limited scope on the overall communication amongst the elephants. The sounds are important, but like humans, elephants express their communication in more ways than just their voices. Elephants also express body language (visual communication), chemical communication, and tactile communication. We understand that this means the audio only component will not be able to capture the full picture. It will, however, add to the Elephant Sound Database.


• The MATLAB packages locked us into a specific vendor and made it impossible for us to use the libraries in an IoT Edge container. This was due to licensing restrictions. We used MATLAB as the spectrogram resolutions were promising.

FUTURE WORK

• Polishing baseline code to a more production ready code base.


• More literature review and access to labelled data for building machine learning models.


• Use of semi-supervised approaches to develop a clean and annotated dataset for rumble classification. Validating this dataset with domain experts to serve as a gold standard.


• Use of denoising auto-encoders for background noise cancellation in audio files.


• Use deep learning (CNNs and graph CNNs) for extracting spectrogram data when building models instead of feature engineering. Comparison of both approaches.


• Use of advanced unsupervised methods to get more insights on the data extracted from spectrograms.


• Investigate sustainability and scalability of the proposed architecture in more detail.


• Investigate paid and open-source datasets available for ML.


• Improve existing custom vision ai model by:
  
  
  
  •  Incorporating more training data and researching data augmentation strategies for spectrogram data.
  
  
  • Using better data augmentation methods via spectrograms such as the following SpecAugment package [8]. Having a unified platform via python for improving spectrogram resolutions would be preferred.
  
  
  •  Algorithm development around bounding box detection of fundamental frequency and harmonics in the rumbles.
  
  
  
  

REFERENCES

[1] https://jeb.biologists.org/content/216/21/4054

[2] https://engineering.ucdenver.edu/current-students/capstone-expo/archived-expos/spring-2020/electrical-engineering/elec6-improving-infrasound-detection

[3] https://docs.microsoft.com/en-us/azure/iot-edge/how-to-store-data-blob?view=iotedge-2018-06

[4] https://jaredrhodes.com/2017/11/27/create-azure-function-to-process-iot-hub-file-upload/

[5] https://www.nature.com/articles/srep27585/tables/3

[6] https://docs.microsoft.com/en-us/learn/modules/implement-cicd-iot-edge/6-exercise-create-release-pipeline-iot-edge

[7] https://augmentor.readthedocs.io/en/master/code.html

[8] https://github.com/DemisEom/SpecAugment.

LEARN MORE ABOUT PROJECT15

Project 15: Empowering all to preserve endangered species & habitats.

Date: April 22, 2021

Time: 08:00 AM – 09:00 AM (Pacific)

Location: Global

Format: Livestream

Topic: DevOps and Developer Tools

Microsoft Project15 & University of Oxford Capstone Project with Elephant Listening Project Team 1

by Contributed | Apr 8, 2021 | Technology

This article is contributed. See the original author and article here.

Oxford’s AI Team 1 solution to Project 15’s Elephant Listening Project

Who are we?

Members of Microsoft Project 15 have approached Oxford University to help analyse these audio recordings to better monitor for threats and elephant well-being. We will use the machine learning algorithms to help identify, classify, and respond to acoustic cues in the large audio files gathered from Elephant Listening Project 

What are we trying to do?

Microsoft’s Project 15’s
Elephant Listening Project Challenge document

The core of this project is to assist in Project 15’s mission to save elephants and precious wildlife in Africa from poaching and other threats. Project 15 has tied with the Elephant Listening Project (ELP) to cleverly record acoustic data of the rural area in order to listen for threats to elephants and monitor their health

Baby elephant calling out for mama elephant.

How are we going to do it?

Documentation for video (here)

Data preview / feature engineering

The key to many machine learning challenges boils down to data quality: not enough data or too much data. In this case, the rich audio files proves expensive and time-consuming to analyse in full. Hence, MATLAB 2020b and the Audio Toolbox was used to truncate audio files down into processable sizes.

Before we began, we conducted a brief literaure search on what kind of audio snippets we needed to truncate out. A paper analysed gunshots in cultivated environments (Raponi et al.) and characteristed two distinctive signals in the acoustic profile in most guns: a muzzle blast and ballistic shockwave. From this, we had a better clue of what kind of audio snippet profiles we should extract.

A variety of ELP audios were obtained and placed into a directory to work within MATLAB 2020b to ensure the audio snippets obtained reflected the diverse African’ environment. DetectSpeech, a function in MATLAB’s audio toolbox was used to extract audios of interest; audio snippets whose signal to noise ratio suggested speech due to it’s high frequency signal. These extracted snippets were compared to a gunshot profile obtained from the ELP with the spectrogram shown. This method allowed us to only analyse segments of the large 24hr audio file.

This datastore will be used later in validating and testing our models.

Modelling

We developed and designed a deep learning model that would take in a dataset of audio files and classify elephant sounds, gunshot sounds and other urban sounds. We had approximately 8000 audio files as our training set and 2000 audio files as our testing set ranging from elephant sounds, gunshot sounds, to urban sounds. The procedure we followed was to convert these audio files into spectrograms and then classify based on the spectrogram images. The step-by-step process is highlighted below:

After getting the spectrograms, we converted the spectrograms into Numpy arrays with each pixel of an image is represented as an array. Spectrograms frequencies vary for each class of an audio file. The sound frequency pattern is different for elephants, gunshots, etc. We want to be able to classify each audio based on the differences in sound frequencies captured in spectrogram images. After converting spectrograms into Numpy arrays, we normalized each numpy representation of a spectrogram by taking the absolute mean and subtracting from each image and then diving by 256. From this point, we applied convolutional later and started training the model. We adjusted the hyperparameters to see which would yield better accuracy in classifying audio files. SGD has better generalization performance than ADAM. SGD usually improves model performance slowly but could achieve higher test performance. The figure below shows the results of the model:

Deployment Strategy

Our actual deployment strategy (as mentioned in our video presentation) is to deploy the built model to an Edge Device (NVIDIA jetson Nano connected to a microphone – in our case) which would be present in the location and perform real time inferencing. The device would wake up whenever there is a sound and perform the inferencing to detect either gunshots or elephant noise.

As of now we have our model deployed on cloud – using Azure ML Studio and is hosted through a docker endpoint (through ACI). Presently inferencing is being performed on the cloud using this endpoint. As an arrangement to this approach, all our data is hosted locally in our cloud workspace environment for the time being. 

Simultaneously we are also testing the model deployment on our preferred edge device. Once this is done, we will be able to enable real time inferencing on the edge. Post successful completion of this we will also modify our data storage approach.

Conclusion and Future Directions

Currently we have a working model which is able to detect gunshots as well as elephant sound. Our model is deployed on cloud and is ready for inferencing.

As per our end to end workflow – we are testing our model container deployed on our preferred IoT Device. Once this test is a success, we will configure our edge device with azure IoT runtime and docker. And then, test the same for real time edge inferencing with gunshot and elephant sound.

Upon the success of this, we will automate the entire deployment process using an ARM template. Once we reach a certain level of maturity technically and are ready to scale – we will develop an IoT Edge Deployment Manifest for replicating the desired configuration properties across a group of Edge Devices.

Our next steps solely for improving model performance – 

• Classify elephant sounds based on demographics


• Combine gunshots with ambient sounds (Fourier transform)


• Eventually rework the way we infer so that a spectrogram file does not have to be created and ingested by the Keras model. This would be more of an in memory operation

Contact us if you’d like to join us in helping Project 15

Although we are proud of our efforts, we recognise more brains out there can help inject creativity into this project to further assist Project 15 and ELP. Please visit our GitHub repo (here) to find more information on our work and where you can help! 

https://github.com/Oxford-AI-Edge-2020-2021-Group-1/AI4GoodP15

Online Register for Classes with Analytics (ORCA) Project

by Contributed | Apr 8, 2021 | Technology

This article is contributed. See the original author and article here.

Team members (from top left to right):

Emeralda Sesari, Khesim Reid, Arzhan Tong, Xinyuan Zhuang, Omar Beyhum, Lydia Tsami, Yanke Zhang, Chon Ng, Elena Aleksieva, Zisen Lin, Yang Fan, Yifei Zhao

Supported by:

Lee Stott, Microsoft

Dean Mohamedally, University College London

Giovani Guizzo, University College London 

The Team

We are a group of 12 students at University College London (UCL).  UCL’s Industry Exchange Network (IXN) programme enabled us to work on this project in collaboration with Microsoft as part of a course within our degree.

Introduction

Attendance and engagement monitoring is an important but exceptionally challenging process universities and colleges; especially during the COVID-19 pandemic in 2020, students attend both online synchronous and asynchronous courses around the world. Predictably, we need a sensible course monitoring system to facilitate teaching staff and learning administrators who set the curriculum to better manage students in their courses, not just during the pandemic. ORCA is a system that was created for this purpose.

ORCA is designed to complement the online learning and collaboration tools of schools and universities, most notably Moodle and Microsoft Teams. In brief, it can generate visual reports based on student attendance and engagement metrics, and then provide them to the relevant teaching staff.

Since ORCA is designed to integrate with both MS Teams and Moodle, our entire development team (12 students in total) is grouped into two small groups (the MS Teams group and the Moodle group). The two groups set up the main functionalities of ORCA together and implemented the integrations with the respective platforms separately.

In short, ORCA is aimed to:

• assess the effectiveness of online learning by gathering data on students’ use of online learning including students’ attendance and engagement from both MS Teams and Moodle;


• keep track of the status of students that may impact their education;


• help teaching staff and learning administrators get actionable insights from data to improve teaching methods.

Design

ORCA allows institutions to gain insights about students’ attendance and engagement with online platforms. To accomplish this, the ORCA system mainly integrates with Moodle and Microsoft Graph

Figure 1: ORCA’s design

These two external systems send out notifications to ORCA whenever students interact with Moodle and Microsoft Teams respectively. ORCA then standardizes the incoming information into a common format and stores it on Sharepoint lists which the relevant teaching and learning administration staff can access to check attendance.

ORCA also gives the option to store student engagement data in a database which can be exposed to the Power BI dashboarding service and embedded into Sharepoint, allowing staff to visualize how students interact with online content.

Figure 2: Student engagement visualized through Power BI

Implementation

Figure 3: ORCA’s architecture

ORCA is a .NET 5 application implemented in C#. Figure 3 shows the main architecture of ORCA. The core of the system, the event aggregator, is where all the information is collected from different sources.  The event aggregator interfaces with pluggable API adapters, which allow data (i.e., student events) from a different system to be standardized so that the event aggregator can persist them for future analysis. 

So far, there are 2 main API adapters, the MS Graph Adapter which is notified by Microsoft’s Graph API whenever students join lectures, and the Moodle Adapter, which receives information about student activity through a plugin that can be installed on an institution’s Moodle server. The Moodle Adapter also follows the IMS Caliper standard, which allows it to integrate with any Learning Management System implementing the standard, such as Blackboard Learn.

ORCA can be automatically deployed to the Azure cloud or configured to run on a custom (on-premises) Windows, Linux, or macOS machine of an IT Administrator’s choosing. Powershell scripts and precompiled binaries are provided in our Github Releases page to automate configuration and deployment, although ORCA’s centralized configuration can also be manually changed.

ORCA makes use of Azure Resource Manager (ARM) Templates to define the resources it requires to run on the Azure cloud. By packaging a script which dynamically generates an ARM template based on the IT Administrator’s configuration choices, we can automatically provision and configure a webserver and optional database to run ORCA on the cloud within minutes. If a database was selected, the script also generates Power BI dashboards which automatically connect to the provisioned database.

Future Work

While the ORCA project is a functional software product that can be deployed to an institution’s infrastructure, we have yet to run it against real student data during the development stage due to privacy concerns. To comply with privacy regulations, current testing is based on synthetically generated data. We are in the process of piloting ORCA with UCL’s Information Services Division to better assess how the system meets institutions’ needs in a more productionized environment.

We also hope to increase ORCA’s level of customization and allow administrators to configure which activities count towards engagement, and at what point they should be automatically alerted about student inactivity.

ORCA is an open source project which any institution can adopt, contribute to, or simply inspire themselves from to build their own solution.

You can check it out at https://github.com/UCL-ORCA/Orca