PROJECT TITLE :

Medical data wrangling with sequential variational autoencoders

ABSTRACT:

The majority of the time, medical data sets are flawed due to noise and gaps in the data. It is a common misconception that these missing patterns are completely at random. However, in medical settings, the reality is that these patterns occur in bursts for a variety of reasons, including sensors that are turned off for a period of time or data collected in an uneven manner. In this paper, sequential variational autoencoders are proposed as a method for modeling medical data records that contain a variety of data types and sporadic instances of missing data (VAEs). In particular, we propose a new methodology called the Shi-VAE, which extends the capabilities of VAEs to sequential streams of data that contain missing observations. This allows VAEs to better model real-world data. The state-of-the-art solutions in an intensive care unit database (ICU) and a dataset of passive human monitoring are used to evaluate our model and compare it to those solutions. In addition, we find that standard error metrics like RMSE are not conclusive enough to evaluate temporal models, and we incorporate the cross-correlation between the ground truth and the imputed signal into our analysis. This is because RMSE is a measure of absolute error. We demonstrate that Shi-VAE achieves the best performance in terms of using both metrics, and it does so with less computational complexity than the GP-VAE model, which is the method that is currently considered to be the state-of-the-art for medical records.