Time Series Anomaly Detection: Uncovering Anomalous Patterns and Outliers in Time Series Data using Machine Learning Techniques

This tutorial aims to provide a comprehensive guide to time series anomaly detection using machine learning techniques. We will explore various methods to uncover anomalous patterns and outliers in time series data. By the end of this tutorial, you will have a solid understanding of the concepts and practical knowledge to apply anomaly detection techniques to your own time series datasets.

Foto de Austin Distel en Unsplash

Introduction

Time series data is a sequence of observations recorded at regular intervals over time. It is commonly encountered in various domains such as finance, weather forecasting, stock market analysis and many more. Analyzing time series data can provide valuable insights and help in making informed decisions.

Anomaly detection is the process of identifying patterns in data that do not conform to expected behavior. In the context of time series data, anomalies can represent significant events or outliers that deviate from the normal patterns. Detecting anomalies in time series data is crucial for various applications, including fraud detection, network monitoring and predictive maintenance.

In this tutorial, we will explore different machine learning techniques for time series anomaly detection. We will cover the following topics:

1. Understanding Time Series Data
2. Preprocessing Time Series Data
3. Statistical Methods for Anomaly Detection
4. Machine Learning Methods for Anomaly Detection

For readers interested in further exploration:

5. Evaluating Anomaly Detection Models

6. Building an Anomaly Detection System

Let’s get started by importing the necessary libraries and downloading the time series data for our tutorial.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import yfinance as yf

# Download time series data using yfinance
data = yf.download('AAPL', start='2018-01-01', end='2023-06-30')

Understanding Time Series Data

Before diving into anomaly detection techniques, it is essential to understand the characteristics of time series data. Time series data typically exhibits the following properties:

1. Trend: A long-term increase or decrease in the data values over time.
2. Seasonality: Repeating patterns or cycles at fixed intervals.
3. Autocorrelation: Correlation between the current observation and previous observations.
4. Noise: Random fluctuations or irregularities in the data.

Let’s visualize the downloaded time series data to gain insights into its properties.

# Plot the time series data
plt.figure(figsize=(12, 6))
plt.plot(data['Close'])
plt.xlabel('Date')
plt.ylabel('Closing Price')
plt.title('AAPL Stock Price')
plt.xticks(rotation=45)
plt.grid(True)

plt.show()

Figure 1: AAPL Stock Price. Created by Author

From the plot, we can observe the trend of increasing stock prices over time. There are also periodic fluctuations, indicating the presence of seasonality. Additionally, there seems to be some autocorrelation between consecutive closing prices.

Preprocessing Time Series Data

Before applying anomaly detection techniques, it is crucial to preprocess the time series data. Preprocessing involves handling missing values, smoothing the data and removing outliers.

Handling Missing Values

Missing values can occur in time series data due to various reasons such as data collection errors or gaps in the data. It is essential to handle missing values appropriately to avoid bias in the analysis.

# Check for missing values
missing_values = data.isnull().sum()
print(missing_values)

Output:

Open         0
High         0
Low          0
Close        0
Adj Close    0
Volume       0
dtype: int64

Fortunately, the downloaded data does not contain any missing values. If there were missing values, we could handle them by either imputing the missing values or removing the corresponding time points.

Smoothing the Data

Smoothing the time series data can help in reducing noise and highlighting the underlying patterns. One common technique for smoothing time series data is moving averages.

# Smooth the time series data using a moving average
window_size = 7
data['Smoothed'] = data['Close'].rolling(window_size).mean()

# Plot the smoothed data
plt.figure(figsize=(12, 6))
plt.plot(data['Close'], label='Original')
plt.plot(data['Smoothed'], label='Smoothed')
plt.xlabel('Date')
plt.ylabel('Closing Price')
plt.title('AAPL Stock Price (Smoothed)')
plt.xticks(rotation=45)
plt.legend()
plt.grid(True)

plt.show()

Figure 2: AAPL Stock Price (Smoothed). Created by Author

The plot shows the original closing prices and the smoothed version obtained using a moving average. Smoothing helps in visualizing the overall trend and reducing the impact of short-term fluctuations.

Removing Outliers

Outliers can significantly affect the performance of anomaly detection algorithms. It is essential to identify and remove outliers before applying anomaly detection techniques.

# Calculate z-scores for each data point
z_scores = (data['Close'] - data['Close'].mean()) / data['Close'].std()

# Define a threshold for outlier detection
threshold = 3

# Identify outliers
outliers = data[np.abs(z_scores) > threshold]

# Remove outliers from the data
data = data.drop(outliers.index)

# Plot the data without outliers
plt.figure(figsize=(12, 6))
plt.plot(data['Close'])
plt.xlabel('Date')
plt.ylabel('Closing Price')
plt.title('AAPL Stock Price (Without Outliers)')
plt.xticks(rotation=45)
plt.grid(True)

plt.show()

Figure 3: AAPL Stock Price (Without Outliers). Created by Author

The plot shows the time series data after removing the identified outliers. Removing outliers helps in improving the accuracy of anomaly detection algorithms by reducing the influence of extreme values.

In the next section, we will explore statistical methods for anomaly detection in time series data.

Statistical Methods for Anomaly Detection

Statistical methods provide a foundation for detecting anomalies in time series data. We will explore two commonly used statistical techniques: z-score and moving average.

Z-Score

The z-score measures the number of standard deviations an observation is from the mean. By calculating the z-score for each data point, we can identify observations that deviate significantly from the expected behavior.

# Calculate z-scores for each data point
z_scores = (data['Close'] - data['Close'].mean()) / data['Close'].std()

# Plot the z-scores
plt.figure(figsize=(12, 6))
plt.plot(z_scores)
plt.xlabel('Date')
plt.ylabel('Z-Score')
plt.title('Z-Scores for AAPL Stock Price')
plt.xticks(rotation=45)
plt.axhline(y=threshold, color='r', linestyle='--', label='Threshold')
plt.axhline(y=-threshold, color='r', linestyle='--')
plt.legend()
plt.grid(True)

plt.show()

Figure 4: Z-Scores for AAPL Stock Price. Created by Author

The plot shows the calculated z-scores for each data point. Observations with z-scores above the threshold (red dashed lines) can be considered as anomalies.

Moving Average

Another statistical method for anomaly detection is based on the moving average. By calculating the moving average and comparing it with the original data, we can identify deviations from the expected behavior.

# Calculate the moving average
window_size = 7
moving_average = data['Close'].rolling(window_size).mean()

# Calculate the deviation from the moving average
deviation = data['Close'] - moving_average

# Plot the deviation
plt.figure(figsize=(12, 6))
plt.plot(deviation)
plt.xlabel('Date')
plt.ylabel('Deviation')
plt.title('Deviation from Moving Average')
plt.xticks(rotation=45)
plt.axhline(y=0, color='r', linestyle='--', label='Threshold')
plt.legend()
plt.grid(True)

plt.show()

Figure 5: Deviation from Moving Average. Created by Author

The plot shows the deviation of each data point from the moving average. Positive deviations indicate values above the expected behavior, while negative deviations indicate values below the expected behavior.

In the next section, we will explore machine learning methods for anomaly detection in time series data.

Machine Learning Methods for Anomaly Detection

Machine learning methods provide more advanced techniques for anomaly detection in time series data. We will explore two popular machine learning algorithms: Isolation Forest and LSTM Autoencoder.

Isolation Forest

Isolation Forest is an unsupervised machine learning algorithm that isolates anomalies by randomly partitioning the data into subsets. It measures the average number of partitions required to isolate an observation and anomalies are expected to require fewer partitions.

from sklearn.ensemble import IsolationForest

# Prepare the data for Isolation Forest
X = data['Close'].values.reshape(-1, 1)

# Train the Isolation Forest model
model = IsolationForest(contamination=0.05)
model.fit(X)

# Predict the anomalies
anomalies = model.predict(X)

# Plot the anomalies
plt.figure(figsize=(12, 6))
plt.plot(data['Close'])
plt.scatter(data.index, data['Close'], c=anomalies, cmap='cool', label='Anomaly')
plt.xlabel('Date')
plt.ylabel('Closing Price')
plt.title('AAPL Stock Price with Anomalies (Isolation Forest)')
plt.xticks(rotation=45)
plt.legend()
plt.grid(True)

plt.show()

Figure 6: AAPL Stock Price with Anomalies (Isolation Forest). Created by Author

The plot shows the time series data with anomalies identified by the Isolation Forest algorithm. The anomalies are highlighted in a different color, indicating their deviation from the expected behavior.

LSTM Autoencoder

LSTM (Long Short-Term Memory) Autoencoder is a deep learning model that can learn the patterns in time series data and reconstruct the input sequence. Anomalies can be detected by comparing the reconstruction error with a predefined threshold.

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dense

# Prepare the data for LSTM Autoencoder
X = data['Close'].values.reshape(-1, 1)

# Normalize the data
X_normalized = (X - X.min()) / (X.max() - X.min())

# Train the LSTM Autoencoder model
model = Sequential([
    LSTM(64, activation='relu', input_shape=(1, 1)),
    Dense(1)
])
model.compile(optimizer='adam', loss='mse')
model.fit(X_normalized, X_normalized, epochs=10, batch_size=32)

# Reconstruct the input sequence
X_reconstructed = model.predict(X_normalized)

# Calculate the reconstruction error
reconstruction_error = np.mean(np.abs(X_normalized - X_reconstructed), axis=1)

# Plot the reconstruction error
plt.figure(figsize=(12, 6))
plt.plot(reconstruction_error)
plt.xlabel('Date')
plt.ylabel('Reconstruction Error')
plt.title('Reconstruction Error (LSTM Autoencoder)')
plt.xticks(rotation=45)
plt.axhline(y=threshold, color='r', linestyle='--', label='Threshold')
plt.legend()
plt.grid(True)

plt.show()

Figure 7: Reconstruction Error (LSTM Autoencoder). Created by Author

The plot shows the reconstruction error for each data point. Observations with reconstruction errors above the threshold (red dashed line) can be considered as anomalies.

Note: The following sections on evaluating anomaly detection models and building an anomaly detection system are out of scope for this article. However, for those interested in exploring these topics further, here is a more extensive discussion:

Evaluating Anomaly Detection Models:

To accurately evaluate the performance of anomaly detection models, it is crucial to have labeled data that contains information about the presence or absence of anomalies. Unfortunately, obtaining labeled data with known anomalies can be challenging in real-world scenarios. Therefore, alternative techniques can be employed to evaluate the effectiveness of these models.

One such technique is cross-validation, which involves splitting the available labeled data into multiple subsets or folds. The model is trained on a portion of the data and then evaluated on the remaining fold. This process is repeated several times and the evaluation results are averaged to obtain a more reliable estimate of the model’s performance.

Unsupervised evaluation metrics can also be utilized when labeled data is not readily available. These metrics assess the performance of anomaly detection models based on characteristics intrinsic to the data itself, such as clustering or density estimation. Examples of unsupervised evaluation metrics include silhouette score, Dunn index, or the average nearest neighbor distance.

Building an Anomaly Detection System

Building an anomaly detection system involves integrating various techniques and creating a coherent pipeline. This pipeline typically consists of the following steps:

1. Data Preprocessing: Prior to modeling, it is crucial to preprocess the data. This involves handling missing values, smoothing the data to reduce noise and removing outliers that could affect the performance of the anomaly detection model.
2. Feature Engineering: Extracting relevant features from the time series data plays a vital role in improving the accuracy of the anomaly detection system. Feature engineering involves transforming the raw data into a set of meaningful features that capture the underlying patterns and characteristics of anomalies.
3. Model Training: In this step, the anomaly detection model is trained using either labeled or unlabeled data, depending on the availability of annotated anomalies. The choice of the model depends on the specific requirements of the problem, with options including statistical methods, machine learning algorithms, or deep learning architectures.
4. Model Evaluation: Evaluating the performance of the anomaly detection model is essential to ensure its effectiveness. Appropriate evaluation metrics, such as precision, recall, F1 score, or area under the receiver operating characteristic curve (AUC-ROC), should be used to assess the model’s ability to accurately detect anomalies.
5. Deployment: Once the anomaly detection system has been built and evaluated, it can be integrated into the production environment. This involves deploying the system to the target infrastructure, configuring it to handle real-time data streams and monitoring its performance over time.

By following these steps, you can develop a robust anomaly detection system customized to the characteristics of your specific time series data. However, a detailed exploration of these topics is beyond this article scope but a complete anomaly detection system may be available in the future.

Conclusion

In this tutorial, we explored various techniques for time series anomaly detection using machine learning. We started by understanding the properties of time series data and preprocessing it to handle missing values, smooth the data and remove outliers. We then discussed statistical methods such as z-score and moving average for anomaly detection. Finally, we explored machine learning methods including Isolation Forest and LSTM Autoencoder.

Anomaly detection is a challenging task that requires a deep understanding of time series data and the appropriate techniques to uncover anomalous patterns and outliers. By applying the concepts and techniques covered in this tutorial, you can develop effective anomaly detection systems for your own time series datasets.

Remember to experiment with different algorithms, fine-tune the parameters and evaluate the performance of your models to achieve the best results.

Become a Medium member today and enjoy unlimited access to thousands of Python guides and Data Science articles! For just $5 a month, you’ll have access to exclusive content and support as a writer. Sign up now using my link, and I’ll earn a small commission at no extra cost to you.