How can generate impulse as outliers on periodic or sequenced-based data for doing experiments via outliers detection methods?

Question

I want to carry out some experiments on some time-series data with the KM approach has been suggested in this paper . The problem is I don't access the data in the paper but I have similar type data which has no outliers and I need to generate some outliers artificially/manually in form of the impulse (top & bottom) so that meet the requirements as it is shown in the following picture from this paper achievement:

In the worst case, I was wondering if I could generate something like this on the periodic perfect train of pulse or Sin function to apply to the available data. So far, my implementation is limited to these post1 , post2 and Welch but in order to generate right outliers, I was thinking of identifying/detecting high and low states and then add outliers frequently for further experiments of outlier detectors.

I'm unsure that convolving impulses as noise to target outliers is the right way to contaminate periodic data I access or not.

So far I have generated a train of impulses but I have no idea how I can include to one periodic signal or data:

import scipy.signal as signal
import matplotlib.pyplot as plt
imp = signal.unit_impulse(200, [10,50,60])

fig, ax = plt.subplots(1,1, figsize=(8,6))

ax.set_xlabel('Cycles')
plt.plot(imp)
plt.title('Generating impulse outliers in desired cycles: 10, 50 & 60')
plt.ylim((0.0, 1.4))
plt.show()

Any helps will be appreciated since many people are working on topics like Anomaly and outlier detection.

Answer 1

Synthesizing anomalies in time-series can be done by considering the output as a combination of the normal signal and some noise . The most common noise model is additive noise , which means the the output is simply out = signal + noise . In your case you would scale the unit impulses to be of suitable amplitude before adding. In the example the amplitude of the impulsive noise seems around 15.

PS: Both amplitude, distance between impulses and width/shape of impulse should probably be random variables.

Answer 2

You can generate a signal with eg.: numpy Python module. And you can add your impulses to this signal (of course, if the dimensions are correct). I have written an example for your where I have generated a Sinus signal with numpy and I have added the impulses with the signal.unit_impulse() like in your question. I have added several comments to code for the better understanding.

Code:

import numpy as np
import matplotlib.pyplot as plt
import scipy.signal as signal

positive_impulses = signal.unit_impulse(200, [10, 50, 60])  # Generate positive impulses
negative_impulses = signal.unit_impulse(200, [80, 100, 150])*-1  # Generate negative impulses

# Generate the Sinus signal.
t = np.linspace(1, 200, 200)
x_sin_sig = np.sin(t / (2 * np.pi))
plt.subplot(4, 1, 1)
plt.plot(t, x_sin_sig)
plt.title("Signal")
plt.ylabel("Sin")


plt.subplot(4, 1, 2)
plt.plot(t, x_sin_sig + positive_impulses)  # Add the positive impulses to the original signal
plt.title("Signal with positive impulses")

plt.subplot(4, 1, 3)
plt.plot(t, x_sin_sig + negative_impulses)  # Add the negative impulses to the original signal
plt.title("Signal with negative impulses")

plt.subplot(4, 1, 4)
plt.plot(t, x_sin_sig + positive_impulses + negative_impulses)  # Add the both impulses to the original signal
plt.title("Signal with different impulses")

plt.tight_layout()
plt.show()

Output:

Note:

Probably more efficient to generate random impulses for your signal. You can do it with the random module.

import random

positive_impulses = signal.unit_impulse(200, random.sample(range(200), 5))  # Generate positive impulses
negative_impulses = signal.unit_impulse(200, random.sample(range(200), 5))*-1  # Generate negative impulses

The random.sample(range(200), 5) will return a list of 5 numbers selected from the range 0 to 200, without duplicates.

An example with random impulses:

Answer 3

Thanks for the invitation. Regarding how to generate periodic outliers, I have no such experience till now. But as it is time series based, so I think we can start from Arima concept.

1. if you can find the p , d , q parameter, you already reach the data answer. though there is high computation cost. there is some reference in StackOverflow about how to find the p q d

2. to deal with the time series, the stationary is the most important thing in Arima mode. You can try to do 1st differential or 2nd differential. If the data set has just 1 feature, the outlier already showed out directly.

Good luck.

Hope it was helpful.

WY