Introduction to Anomaly Detection with Auto-Encoder on time series
Posted Mar 18, 2021, 11:28 AM UTC
53930
Just a little advice!
I think you should add more details in the readme file. I know it is anomaly detector but I don't know what methods you used.
Yes, I can find it from the notebook but that would take a long time wouldn't it?
So a summary in the readme would go a long way.
21
Yes, it's true. I will do that. Thanks for the advice.
12
Welcome.
5
This is really interesting, I can think of a bunch of ways this can be useful
10
Thanks !
3
What is the pattern you’re checking against here in the example looks like Nano pattern that’s been fabricated.
3
The data comes from connections to a website.
1
So do you rasterize the data loads ?
1
Really nice Github !
Thanks a lot !
6
Thanks
2
Cool stuff, thanks for sharing.
Question about unsupervised anomaly detection - how are you evaluating your model? Obviously you can train a network using reconstruction loss, but that metric tells you nothing about how good of an anomaly detector your model is.
2
There is a couple of density-based approaches you can take to quantify 'outlierness' as low-probability. One way is to model the probability density of time series fragments with normalizing flow models. Another way is to use the score of the discriminator part of a GAN. Both can be somewhat tricky though, although they will work well for obvious outlier.
My go-to method for outlier detection is isolation forests which give an anomaly score as a metric.
6
Interesting, thanks for the advice. I've seen ML.NET applies kernel density estimation to the output of the regression model, since the residuals should remain constant even though its a timeseries. I'll definitely look up normalizing flow models, cheers.
With regards to a GAN, would I not need labelled anomalies to train the system? Or is the system trained on reconstruction loss, so basically an autoencoder + discriminator?
I'm working on my thesis problem for energy consumption anomaly detection, using multivariate timeseries data. I tried isolation forests, but I was wasn't getting "good-looking" results compared to autoencoders / regression residuals. KMeans and HDBScan were a little better, but also not fantastic.
It's looking unavoidable that I'll have to figure out how to label a ton of data manually. Otherwise it seems impossible to empirically verify how well any method performs comparative to any other, no?
2
Interesting that you're not getting good-looking result. It must mean that your brain can somehow do a better job than the ML methods! Would be interesting to ponder about how that can be. That might give you hints on how to improve it. Btw I think it's great that you compare different methods.
A GAN would indeed need to be trained with "clean consumption data", you should know the outliers in your trainset and remove them.
For modelling densities you can indeed use kernel density methods, like have a NN that outputs the parameters of e.g. a Gaussian Mixture model.
Many (20) years ago I worken as head of modelling at an energy company and one of our tasks was demand forecasting for large pools of customers combined (which is different that what you do I imagine, but still somewhat related). We used support vector machines for that because this was before the deep learning boom. We spend a lot of time building features around calenders (time of day, day of week, holiday indicators etc) and weather (temperature, windspeed, light intensity).. We would basically build a demand forecast models, inspect the events where the model made the biggest error, and see if we could figure out what happens there. Sometimes we would find an event like a big football match, or a school holiday that started a week earlier than we thought. This error/feature research helped a lot in getting the forecast very accurate.
It might also be interesting to look at kaggle competition around electricity demand forecasting, like this one https://www.kaggle.com/c/electricity. Sometimes winners blog about how they did it, and sometimes there is forums with chats about competitions. Some people even share notebooks!
3
https://dl.acm.org/doi/10.1145/3394486.3403392
For multivariate, there is this method that trains autoencoders in an adversarial way like a GAN. This can surely help you with your problem.
2
Wow dude, awesome, I'm definitely gonna try out this architecture. Also super impressed with the presentation, now I wish all papers had a video breakdown lol. Thanks a lot for the insights, I'll be sure to reference you in my thesis!
Can I please ask about a possible extension to your paper?
Your architecture outputs anomaly scores as α + β, which is cool because it allows you to weight each parameter to tune the sensitivity. I believe you are then ultimately classifying anomalies based on a simple threshold, right? If α + β > λ then anomaly.
Do you think your architecture could be improved by combination with one of the density-based approaches mentioned above?
For instance, fitting a Kernel Density Estimator on the model outputs would give you a probability distribution. Then instead of λ being an arbitrary number, you could set it based on likelihood, eg. flag results with less than 5% chance of occurrence.
Would love to hear your thoughts on this, or any other extension ideas you feel might be fun to try. All the best.
2
Absolutely. This is the biggest problem in unsupervised. What we do is we have test data sets that are labeled by experts. This allows us to verify the performance of our models. Finally we can fine-tune the models by having live feedback from users.
2
Is this what they use to check for defects in chips?
1
Can the convolutional autoencoder architecture be replaced with Conv1d as for time series, with the example https://keras.io/examples/timeseries/timeseries_anomaly_detection/.
if we add maxpooling1D layer then it would be the same architecture perhaps. Could I have an example architecture based on conv1d autoencoder for time series anomaly detection. Also, I am not clear on how to select a suitable window size for creating sequences.
1
Not sure why this is 2D but it's interesting graphic nevertheless
0
It's a pairwise distance matrix so pixel(i,j) is the distance between point i and point j in the time series.
2
Oh cool I get it now
1
Put this shit through a cpu layout
0
Here is a notebook to understand how to detect anomalies on time series.
https://github.com/JulienAu/Anomaly_Detection_Tuto/
You can view the notebook on : Nbviewer
I'll improve the notebook in the future. Feel free to ask questions or comment.
Nice stuff. Have you tried other timeseries anomaly detection methods as well?
Yes, especially on multivariate series. You can see examples of slightly more complex methods here: https://dl.acm.org/doi/10.1145/3394486.3403392
Do you think this would work on seismic data time series? For identifying different wave phase arrivals (e.g. P-waves and S-waves).
I don't know the seismic data at all. But it's worth a try. By definition, the auto-encoder learns to reconstruct. So I guess if you can train it on one class of wave, it will be able to detect when it is one or not. However, I suppose that classification models might be more appropriate.