Alerting when there is still time to fix things∞
predict_linear in Prometheus uses linear regression to predict how a time series will evolve in the future. So you can alert when there is still time to fix things.
This is something that the SRE book from Google talks about. Alerting when there is still time to fix things, when stuff isn’t broken yet.
In general, that book seems really interesting. I do find it a bit difficult to read because it is very abstract as well, but interesting nonetheless. (I think it is a bit abstract because the engineers writing it wanted to talk about generally applicable principles, and not the tools they use at Google. However, these tools are somewhat crucial, so it would help to get some hints what kinds of technologies would help implementing something like what they describe in the book.)