Source
r/PrometheusMonitoring•
martbhell

File_sd and exporters

New to Prometheus.

A

I've used ansible and modifies the exporter roles to also run a delegate_to to the Prometheus server to write a target file for service discovery.

A) Is this a good approach?

We have until now never deleted or decommissioned a server but I think the manual work to remove some files for our small env it's acceptable to manually delete files sometimes..

B

I've added some blackbox checks similarly to get http metrics from se servers. Now we also have some servers we don't manage with ansible but we want to get say certificate expiry time.

B) Got any suggestions for the job names? I've just called them blackbox_http_2xx and then blackbox_external ..

C

Bit worried about how long retention this thing can handle. I would like it to be years. No Thanos or such remote read/write setup right now.

C) Is it possible to later add these remotes?

Posted May 8, 2020, 5:49 PM UTC

13
signal_vs_noise•

a) When your servers don't change that often it's fine to use Ansible and file_sd. When you decommission a server, you just have to remove the name from the inventory and apply the role again.

b) Sounds fine to me, or something like blackbox_ansible_managed and blackbox_external.

c) Depends on the amount of machines you're monitoring and the number of metrics you're collecting. Haven't used any long-term storage for metrics, so I don't know what's the best way to handle this.

2
martbhellOP•

Thanks!

Okay well I can when things have stabilized for example quite easily look at how much I'm collecting in MB per day. Right now it's looking like 25 machines and 15s scraping interval, two exporters per machine(node+another). Thinking of adding laptops to this too though as a poor man's way of checking if say laptops are updated. That would add machines but I think scrape interval would be really seldom.

Actually another question is D) : in graphite changing interval was really annoying - is this easy to change in Prometheus? Just change the scrape interval and my grafana graphs will still look ok?

1
signal_vs_noise•

Yes, in Prometheus you can either change the global scrape interval or the interval by job in your config. Defaults:

scrape_interval: 15s
scrape_timeout: 10s
evaluation_interval: 1m

For rate queries in Grafana, you can also use its interval and range variables when you change the time range on the dashboard:

https://grafana.com/docs/grafana/latest/features/datasources/prometheus/#using-interval-and-range-variables

1