Source
r/elasticsearch•
gregolde

Filebeat drop_fields processors not working - any ideas?

I've been working on my filebeat config trying to drop fields that I don't need that are created by filebeat and I'm not having any success. I don't see anything wrong with my config, and filebeat is working, just ignoring the drop_fields.

I've tried this with two different inputs, both syslog and log files without any luck. Can anyone please take a look at my config and tell me what I'm missing? This is driving me crazy. I've omitted everything not related to an input or a processor in the config just to cover relevance.

    # ============================== Filebeat inputs ===============================

filebeat.inputs:

- type: syslog
  enabled: true
  format: auto
  protocol.udp:
    host: "192.168.2.253:514"
  fields:
    event.type: vmware
  fields_under_root: true

  processors: 
  - drop_fields:
      fields: ["agent.ephemeral_id", "agent.id", "agent.version", "ecs.version", "host.architecture", "host.containerized", "host.hostname", "host.id", "host.ip", "host.mac", "host.name", "host.os.codename", "host.os.family", "host.os.kernel", "host.os.name", "host.os.platform", "host.os.version"]

# ================================= Processors =================================
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

Posted Nov 28, 2021, 8:58 PM UTC · edited

45
InvestingIsHard•

I believe that at least some of the fields are probably being added back into the event by the metadata processors that are lower in your config. Try removing the add_*_metadata processors, or moving the drop_fields processor to below those ones.

Additionally, I think you may need to fix the indentation:

processors: 
  - drop_fields:
      fields: ["agent.ephemeral_id", "agent.id", "agent.version", "ecs.version", "host.architecture", "host.containerized", "host.hostname", "host.id", "host.ip", "host.mac", "host.name", "host.os.codename", "host.os.family", "host.os.kernel", "host.os.name", "host.os.platform", "host.os.version"]
2
gregoldeOP•

Definitely worth a shot -

I've commented out the entire lower processor section, still seeing the same fields.

I've tried formatting the indentation a ton of different ways. Based on the example docs (https://www.elastic.co/guide/en/beats/filebeat/current/defining-processors.html) it looks like it's supposed to be

processors: (no spaces)
  - drop_fields: (2 spaces)
      fields:  (6 spaces)

My filebeat wont start formatted as the docs, it's currently formatted

  processors: (2 spaces)
  - drop_fields: (2 spaces)
      fields:  (6 spaces)

What do you think the indent needs to be set to? I wouldn't be surprised if it's not right since I've been burned by config spacing before. Appreciate the input!

EDIT: I think I finally got it. Indent issue like you called it. This appears to be working, though not all the fields are being dropped (specifically "agent.ephemeral_id", "agent.id") but a lot of the extra stuff is gone.

  processors: (2 spaces)
    - drop_fields: (4 spaces)
        fields:  (8 spaces)
1
InvestingIsHard•

I think it should be like this:

filebeat.inputs:

- type: syslog
  enabled: true
  format: auto
  protocol.udp:
    host: "192.168.2.253:514"
  fields:
    event.type: vmware
  fields_under_root: true

  processors: (2 spaces)
    - drop_fields: (4 spaces)
        fields: (8 spaces)
2
InvestingIsHard•

Some of the system fields are added later in the processing pipeline, so dropping them at the input level like you are doing won't work. I think if you either move your processors or add a second one at a global level then it would likely work.

filebeat.inputs:

- type: syslog
  enabled: true
  format: auto
  protocol.udp:
    host: "192.168.2.253:514"
  fields:
    event.type: vmware
  fields_under_root: true
  # Input specific processors
  processors: (2 spaces)
    - drop_fields: (4 spaces)
        fields: (8 spaces)

# Global processors, apply to all events later in the pipeline
processors: (no spaces)
  - drop_fields: (2 spaces)
      fields: (6 spaces)
2
gregoldeOP•

That was it! Thank you!!!

-Dropping the existing global processor cleaned up some of my fields, but not all.

-Fixing the spacing of my input fixed a few more fields, but not all, as you mentioned they're added later.

-Moving the remaining fields to my global processor took care of the rest.

Thank you again for the help. I've been banging my head against this all day.

1