nxlog from Windows to ELK in Linux, trying to get this to work, no index in Kibana
Update: with kind help from u/sysval I had it pointed out to me that there is/was an index,I mistakenly though (as in, misunderstood) what indices are created and how, I thought that the logstash.conf related to an/the relevant index that would be created and that I should have been looking for.
Crux of the matter: Things are setup and "working" but no logged events seem to be arriving at the recipient server,and there's no available nxlog index to choose when attempting to finalize the setup of Kibana.
I'm new to ELK but not linux or Windows, but this is driving me bonkers as I'm clearly missing something. Help me Jedi masters ! Surely I'm missing something obvious or stupid to anyone who has set this up successfully.Details follow, the one hostname (for the Linux VM) has been obfuscated, but it has working DNS resolution to & for it.
I have an existing Linux VM (Ubuntu Server 18.04.1 LTS - nothing against RH or even Gentoo (except wanting to keep more of my time for other things)), with a working install of ELK, which was setup and used briefly by me for a test-flight of Kolide Fleet with a Windows client.
Anywho, I'm now trying to leverage the same (until now) left-unused VM to setup centralized logging from Windows hosts (servers, possible client PCs). I suppose - ideally - I'd love to bypass logstash and log directly to ELK - per https://nxlog.co/documentation/nxlog-user-guide-full#elasticsearch_sending and om_http but had less success attempting that route.
I'm starting with one windows server , and nxlog, using the config (in the end/additions) at Jess Dodsons' very fine post here, https://girl-germs.com/?p=438
I've also contrasted that with the nxlog guide for forwarding to log stash here, https://nxlog.co/documentation/nxlog-user-guide-full#elasticsearch_logstash
The Windows host nxlog "log" shows it's connected to the (linux VM) host (actual hostname obfuscated):2019-04-02 12:02:40 INFO nxlog-ce-2.10.2150 started
2019-04-02 12:02:40 INFO connecting to linux-server-name.company.com:3515
On the linux VM,I can watch the logstash log, and at startup (of logstash), see[2019-04-02T11:50:55,637][INFO ][logstash.pipeline ] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0xc716b2 run>"}
[2019-04-02T11:59:18,587][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.7.0"}
[2019-04-02T11:59:41,272][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-04-02T11:59:42,766][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2019-04-02T11:59:43,443][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-04-02T11:59:43,594][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}<snip>[2019-04-02T11:59:45,876][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
I can use telnet to successfully connect to the (recipient) host just fine viatelnet linux-server-name.company.com 3515
When I connect to/log in to Kibana athttps://linux-server-name.company.com/app/kibana#/management/kibana/index?_g=()
There's no index shown for nxlog.I was able to issue (locally, logged into the linux host via ssh),curl -H 'Content-Type: application/json' -XPUT localhost:9200/_template/nxlog -d '{ "template" : "nxlog*", "mappings" : { "_default_" : { "properties": { "EventTime": { "type": "date", "format": "YYYY-MM-dd HH:mm:ss" } } } } }'which returned:
{"acknowledged":true}
However in Kibana, the listed indices are%{[@metadata][beat]}-%{[@metadata][version]}-2019.03.31
logstash-2019.03.31
logstash-2019.04.01
logstash-2019.04.02
I verified the logstash config viasudo /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf -t
The relevant parts of the logstash config are below, with commented-out items from prior attempts at changes to get the desired result, and the extra tags are probably (?) superfluous:
input {
tcp {
#codec => json_lines { charset => "UTF-8" }
codec => json_lines { charset => "CP1252" }
port => 3515
tags => [ "tcpjson","eventlog","windows","nxlog" ]
type => "nxlog"
#type => "WindowsEventLog"
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]}
}
The nxlog config includes modifications per the Jess Dodson article,
<Input in>
Module im_msvistalog
SavePos TRUE
Query <QueryList>\
<Query Id="0"><Select Path="Security">*</Select>\
<Suppress Path="Security">*[System[(EventID=4624 or EventID=4776 or EventID=4634 or EventID=4672 or EventID=4688)]]</Suppress>\
<Select Path="System">*[System[(EventID=1074 or (EventID >= 6005 and EventID <= 6009) or EventID=6013)]]</Select>\
<Select Path="Microsoft-Windows-TerminalServices-LocalSessionManager/Operational">*</Select>\
</Query>\
</QueryList>
</Input>
<Output out>
Module om_tcp
Port 3515
Exec to_json();
</Output>
<Route 1>
Path in => out
</Route>
Posted Apr 2, 2019, 4:45 PM UTC · edited
I didn't read your wall of text. But I'm willing to help over TeamViewer in 30 minutes or so (kids are tugged in, they just need to admit they're tired)
root cause: couldn't see the forest for the trees
the logs were all there, it was just a matter of searching for them "type: nxlog"
tags is really helpful for these things in the ELK stack.
Thanks again, yes admittedly so.
Great, thanks. PM sent
Upvote to u/sysvival for the assist. Once you got it going - kindly update your post with the underlying issue/fix. SIEM is on the rise, relevant posts stay scarce ;)
Will do !
Agreed about relevant posts being scarce, or so it's been seeming.
Quite simply the original instructions I had (related to my OP here), had some unstated assumptions and are now a bit dated. Again, big thanks and props to u/sysvival
I was (am) looking to gain some, ANY (centralized) insight into AD-related activity.
Had good success installing Kolide Fleet and using it to run some osquery checks (queries) on a Windows client box I had some suspicions about. Used some additional items from the Palantir project, https://github.com/palantir/osquery-configuration
And the box came up OK. There IS anti-virus, I wanted a means of additional checks for things that could/have been known to bypass some AV products.
Next stop, Sysmon.
Thanks for sharing the update! Perchance a lucky coincidence - a few hours ago the following had been shared here on reddit: https://github.com/clong/DetectionLab Automated lab setup implementing a similar tool-set - maybe you can find some re-usable content there.
Some links - in case you haven`t already collected those docs: https://hitcon.org/2017/CMT/slide-files/d2_s1_r1.pdf (Tracking mimikatz with sysmon and elk) https://www.sans.org/cyber-security-summit/archives/file/summit-archive-1511995081.pdf (Mitre perspective, basic doc) https://github.com/MHaggis/sysmon-dfir/blob/master/README.md (great sysmon collection) https://www.echotrail.io/blog/hunting_on_sysmon_data/ (some additional basic stuff) https://posts.specterops.io/putting-sysmon-v9-0-and-or-grouping-logic-to-the-test-c3ec27263df8 (Sysmon grouping logic)
Happy hunting ;)
Not uncommon. A lot of the guides I've used to set stuff up require extra research to actually make it work.
For Active Directory related activity and Sysmon, you can also use NXLog to collect logs from those sources (and more)
See https://nxlog.co/documentation/nxlog-user-guide/ad-domain-controller.html for AD DC
See https://nxlog.co/documentation/nxlog-user-guide/sysmon.html for Sysmon