Exploring Osquery, Kolide's Fleet and Graylog for Endpoint Visibility

There's a lot of bias here, but again I wanted it to be similar to what I am already doing, and as much as possible fit into infrastructure I have already running.

The pieces...\

Why?

  • Desire for fleet visibility
  • Visibility on both clients and hosts
  • Ability to be alerted if something changes, but not necessarily enforce
  • There's one of me
  • Easy to stand up
  • Easy to maintain
  • Automation
  • As much as I wish I had time to "dev" it- I just need it to work.

osquery

What is osquery? (abridged)

osquery exposes an operating system as a high-performance relational database. This allows you to write SQL-based queries to explore operating system data. With osquery, SQL tables represent abstract concepts such as running processes, loaded kernel modules, open network connections, browser plugins, hardware events or file hashes.

SQL tables are implemented via a simple plugin and extensions API. A variety of tables already exist and more are being written: https://osquery.io/schema. To best understand the expressiveness that is afforded to you by osquery, consider the following SQL queries:

List the users:

[code lang=sql]
SELECT * FROM users;
[/code]

Check the processes that have a deleted executable:

[code lang=sql]
SELECT * FROM processes WHERE on_disk = 0;
[/code]

Get the process name, port, and PID, for processes listening on all interfaces:

[code lang=sql]
SELECT DISTINCT processes.name, listening_ports.port, processes.pid
FROM listening_ports JOIN processes USING (pid)
WHERE listening_ports.address = '0.0.0.0';
[/code]

Find every OS X LaunchDaemon that launches an executable and keeps it running:

[code lang=sql]
SELECT name, program || program_arguments AS executable
FROM launchd
WHERE (run_at_load = 1 AND keep_alive = 1)
AND (program != '' OR program_arguments != '');
[/code]

A note on osqueryi vs osqueryd

osqueryi is the osquery interactive query console/shell. It is completely standalone and does not communicate with a daemon and does not need to run as an administrator. Use the shell to prototype queries and explore the current state of your operating system.

and

osqueryd is the host monitoring daemon that allows you to schedule queries and record OS state changes. The daemon aggregates query results over time and generates logs, which indicate state change according to each query. The daemon also uses OS eventing APIs to record monitored file and directory changes, hardware events, network events, and more.

Fleet by Kolide

So now we know just a little bit of what osquery can do, so how canwe automate that? Make that work for use en mass?

Thats where Fleet by Kolide comes in... Kolide offers another product as a SaaS option, Kolide Cloud:

Kolide Cloud is the fastest way to get started with Osquery in your organization. Following our setup guide, you can have Kolide on your machine and reporting insights in less than two minutes flat. Our pre-built packages make organization-wide deployment a piece of cake with the tools you already use today.

Understanding Fleet

Fleet will:\

  • hold our client reported information\
  • queue and send queries to clients/servers\
  • fleet will log any findings

Installing Fleet

NOTE: You can also just use the Quickstart method, and skip to Exploring Fleet...

Quickstart
Github Docs

Docker

If you want to use Docker, you can pull the latest Fleet docker image:
docker pull kolide/fleet

Binary

If want to use a linux vm or otherwise, here is an example of an Ubuntu 16.04 setup process, after your initial vm setup.

Get the latest binary, move it to /usr/bin as fleet

[code lang=text]
$ wget https://dl.kolide.co/bin/fleet_latest.zip
$ unzip fleet_latest.zip 'linux/*' -d fleet
$ sudo apt install unzip
$ sudo cp fleet/linux/fleet_linux_amd64 /usr/bin/fleet
[/code]

Installing MySQL:

[code lang=text]
$ sudo apt-get install mysql-client
[/code]

You can run this db local or on a remote db server thats up to you, for sake of illustration, lets say tou want to run it on a remote db server...

On your remote db server:

[code lang=sql]
$ mysql -u root -p
mysql> create database fleet;
create user 'fleet'@'$FLEET_IP' identified by '$PSW';
create user 'fleet'@'$FLEET_FQDN' identified by '$PSW';
grant all on fleet.* to 'fleet'@'$FLEET_IP';
grant all on fleet.* to 'fleet'@'$FLEET_FQDN';
flush privileges;
mysql> flush privileges;
mysql> select user,host from mysql.db where db='fleet';
+-------+-------------------+
| user | host |
+-------+-------------------+
| fleet | $FLEET_IP |
| fleet | $FLEET_FQDN |
+-------+-------------------+
2 rows in set (0.00 sec)
mysql> exit;
[/code]

So now we can prepare the fleet database via the binary we installed on the fleet host:

[code lang=text]
/usr/bin/fleet prepare db \ --mysql_address=your.database.com:port \
--mysql_database=fleet --mysql_username=fleet \ # or whatever was created
--mysql_password='$PSW' # or prompt for input
[/code]

This will prepare the db as needed, once this is done, you could serve fleet service via cli such as:

[code lang=text]
/usr/bin/fleet serve \
--mysql_address=your.database.com:port \ --mysql_database=fleet --mysql_username=fleet
--mysql_password='CiKv_Gzy9:>B]vJ-jifsm&'
--redis_address=127.0.0.1:6379
--server_cert=/your/host/Certs/fleet.pem
--server_key=/your/host/Certs/fleet.key \ --logging_json --auth_jwt_key $auth_jwt_key
[/code]

But fleet can also take a kolide.yaml config file, docs, so above becomes:

[code lang=yaml]
mysql:
address: your.database.com:port
database: fleet
username: fleet
password: $PSW
redis:
address: 127.0.0.1:6379
server:
address: 0.0.0.0:443
cert: /your/host/Certs/fleet2.pem
key: /your/host/Certs/fleet2.key
auth:
jwt_key: $auth_jwt_key
osquery:
status_log_file: /var/log/osquery/status.log
result_log_file: /var/log/osquery/result.log
label_query_update_interval: 1h
enable_log_rotation: true
logging:
json: false
debug: false
[/code]

I won't get into all of Fleets options, but most are the above are self explanatory.

Now we could launch fleet as such:

[code lang=text]
/usr/bin/fleet serve --config /your/config/kolide.yaml
[/code]

But I'd rather have it be controlled through systemd, launch automatically etc, so before we launch we can configure systemd

Lets create /etc/systemd/system/fleet.service

[code lang=text]
[Unit]
Description=Fleet web service

[Service]
ExecStart=/usr/bin/fleet serve --config /your/config/kolide.yaml
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=Fleet

[Install]
WantedBy=multi-user.target
[/code]

As well I edited rsyslog.d to send service logs to a stand alone file, /var/log/fleet/fleet-out.log dump them w/o sending to syslog, appending or created an rsyslog.d conf with the follow lines should ensure that:

[code lang=text]
if $programname == 'Fleet' then /var/log/fleet/fleet-out.log
if $programname == 'Fleet' then ~
[/code]

We can enable this service to be started at boot, if desired:

[code lang=text]
systemctl enable fleet.service
[/code]

And now we can start the service with:

[code lang=text]
systemctl start fleet.service
[/code]

As well get systemctl information on the server:

[code lang=text]
$ systemctl status fleet.service
● fleet.service - Fleet web service
Loaded: loaded (/etc/systemd/system/fleet.service; enabled; vendor preset: enabled)
Active: active (running) since ... ; x days ago
Main PID: 8722 (fleet)
Tasks: 11
Memory: 546.1M
CPU: 17min 39.910s
CGroup: /system.slice/fleet.service
└─8722 /usr/bin/fleet serve --config /your/config/kolide.yaml
[/code]

Fleet service handles the rotation of the logs as well, no need to worry about implementing additional logrotate

Exploring Fleet

For testing/demo, I recommend checking out Fleet via Docker which is found on Kolide's Fleet "Quickstart" Guide.
It can be as simple as

[code lang=text]
git clone https://github.com/kolide/kolide-quickstart.git
./demo.sh up
[/code]

and is well documented.

Enrolling Clients with Fleet

macOS

If you chose to do the Quickstart, you can run this from the cloned directory:
./demo.sh enroll mac which will generate a macOS installer package into ../out.

If you are running another iteration, perhaps more production, I encourage you to take a look at Kolide/Launcher for an osquery management tool...

Keep in mind the docker instance is bound to localhosts so demo clients need to be configured as such...

What this package does is install a few things:\

  • /LibraryLaunchDaemons/\
  • co.kolide.osquery.enroll.plist\
  • LaunchDaemon\
  • starts osqueryd on your system\
  • /etc/osquery/\
  • kolide.crt\
  • allows for tls communication\
  • will be self-signed in demo mode\
  • kolide_secret\
  • file containing secret\
  • allows for the client enrollment\
  • kolide.flags file\
  • list of preferences / parameters

NOTE: Fleet just configures kolide.flags based on osqueryd's built in clip-flags to communicate with the server, (which spec is outlined here).

16.04

If you decide to just bootstrap for funsies or demo via the Quickstart we can get this bootsrapped on 16.04 by...

You can install osquery with apt its required to be installed before its configured with the kolide.flags

[code lang=text]
export OSQUERY_KEY=1484120AC4E9F8A1A577AEEE97A80C63C9D8B80B
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys $OSQUERY_KEY
sudo add-apt-repository 'deb [arch=amd64] https://pkg.osquery.io/deb deb main'
sudo apt-get update
sudo apt-get install osquery -y
[/code]

As with the macOS installer we need all the same parts,\

  • /etc/osquery/\
  • kolide.crt\
  • kolide_secret\
  • kolide.flags

You can manually put these into /etc/osquery/

And then manually kick of osqueryd with this command:
/usr/bin/osqueryd --flagfile=/etc/osquery/kolide.flags

You should now see your clients in the server if the config was successful.

Fleet Documentation

#kolide in osquery Slack
API Documentation
Github kolide/fleet
Documentation

Graylog

A little out of scope here, but there are loads of options and plenty of good docs out there:\

Tips on ingesting osquery logs into Graylog

On our Fleet sever config.yaml we designated the following logs:\

  • status_log_file: /var/log/osquery/status.log\
  • result_log_file: /var/log/osquery/result.log

Now we can setup sending those to Graylog.

Aside:
I much prefer to send logs to individual files and set rules surrounding them individually. And keeping them separate from the syslog

If you chose to send those logs to syslog this portion may vary for you...

You can do this a couple different ways,

A) rsyslog

You could set up a rsyslog entry for that log and send it that way-

B) agent, ie Filebeat by Elastic

Filebeat is a tool from Elastic that allows for managing log forwarding for multiple files, even parsing log files, see the Filebeat Documentation for more...

Let me explain why I like B, primarily because I can deploy a config that manages all the needed settings for multiple log files, and its as simple as:

[code lang=yaml]
filebeat.prospectors:\

  • input_type: log
    paths:\
  • /var/log/osquery/status.log\
  • /var/log/osquery/result.log

# output
output.logstash:
hosts: ["your.graylog.com:PORT"]

#logging info

logging.level: info
logging.to_files: true
logging.to_syslog: false
logging.files:
path: /var/log/Filebeat/
name: filebeat.log
keepfiles: 7
[/code]

This config will watch just the listed log files, as well as log its doings to /var/log/Filebeat/filebeat.log. One BIG bonus I like about Filebeat is that its aware of what it has sent and hasn't...

In any environment, application downtime is always lurking on the edges. Filebeat reads and forwards log lines and — if interrupted — remembers the location of where it left off when everything is back online. filebeat

So if your syslog is down, or overloaded, and a successful log forward is not sent, then it knows and "catches up" later.

Once Filebeat is setup, you can configure an input for it on Graylog. Detailed info over at Graylog: Inputs

Once you are on youre inputs page, select Beats option:

{.alignnone .size-medium .wp-image-2197 width=”300” height=”31”}

This will bring up a config window, update as needed:

{.alignnone .size-medium .wp-image-2196 width=”153” height=”300”}

Additionally once the input is configured, "Manage Extractor":

{.alignnone .size-medium .wp-image-2195 width=”300” height=”26”}

And find an example log file, (you may have to come back to this part), and load it-
Select the message portion of the log, and then select Select extractor type which is JSON and that should be it!

The message block received will look like this:

[code lang=text]
{"name":"pack\/Example Check\/etc_hosts","hostIdentifier":"macOS","calendarTime":"Fri Apr 27 20:38:36 2018 UTC","unixTime":"1524861516","epoch":"0","counter":"0","decorations":{"host_uuid":"564D4617-52F7-6924-9698-05E433322827","hostname":"macOS"},"columns":{"address":"8.8.8.8","hostnames":"whatever"},"action":"added"}
[/code]

Now will be converted to:

[code lang=text]
action
added
calendarTime
Fri Apr 27 20:38:36 2018 UTC
columns_address
8.8.8.8
columns_hostnames
whatever
counter
0
decorations_host_uuid
564D4617-52F7-6924-9698-05E433322827
decorations_hostname
macOS
epoch
0
facility
filebeat
file
/var/log/osquery/result.log
hostIdentifier
macOS
message
{"name":"pack\/Example Check\/etc_hosts","hostIdentifier":"macOS","calendarTime":"Fri Apr 27 20:38:36 2018 UTC","unixTime":"1524861516","epoch":"0","counter":"0","decorations":{"host_uuid":"564D4617-52F7-6924-9698-05E433322827","hostname":"macOS"},"columns":{"address":"8.8.8.8","hostnames":"whatever"},"action":"added"}
name
pack/Example Check/etc_hosts
offset
10448
source
fleet
timestamp
2018-04-27T20:44:03.545Z
type
null
unixTime
1524861516
[/code]

And once an item is extracted as such, each of these keys will be stand alone inside Graylog. Allowing you to alert, sort and graph based on their contents.

Monitoring and Alerting with Graylog

Again we are quickly approaching scope creep for this session, Graylog has great docs and here are some good places to start:

Graylog Streams
Graylog Alerts

Resources

osquery

osquery
osquery Docs
osquery tables
Github facebook/osquery
osquery Slack

Fleet

#kolide in osquery Slack
API Documentation
Github kolide/fleet
Documentation

Graylog

Downloads
Documentation
Inputs
Filebeat