Vega-Lite with Kibana to Parse and Display IP Activity over Time, (Tue, Aug 27th)

I have been curious for a while looking at Kibana's Vega log parsing options to try to come up with displays and layout that aren't standard in Kibana. A lot of the potential layouts already exists in Kibana but some of the other aren't easily created and using Vega [2] provides some of the building block to create some of the output that I am researching and testing with DShield sensor data captured by cowrie honeypot [4].

Building a Query in the Visualize Library

In my test query, I wanted to display on the left of the graph the IP List and in the bottom, the date of the activity. This way when I choose to summarize the activity by IP or any of the other fields I happen to select, it will display the activity by date of any IP that was active over time.

A text copy of the JSON code is posted at the bottom. This simple query takes the data from cowrie logs as its input to format the output:

The Data Output

Before I zoom in the time of interest to see some of the long-term activity, this is what the up to 10000 records looks like with all the IPs displayed in this picture. It is now easy to see a cluster of activity in this picture, next we need to zoom in the time of the activity to find which IP has this cluster.

After I zoom in the data, the result of a 7-day query provides the following data of a cluster of activity over time. In this picture, you can see that one IP 193.201.9.156 was active for several hours between 22 Aug 06:00 - 22 Aug 21:00.

DShield SIEM Integration

The primary goal of this test is to integrate this into the DShield SIEM [1] ELK Stack to be able to see overtime which actor are active and how long can they be seen over time in one of the dashboards. Now that we have an IP to look at, the time range can be expended as far as I want, and this picture shows activity of IP 193.201.9.156 over the past 30 days.

Sample Vega-Lite Query

This is the code used in the above example:

{   $schema: https://vega.github.io/schema/vega-lite/v5.json   title: Cowrie Logs - Actor Activity over Time   data: {     url: {       %context%: true       %timefield%: @timestamp       interval: {%autointerval%: true}       index: cowrie*       body: {         size: 10000         _source: ["@timestamp","related.ip", "source.address", "user.name"]       }     }     format: {property: "hits.hits"}   }

  transform: [    {calculate: "toDate(datum._source['@timestamp'])", as: "Time"},   {"calculate": "datum._source['related.ip']", "as": "IP"},   {"calculate": "datum._source['user.name']", "as": "Name"}   ]   mark: square   encoding: {     // https://vega.github.io/vega-lite/docs/timeunit.html#input     // Change timeUnit to display Month Day and Hour of activity     x: {"timeUnit": "monthdatehours",field: "Time", type: "ordinal", title: "Date/Time" }     y: {field: "IP", type: "ordinal", title: "Actor IP Address"}     color: {field: "IP", type: "ordinal", legend: null}   }  }

[1] https://www.elastic.co/guide/en/kibana/current/vega.html [2] https://vega.github.io/vega/examples/ [3] https://github.com/bruneaug/DShield-SIEM [4] https://github.com/DShield-ISC/dshield

----------- Guy Bruneau IPSS Inc. My Handler Page Twitter: GuyBruneau gbruneau at isc dot sans dot edu

(c) SANS Internet Storm Center. https://isc.sans.edu Creative Commons Attribution-Noncommercial 3.0 United States License.