Building a SEIM Dashboard with R, Jupyter, and Logstash/Elastic Search

January 5, 2018 by

Building a SEIM Dashboard with R, Jupyter, and Logstash/Elastic Search

This was originally posted on blogger here.

Motivation:

I am disappointed with the dashboards offered by today's SEIMs. SEIM dashboards offer limited data manipulation through immature, proprietary query languages and limited visualization options. Additionally, they tend to have proprietary data stores that limit expansion and evolution to what the vendor supports. Maybe I'm spoiled by working in R and Rstudio for my analysis, but I think we can do better.

Plan:

This blog is mainly going to be technical steps vs a narrative. It is also not the easiest solution. The easiest solution would be to already have the ELK stack, install interact.io, R, the R libraries, and the R jupyter kernel on your favorite desktop, and connect. That said, I'm going to walk through the more detailed approach below. You can view the example notebook HERE. Make sure to scroll down to the bottom where the figures are as it has a few long lists of fields.

Elastic search is becoming more common in security, (e.g. 1, e.g. 2). Combine that with the elastic package for R, and that should bring all of the great R tools to our operational data. Certainly we can create regular reports using Rmarkdown, but can we create a dashboard? Turns out with Jupyter you can! To test it out, I decided to stand up a Security Onion VM, install everything needed, and build a basic dashboard to demonstrate the concept.

Process:

Install security onion:

Security onion has an EXCELLENT install process. Simply follow that.

Install R:

Added ‘deb https://mirrors.nics.utk.edu/cran/bin/linux/ubuntu trusty/‘ to packages list

sudo apt-get install r-base

sudo apt-get install r-base-dev

— based off r-project.org

Install R-studio (not really necessary but not a bad idea)

Downloaded r-studio package from R-studio and installed

Sudo apt-get install libjpeg62

sudo dpkg -I package.deb

Install Jupiter:

(https://www.digitalocean.com/community/tutorials/how-to-set-up-a-jupyter-notebook-to-run-ipython-on-ubuntu-16-04)

Sudo apt-get install python-pip

sudo pip install —upgrade pip (required to avoid errors)

sudo -H pip install jupyter

Install Jupyterlab: (probably not necessary)

Sudo -H pip install jupyterlab

Sudo jupyter serverextension enable --py jupyterlab --sys-prefix

Install Jupiter dashboard

(https://github.com/jupyter/dashboards)

sudo -H pip install jupyter_dashboards

sudo -H pip install --upgrade six

Sudo jupyter dashboards quick-setup --sys-prefix

Install R packages & Jupypter R kernel:

Sudo apt-get install libcurl4-openssl-dev

sudo apt-get install libxml2-dev

Start R

install.packages("devtools") # (to install other stuff)

install.packages(“elastic”) # talk to elastic search

install.packages(“tidyverse”) # makes R easier

install.packages("lubridate") # helps with working with dates

install.packages("ggthemes") # has good discrete color palettes

install.packages("viridis") # has great continuous colors

# https://github.com/IRkernel/IRkernel

devtools::install_github('IRkernel/IRkernel')

# or devtools::install_local('IRkernel-master.tar.gz')

IRkernel::installspec() # to register the kernel in the current R installation

quit() # leave. Answer ’n’ to the question “save workspace?”

Install nteract: (Not necessary)

(nteract.io)

Download the package

Sudo apt-get install libappindicator1 libdbusmenu-gtk4 libindicator7

sudo dpkg -i nteract_0.2.0_amd64.deb

Set up the notebook:

Rather than type this all out, you can download an example notebook. In case you don't have an ES server populated with data, you can download this R data file which is a day of windows and linux server logs queried from ES from a blue vs red CTF.

I created the notebook using nteract.io so it is in a single order. However, if you open it on the juypter server, you can use the dashboards plugin to place the cells where you want them in a dashboard.

Results:

A lot of time spent compiling.

No need to download R/jupyter stuff on security onion if elastic search is remotely reachable.

Elastic search is not intuitive to query. Allowing people an 'easy mode' to generate queries would be significantly helpful. the `ES()` function in the workblook is an attempt to do so.

It would be nice to be able to mix interactive and dashboard cells.

This brings MUCH more power for both analysis _and_ visualization to the dashboard.

This brings portability, maintainability (ipynb files can be opened anywhere that has the R/jupyter environment and can access elastic search. They can also be forked, version controlled, etc.)

Future Work:

Need a way to have cells refresh every few minutes, likely a jupyter notebook plugin.

Interactive figures require interactive plotting tools such as Vega. This would also bring the potential ability to stream data directly to the notebook. It may even solve the ability to auto-refresh.

Conclusion:

In conclusion, you really don't want to roll-your-own-SEIM. That said, if you already have ES (or another data store R can talk to) in your SEIM and want less lock-in/more analysis flexibility, R + Jupyter may be a fun way to get that extra little emph. And hopefully in the future we'll see SEIM vendors supporting general data science tools (such as R or Python) in their query bars and figure grammars (ggplot, vega, vegalite), in their dashboards.

25 comments captured from original post on Blogger

Gabe said on 2018-01-09

I’m super excited that someone appears to be way farther along in this than me! Check it out at: https://github.com/Cyb3rWard0g/HELK

jhon said on 2019-02-27

I would definitely thank the admin of this blog for sharing this information with us. Waiting for more updates from this blog admin. Melbourne CCTV Installers

Jack Johnny said on 2019-04-10

I like viewing web sites which comprehend the price of delivering the excellent useful resource free of charge. I truly adored reading your posting. Thank you! Building surveillance camera system upgrade

Richard H. Black said on 2019-06-08

This is my first time visit to your blog and I am very interested in the articles that you serve. Provide enough knowledge for me. Thank you for sharing useful and don’t forget, keep sharing useful info: security camera installation

Tanika Co Valda said on 2019-06-25

Great Article R Project Topics for Computer Science FInal Year Project Centers in Chennai JavaScript Training in Chennai JavaScript Training in Chennai

Richard H. Black said on 2019-07-11

I’m going to read this. I’ll be sure to come back. thanks for sharing. and also This article gives the light in which we can observe the reality. this is very nice one and gives indepth information. thanks for this nice article… Serious Security Melbourne

Richard H. Black said on 2019-10-01

james john said on 2019-10-17

The article was up to the point and described the information very effectively. Thanks to blog author for wonderful and informative post. Security Solution consultant

Faiza Jee said on 2019-12-21

Thanks for sharing the post.. parents are worlds best person in each lives of individual..they need or must succeed to sustain needs of the family. CCTV camera

Best said on 2019-12-29

This is such a great resource that you are providing and you give it away for free. I love seeing blog that understand the value of providing a quality resource for free. Hikvision DS-2CD2385G1-I

Richard H. Black said on 2020-01-10

Awesome and interesting article. Great things you’ve always shared with us. Thanks. Just continue composing this kind of post. CCTV Installers Melbourne

Jason Roy said on 2020-03-09

This outdoor security camera system provides clear colored images during the day and black and white images at night. This type facilitates nighttime surveillance with various resolutions and lens angles. best security cameras

Richard H. Black said on 2020-03-20

If you are looking for more information about flat rate locksmith Las Vegas check that right away. Security Systems Melbourne

Richard H. Black said on 2020-04-16

Hello, this weekend is good for me, since this time i am reading this enormous informative article here at my home. Sydney CCTV Installation

yoyo said on 2020-04-22

Great post, and great website. Thanks for the information! cctv camera

Touch Tec said on 2020-05-23

TouchTec is a leading company in region with more than 10 years of experience that provides Security, Safety and Surveillance Solutions with high assurance to improve security and efficiencies for identity management, access to critical facilities, intelligence analysis, guest worker programs, and national identity programsCCTV for SchoolCctv Camera for Office

Best said on 2020-06-11

We have sell some products of different custom boxes.it is very useful and very low price please visits this site thanks and please share this post with your friends. Home Security Systems

for ict 99 said on 2020-08-09

Great Article Artificial Intelligence Projects Project Center in Chennai JavaScript Training in Chennai JavaScript Training in Chennai

Domar Solutions Ltd said on 2020-08-27

I totally agree with this article and I just want to say that this article is a very nice and very informative article.I will make sure to be reading your blog more. Opt for the home wireless cctv kits

Michael Smith said on 2020-09-16

Many structure execution issues can be followed to air spillage through the structure envelope. These issues go from high warming expenses and helpless temperature control in consumed spaces, to rain entrance and the decay different segments inside a structure get together. maintenance scheduling

SHAHZAIB said on 2021-06-09

Great write-up, I am a big believer in commenting on blogs to inform the blog writers know that they’ve added something worthwhile to the world wide web!.. Hikvision Acusense

vivikhapnoi said on 2021-06-17

I simply couldn’t resist praising the way you play with words. This is a perfect example of a well-written blog post.khi nào có vé máy bay từ mỹ về việt namchuyến bay từ paris về hà nộivé máy bay từ singapore về hà nội vietjetđặt vé máy bay từ úc về việt namLịch bay từ Hàn Quốc về Việt Nam hôm nayđặt vé máy bay giá rẻ tu Nhat Ban ve Viet Nam

RazaSEO5 said on 2021-09-17

Thank you because you have been willing to share information with us. we will always appreciate all you have done here because I know you are very concerned with our. Serious CCTV Melbourne

Iyaz Khan said on 2022-08-18

Thanks for sharing this.

William Woodruff said on 2022-08-18

When collecting data, whether qualitative or quantitative, we can use several tools, including surveys, focus groups, interviews, and questionnaires. It depends on you what kind of tool you use to help organize the data. Dissertation Editing Service