KSQL provides the ability to create windowed aggregations. For example, count the number of messages in a 1 minute window, grouped by a particular column: CREATE TABLE RATINGS_BY_CLUB_STATUS AS \ SELECT CLUB_STATUS, COUNT(*) AS RATING_COUNT \ FROM RATINGS_WITH_CUSTOMER_DATA \ WINDOW TUMBLING (SIZE 1 MINUTES) \ GROUP BY CLUB_STATUS; How KSQL, and Kafka Streams, stores the window timestamp associated with an aggregate, has recently changed. See #1497 for details. Whereas previously the Kafka message timestamp (accessible through the KSQL ROWTIME system column) stored the start of the window for which the aggregate had been calculated, this changed in July 2018 to instead be the timestamp of the latest message to update that aggregate value.

Continue Reading

Using the holt function in Timelion to do anomaly detection on Metricbeat data in Kibana: Expression: $thres=0.02, .es(index='metricbeat*',metric='max:system.cpu.user.pct').lines(1).if(eq, 0, null).holt(0.9, 0.1, 0.9, 0.5h).color(#eee).lines(10).label('Prediction'), .es(index='metricbeat*',metric='max:system.cpu.user.pct').color(#666).lines(1).label(Actual), .es(index='metricbeat*',metric='max:system.cpu.user.pct').lines(1).if(eq, 0, null).holt(0.9, 0.1, 0.9, 0.5h).subtract(.es(index='metricbeat*',metric='max:system.cpu.user.pct')).abs().if(lt, $thres, null, .es(index='metricbeat*',metric='max:system.cpu.user.pct')).points(10,3,0).color(#c66).label('Anomaly').title('max:system.cpu.user.pct / @rmoff') References: https://twitter.com/rashidkpc/status/762754396111327232 https://github.com/elastic/timelion/issues/87 https://github.com/elastic/timelion/blob/master/FUNCTIONS.md

Continue Reading

I wrote recently about Kibana’s excellent Timelion feature, which brings time-series visualisations to Kibana. In the comments Ben Huang asked: do you know how to show whats the difference between this Friday and last Friday by Timelion? So I thought I’d answer properly here. Timelion includes mathematical functions including add and subtract, as well as the ability to show data offset by an amount of time. So to answer Ben’s query, we combine the two.

Continue Reading

Yesterday I wrote about Monitoring Logstash Ingest Rates with InfluxDB and Grafana, in which InfluxDB provided the data store for the ingest rate data, and Grafana the frontend. Mark Walkom reminded me on twitter that the next release of Logstash will add more functionality in this area - and that it’ll integrate back into the Elastic stack: @rmoff nice, LS 5.0 will have APIs exposing metrics too. they’ll be integrated back into Marvel/Monitoring!

Continue Reading

Kibana is a tool from Elastic that makes analysis of data held in Elasticsearch really easy and very powerful. Because Elasticsearch has very loose schema that can evolve on demand it makes it very quick to get up and running with some cool visualisations and analysis on any set of data. I demonstrated this in a blog post last year, taking a CSV file and loading it into Elasticsearch via Logstash.

Continue Reading

Timelion was released in November 2015 and with the 4.4.2 release of Kibana is available as a native visualisation once installed. It adds some powerful capabilities to Kibana as an timeseries analysis tool, using its own data manipulation language. Installing Timelion is a piece of cake: ./bin/kibana plugin -i kibana/timelion After restarting Kibana, you’ll see it as an option from the application picker There’s a bit of a learning curve with Timelion, but it’s worth it.

Continue Reading

OK, maybe that’s not entirely true. But my read-only client, certainly. I was perusing the Logstash input plugins recently when I noticed that there was one for IRC. Being a fan of IRC and a regular on the #obihackers channel, I thought this could be fun and yet another great example of how easy the Elastic stack is to work with. Installation is a piece of cake: wget https://download.elasticsearch.org/elasticsearch/release/org/elasticsearch/distribution/zip/elasticsearch/2.2.1/elasticsearch-2.2.1.zip wget https://download.

Continue Reading