Feature Request: saved search queries and/or search history

[wjimenez5271]

If there was a way to save search queries that are frequently used. Finding that I build really useful queries that I would like to reuse often, and I have to save it outside the UI. If there was a way to create a list of "favorites" or something like that would make finding the right data in Kibana even easier. Having a history of searches would be nice as well, or in place of it is easier to implement.

[jgoldschrafe]

Seconding this. I know it's a pain to turn what's currently a fairly stateless application into something that will need to do significant CRUD work to implement features people are asking for, but there's a lot of potential in the (really fast!) search interface that's getting passed over by my group because Graylog2 can just do a lot more right now.

[tpischke]

This is something my group is hoping for as well. I assume that it will be implemented sooner or later, and for that reason we have chosen kibana over graylog2 for it's simplicity and obvious strengths.

[awheeler]

You can work around this somewhat as the url can be bookmarked, and it contains the complete query.
There is thought on how to implement this in the GUI though, so it is coming.
The hashcodes in the url can be decoded here: http://tools.web-max.ca/encode_decode.php using base64 decode.

[crashdump]

This is definitely the missing feature of this UI !
(And also save the columns somewhere..)

[azulcactus]

I want this too. I had actually coded something to do this in the old PHP version of Kibana. I had a "Saves" button on the right side of the search bar (between "Search" and "Reset") and it triggered a popup allowing you to either save your current "query" and assign it a name, or select an existing save from the drop down. It simply reads/writes the URL hash plus your defined name to a file on the file system. No need to encode/decode anything.

I had hacked up my PHP Kibana pretty well, but if I can find time I'll see if I can extract that code and port the changes to Ruby.

[invadersmustdie]

+1 for this feature

I think it wouldn't be hard to implement a simple user-management using redis or another "light" storage solution, once you got there adding and sharing "saved queries" is just a minor step.

I'm currently in the situation of introducing logstash + kibana as a splunk alternative, but "saved queries" seems to be a must-have feature for the client. Thus I'm willing to spent some time on this if you're up to this ..I'm not interested in maintaining a fork.

[pkubat]

No really "the correct" solution, but Kibana is already connect to a "storage solution" ... ElasticSearch. Could just store it there. Something to be said for KISS and less "parts"

[invadersmustdie]

I must agree that the idea of using the already present ElasticSearch as storage is tempting, but in my opinion it is not clever to use the logstash "data sink" as store for users and other items (eg saved searches) .. that clearly violates the boundary between those two systems. Kibana is a wonderful drop-in replacement for the existing logstash-web frontend and so as an "drop-in" it shouldn't mess around with primary data.

Anyways, when you're going to use ElasticSearch as storage I would recommend using a separate instance .. though I still prefer using a more lightweight store like redis.

[untergeek]

It wouldn't interfere with logstash at all! Make an index, call it KibanaConfig or something like that. It might not be the lightest weight, but it certainly wouldn't interfere with logstash's indexes if named differently.

Not everyone will have redis, but everyone will have elasticsearch. As silly as it sounds, the idea is simpler to handle in Elasticsearch. It also becomes trivial to add other saved searches, cache results from logstash searches, etc. I think using Elasticsearch is a fine idea.

[invadersmustdie]

Again, it's tempting but lets assume the typical setup. There are shipper nodes and the indexer which are assembled in the preferred way:

node1:[logstash-shipper] -> node2:[(redis|messagebroker) ~> logstash-indexer -> elasticsearch]

Using the existing ElasticSearch as storage would make sense if Kibana is running on node2, but in a larger setup you will extract Kibana on a separate node and treat the indexer (node2) as a blackbox which provides a read-only interface to any frontend you want to use.

The resulting setup will look like this:

node1:[logstash-shipper] -> node2:[(redis|messagebroker) ~> logstash-indexer -> elasticsearch] *<--(read)-- node3:[(Kibana|frontend)]

Just imagine someone else is going to use another logstash frontend and wants to store user related data, following your proposal it would be totally fine to use the ElasticSearch instance on node2. Looking at the scenario the architecture will result in a shared storage for unrelated systems (unnecessary coupling).

In the end it's Rashids (@rashidkpc) decision (not offloading any responsibility here ;) ), we got a bunch of good proposals and I'm sure we can keep this discussion running for some more days but lets wait for his opinion.

[untergeek]

Elasticsearch has no security whatsoever. There is no concept of read-only access to the elasticsearch data. If one can access the elasticsearch IP on port 9200 to read, one will have access to do writes as well. Therefore it won't matter where Kibana is, whether in another node or not. My own setup has apache fronting kibana by way of mod_proxy, and then using passwords, SSL, and other apache security measures.

[rashidkpc]

I'll start by saying: This feature is important and well, inevitable.

As untergeek pointed out, Elasticsearch provides no security at all. While I take serious issue with that, there exists little other option for a transparent implementation. Logstash and Elasticsearch are Kibana's only absolute infrastructure dependencies and I'd like to keep moving parts to a minimum.

That said, when we do implement this, I would like to base it on the work done by @jimi1283 and @awheeler who have designed the start of a fantastic, and flexible, auth subsystem. We could provide the same sort of modularity for stored queries, with a choice of storage backends and Elasticsearch as the default. If someone wants redis, or mongo, or sqlite or X, it should be fairly easy to make that happen.

Side note: Firewalls aren't a perfect security paradigm, but they're the best we have for Elasticsearch right now. I suggest firewalling off your ES cluster at the host level to only be accessible from hosts that need it. Puppet + exported resources + firewall module make this pretty convenient

[invadersmustdie]

Fair enough. I took a peek into @jimi1283 implementation. The abstraction of the "StorageModule" provides an exchangeable interface I can work with.

So what's the plan on this feature request and how can we help?

PS: There are ways of securing ElasticSearch (https://github.com/Asquera/elasticsearch-http-basic, https://github.com/sonian/elasticsearch-jetty, ...) and just because it can't be ensured on a practical level doesn't mean you can't do it on a conceptual level.

[awheeler]

Take a look at @rashidkpc kibana-ruby-auth branch to see how this is developing.

[azulcactus]

I wouldn't try to mix authentication/authorization into this. There isn't any reason for the two to be clubbed together. Saves/recent queries can be supported without that, they just wouldn't be able to be associated to any account. Not a big deal. When account management comes along, then this could be tweaked.