Index Routing
When using logstash as a broker, you will want to route events to different indexes according to their type. You have two basic ways to do this;
- Using Mutates with a single output
- Using multiple Outputs
The latter is significantly better for performance. The less you touch the event, the better it seems. When testing these two different configs in the lab, the multiple output method was about 40% faster when under CPU constraint. (i.e. you can always add more CPU if you want to mutate the events.)
Multiple Outputs
input {
...
...
}
filter {
...
...
}
output {
if [type] == "RADIUS" {
elasticsearch {
hosts => ["localhost:9200"]
index => "logstash-radius-%{+YYYY.MM.dd}"
}
}
else if [type] == "RADIUSAccounting" {
elasticsearch {
hosts => ["localhost:9200"]
index => "logstash-radius-accounting-%{+YYYY.MM.dd}"
}
}
else {
elasticsearch {
hosts => ["localhost:9200"]
index => "logstash-test-%{+YYYY.MM.dd}"
}
}
}
Mutates
If your source system includes a field to tell you want index to place it in, you might be able to skip mutating altogether, but often you must look at the contents to make that determination. Doing so does reduce performance.
input {
...
...
}
filter {
...
...
# Add a metadata field with the destination index based on the type of event this was
if [type] == "RADIUS" {
mutate { add_field => { "[@metadata][index-name]" => "logstash-radius" } }
}
else if [type] == "RADIUSAccounting" {
mutate { add_field => { "[@metadata][index-name]" => "logstash-radius-accounting" } }
}
else {
mutate { add_field => { "[@metadata][index-name]" => "logstash-test" } }
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "%{[@metadata][index-name]}-%{+YYYY.MM.dd}"
}
}
https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#metadata
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.