Index Routing

When using logstash as a broker, you will want to route events to different indexes according to their type. You have two basic ways to do this;
  • Using Mutates with a single output
  • Using multiple Outputs
The latter is significantly better for performance. The less you touch the event, the better it seems.  When testing these two different configs in the lab, the multiple output method was about 40% faster when under CPU constraint.  (i.e. you can always add more CPU if you want to mutate the events.)

Multiple Outputs


input {
  ...
  ...
}
filter {
  ...
  ...
}
output {

  if [type] == "RADIUS" {
    elasticsearch {
      hosts => ["localhost:9200"]
      index => "logstash-radius-%{+YYYY.MM.dd}"
    }
  }

  else if [type] == "RADIUSAccounting" {
    elasticsearch {
      hosts => ["localhost:9200"]
      index => "logstash-radius-accounting-%{+YYYY.MM.dd}"
    }
  }

  else {
    elasticsearch {
      hosts => ["localhost:9200"]
      index => "logstash-test-%{+YYYY.MM.dd}"
    }
  }

}



Mutates

If your source system includes a field to tell you want index to place it in, you might be able to skip mutating altogether but often you must look at the contents to make that determination. Doing so does reduce performance.

input {
  ...
  ...
}
filter {
  ...
  ... 

  # Add a metadata field with the destination index based on the type of event this was
  if [type] == "RADIUS" {
    mutate { add_field => { "[@metadata][index-name]" => "logstash-radius" } } 
  }
  else  if [type] == "RADIUSAccounting" {
    mutate { add_field => { "[@metadata][index-name]" => "logstash-radius-accounting" } } 
  }
  else {
    mutate { add_field => { "[@metadata][index-name]" => "logstash-test" } } 
  }
}
output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "%{[@metadata][index-name]}-%{+YYYY.MM.dd}"
  }
}

Comments