Setup Heka to ES

Hello . We have some log for collective management, so we choose Heka open source software .

Advantage

  • Golang development.
  • Like Logstash
  • better performance

Topology

heka-client ——-> ES server

Setup heka

1
2
https://github.com/mozilla-services/heka/releases/download/v0.9.2/heka_0.9.2_amd64.deb
dpkg -i heka_0.9.2_amd64.deb

Configure heka

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
[hekad]
maxprocs = 1 # control heka use CPU core
# define log path and decoder
[strongs_log]
type = "LogstreamerInput"
log_directory = "/var/log/myselflog"
file_match = 'myselflog\.log'
decoder = "myselflog_decoder"
# define input decoder
[myselflog_decoder]
type = "PayloadRegexDecoder"
match_regex = '^(?P<Time>\w+\s+\d+ \d+:\d+:\d+) (?P<ID>[0-9]{2}\[[a-zA-z]{3}\]) (?P<Data>.*)'
timestamp_layout = "May 25 02:07:37"
timestamp_location = "UTC"
# add some fields at one line json
[myselflog_decoder.message_fields]
Type = "myselflog_log"
Logger = "Hello"
Hostname = "ec2-hostname"
Zone = "us-west-2c"
Time = "%Time%"
ID = "%ID%"
Data = "%Data%"
# define ES index and fields
[ES_Encoder]
type = "ESJsonEncoder"
index = "%{Type}-%{2006.01.02}"
es_index_from_timestamp = true
type_name = "%{Type}"
fields = ["Timestamp", "Type", "Hostname", "Logger", "Fields"]
# define ES Output option
[ElasticSearchOutput]
message_matcher = "Type =~ /.*/"
server = "http://ES_Server:9200/"
flush_interval = 500
flush_count = 200
encoder = "ES_Encoder"
queue_full_action = "shutdown"
queue_max_buffer_size = 10737418240 #10G
  • We have 200G log at every day . The heka also work to good .