Skip to content

multi es-instances support - #1225

Closed
clarkchen wants to merge 3 commits into
YelpArchive:masterfrom
clarkchen:master
Closed

multi es-instances support#1225
clarkchen wants to merge 3 commits into
YelpArchive:masterfrom
clarkchen:master

Conversation

@clarkchen

Copy link
Copy Markdown

I take advantages of Elasticsearch, make host param as a list.

Host Config Change to
"host1:port1, host2:port " can handle multi es-instances

while the original port and host can also work well here

@clarkchen

Copy link
Copy Markdown
Author

hello?

@Qmando Qmando left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documentation?

Comment thread elastalert/util.py
if "es_host" in conf:
es_host = os.environ.get('ES_HOST', conf['es_host'])

if "es_host" in conf:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

es_port

Though, this also isn't necessary.

Comment thread elastalert/util.py
parsed_conf['es_conn_timeout'] = conf.get('es_conn_timeout', 20)
parsed_conf['send_get_body_as'] = conf.get('es_send_get_body_as', 'GET')

if "es_host" in conf:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This check isn't necessary, it's always there.

Comment thread elastalert/util.py
return parsed_conf


def parse_host(host, port="9200"):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not accept a list instead of a comma separated string?

@YoavBZ

YoavBZ commented Sep 7, 2017

Copy link
Copy Markdown

Nice! I've just started working on a similar feature :)
In my case, I wanted to run rules on few different hosts and didn't wanted to duplicates my rules, so I added a list of objects to the rule .yaml file that includes the ES hosts, ports and names, so the rule will run separately on each host.
I also appended the environment name to the rule name, i.e "Rule name [Env name]".

How this one is going to work?

@Qmando

Qmando commented Sep 7, 2017

Copy link
Copy Markdown
Contributor

I guess the motivation here is that you could replace a load balancer by having a pool of servers to roundrobin requests to.

@YoavBZ

YoavBZ commented Sep 7, 2017

Copy link
Copy Markdown

I see.. My idea was more for multiple environments support

@Qmando

Qmando commented Sep 7, 2017

Copy link
Copy Markdown
Contributor

So you implemented this feature to avoid duplicating YAML files, is that correct?

You can import bits and pieces of rules, so you could specify two files with just a es_host, name and import parameter.

@YoavBZ

YoavBZ commented Sep 7, 2017

Copy link
Copy Markdown

Yes, I wanted to avoid duplicating cause I have one rule with 2/3 ES hosts I want to run it on.
So instead of creating few duplicate rules but with different es_hosts I created one rule with few es_hosts in it

@YoavBZ

YoavBZ commented Sep 7, 2017

Copy link
Copy Markdown

*Not one rule actually, I need to run all my rules on few es_hosts

@sathishdsgithub

Copy link
Copy Markdown

@YoavBZ @Qmando @clarkchen

Can we make use of the above modified util.py for multi ES instance? I tried with below format but it did not work.
Please, can someone suggest me the right method to mention the list of ES host entry with port?

es_host: "192.168.96.157:9200, 192.168.96.141:9200"

@nsano-rururu

Copy link
Copy Markdown
Contributor

Can you resolve conflicts please :)

@clarkchen

clarkchen commented Jan 30, 2021

Copy link
Copy Markdown
Author

Can you resolve conflicts please :)

sorry, I have lost my origin repo, so I start a new pull request, where you can review code there.

thx~

@clarkchen clarkchen closed this Jan 30, 2021
@clarkchen

Copy link
Copy Markdown
Author

@sathishdsgithub

the ElasticSearchClient init should change to hosts
image

I add a test case which connect with a docker-composed elastic cluste

you can find the test case here

hope it will work

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants