New day, new task as an operation guy. This time I have been assigned a task to expand one of our production ElasticSearch clusters. The target; perform it without the downtime of our application nor the cluster. Simple enough, but you never know what can happen down the road, so let's try to prepare the expansion and see what we can expect of this activity.Current cluster setupAt the moment our current cluster consist of 12 old nodes which we are going to decommission and replace with a shiny brand new 6 nodes, this should give us more breathing space for the future.Current ElasticSearch cluster 12 data nodes + 1 master nodeThe planThanks to the flexibility of ElasticSearch, the expansion should be quite easy and almost fully handled by ElasticSearch itself due to its scalability, the high level steps should look as follow, we will get into the details of each step later: Roll out ElasticSearch onto the new hardware. Disable shard allocation in the cluster Startup newly installed ElasticSearch nodes and check, if they have joined the cluster successfully Enable shard allocation in the cluster and wait until the shards within the cluster get distributed to the new nodes Start to decommission old nodes from the cluster one by one. Roll out ElaticSearch onto the new hardwareI won't go into much details in this part as your ElasticSearch installation might differ, but you should make sure that the following is set properly for the new nodes: The cluster name is the same as on the old nodes. New data nodes can reach your master nodes, it is enough just to set master nodes into the zen discovery. If you use ElasticSearch backup repositories, make sure that it's set on all the nodes, otherwire the node won't startup. No issue on our side as the nodes are provisioned via Ansible playbook, we can proceed further onto the next step. Disable shard allocation in the clusterUsually you don't have to do this step, but my plan is to join the nodes into the cluster all at once, but I don't want ElasticSearch to assign the shards to the nodes straight away as I want to check, if the nodes are fully operational, hence this step. Shard allocation can be disabled cluster wide via following API call:root@es-master-1:~# curl -X PUT localhost:9200/_cluster/settings \ > -H 'Content-Type: application/json' \ > -d'{ > persistent : { > cluster.routing.allocation.enable : none > } > }' { acknowledged :true, persistent :{ cluster :{ routing :{ allocation :{ enable : none }}}}, transient :{}} Once you disable the shard allocation we can move on and startup new ElasticSearch instances.Startup newly installed ElasticSearch nodes and check, if they have joined the cluster successfullyStartup your nodes via whatever method is most suitable for your ElasticSearch deployment; check, if the nodes have joined the cluster without any issues.New 6 nodes added to the clusterEnable shard allocation in the cluster and wait until the shards within the cluster get distributed to the new nodesTime to enable shard allocation and wait until the cluster rebalances and some shards get moved to our new data nodes.root@es-master-1:/app/products/elasticsearch/logs# curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d' > { > "persistent": { > "cluster.routing.allocation.enable": null > } > } > ' {"acknowledged":true,"persistent":{},"transient":{}}Start to decommission old nodes from the cluster one by oneNow the fun begins, the shards have been rebelanced and we can start to decommission our old nodes. First we have to make sure that no shards are allocated to the node that we are about to decommission. To do that we have to move all shards away from it via following command:root@es-master-1:/app/products/elasticsearch/logs# curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d' > { > "transient" : { > "cluster.routing.allocation.exclude._ip" : "10.135.136.62" > } > } > ' {"acknowledged":true,"persistent":{},"transient":{"cluster":{"routing":{"allocation":{"exclude":{"_ip":"10.135.136.62"}}}}}}Shards being moved from the old nodeAs you can see ElasticSearch is moving away all the primary shards from the node, once there are no shards on the node, we can switch off the node to decommission it from the cluster.Empty nodeNow that all the shards have been moved out of es-data-1 node we can switch it off and remove it from the cluster, rinse and repeat until you are left with only the new nodes in your cluster.Finishing upOnce I've decommissioned all the old nodes I ended up with something like this:Fully functional cluster running on brand new 6 nodes, no downtime needed at all, just make sure that you are of course not performing this during the peak time. :)
Your Content Goes Here Online koncerty už čoskoro. The post Online koncerty appeared first on Súkromná základná umelecká škola.
prejsť na článokVe středu 26. 8. 2015 bude spuštěna volná registrace nové generické domény .online. Buďte online s Czechia.com
prejsť na článokRegistrujte nové domény ONLINE, nebo doménu PW se super slevou!
prejsť na článokRegistrujte doménu ONLINE za super akční cenu 1099 Kč přímo na 10 let a získejte spoustu výhod!
prejsť na článokDnes tu máme niečo pre elearning designérov. iSpring práve ukončil každoročnú online konferenciu venovanú digitálnemu vzdelávaniu iSpring EDU Days 2024. Ak ste ho nestihli, online záznamy jednotlivých prednášok sa nachádzajú tu: >> iSpring EDU Days 202