Consul user manual (feels quite complete)
Using consumer
introduce
Consul contains multiple components, but as a whole, it provides tools for service discovery and service configuration for your infrastructure. It provides the following key features:
Service discovery Consul clients can provide a service, such as api or mysql. Other clients can use consul to find a provider of a specified service. It is easy to find the service they depend on through DNS or HTTP applications
health examination The consult client can provide any number of health checks, specify a service (for example, whether the webserver returns 200 OK status code) or use a local node (for example, whether the memory usage is greater than 90%). This information can be used by the operator to monitor the health of the cluster. It is used by the service discovery component to avoid sending traffic to unhealthy hosts
Key/Value storage Applications can use Consul's hierarchical Key/Value storage according to their own needs, such as dynamic configuration, function marking, coordination, leader election, etc. the simple HTTP API makes it easier to use
Multi data center Consul supports multi data centers out of the box. This means that users don't have to worry about establishing additional abstraction layers to expand business to multiple regions
Consul is friendly to DevOps and application developers. It is a modern flexible infrastructure
Infrastructure
Consul is a distributed and highly available system. This section will contain some basics. We will ignore some details so that you can quickly understand how consul works. For more details, please refer to the in-depth architecture description
Each stage of providing services to consult runs a consult agent. It is not necessary to run the agent to discover services or set and obtain the data stored in key/value. This agent is responsible for health inspection of the node itself and the services on the node
The Agent interacts with one or more consul servers. Consul servers are used to store and copy data. The server elects a leader by itself. Although consul can run on one server, it is recommended to use 3 to 5 servers to avoid data loss in case of failure. It is recommended to configure a server cluster for each data center
Components that need to find other services in your infrastructure can query any Consul server or agent. The agent will automatically forward requests to the server
A consul server cluster runs in each data center. When a cross data center service discovery and configuration request is created, the local consul server forwards the request to the remote data center and returns the result
See the official website for more information Click go
Install Consul
Install consul, find the package suitable for your system, and download it. Consul is packaged into a 'Zip' file Go to download
Unpack the compressed package after downloading. Copy Consul to your PATH. In Unix system, ~ / bin and / usr/local/bin are the usual installation directories. You can choose according to whether you want to install for a single user or the whole system. In Windows system, you can install to% PATH%
Verify installation
After completing the installation, check whether the consumer installation is successful by opening a new terminal window Consumer you should see output similar to the following
[root@dhcp-10-201-102-248 ~]# consul
usage: consul [--version] [--help] <command> [<args>]
Available commands are:
agent Runs a Consul agent
configtest Validate config file
event Fire a new event
exec Executes a command on Consul nodes
force-leave Forces a member of the cluster to enter the "left" state
info Provides debugging information for operators
join Tell Consul agent to join cluster
keygen Generates a new encryption key
keyring Manages gossip layer encryption keys
kv Interact with the key-value store
leave Gracefully leaves the Consul cluster and shuts down
lock Execute a command holding a lock
maint Controls node or service maintenance mode
members Lists the members of a Consul cluster
monitor Stream logs from a Consul agent
operator Provides cluster-level tools for Consul operators
reload Triggers the agent to reload configuration files
rtt Estimates network round trip time between nodes
snapshot Saves, restores and inspects snapshots of Consul server state
version Prints the Consul version
watch Watch for changes in ConsulIf you get a consumer not be found error, your PATH may not be set correctly. Please go back and check whether your consumer installation PATH is included in your PATH
Run Agent
After Consul installation, the agent must be run. The agent can run in server or client mode. Each data center must have at least one server. It is recommended to have 3 or 5 servers in a cluster. Deploying a single server will inevitably cause data loss in case of failure
Other agents run in client mode. A client is a very lightweight process. It is used to register services, run health checks and forward queries to the server. The agent must run on each host in the cluster
To view the details of starting the data center, please see here.
Start Consul Server
consul agent -server -bootstrap-expect 3 -data-dir /tmp/consul -node=s1 -bind=10.201.102.198 -ui-dir ./consul_ui/ -rejoin -config-dir=/etc/consul.d/ -client 0.0.0.0
- 1
Run cosnul agent in server mode,
- -server : Define that the agent runs in server mode
- -bootstrap-expect : The number of server nodes expected to be provided in a data center. When this value is provided, the consumer will not boot the whole cluster until the specified number of sever s is reached. This flag cannot be shared with bootstrap
- -bind: this address is used for communication within the cluster. All nodes in the cluster must be reachable. The default is 0.0.0.0
- -Node: the name of the node in the cluster. It must be unique in a cluster. The default is the host name of the node
- -UI dir: provide the path to store web ui resources, which must be readable
- -rejoin: make the consumer ignore the previous departure and still try to join the cluster after starting again.
- -Config dir: configuration file directory, in which all files ending in. json will be loaded
- -client: the listening address of the consumer service. This address provides HTTP, DNS, RPC and other services. The default is 127.0.0.1, so it does not provide external services. If you want to provide external services, change it to 0.0.0.0
[root@dhcp-10-201-102-198 consul]# consul agent -server -bootstrap-expect 1 -data-dir /tmp/consul -node=s1 -bind=10.201.102.198 -ui-dir ./consul_ui/ -rejoin -config-dir=/etc/consul.d/ -client 0.0.0.0
==> WARNING: Expect Mode enabled, expecting 3 servers
==> Starting Consul agent...
==> Starting Consul agent RPC...
==> Consul agent running!
Version: 'v0.7.4'
Node ID: '422ec677-74ef-8f29-2f22-01effeed6334'
Node name: 's1'
Datacenter: 'dc1'
Server: true (bootstrap: false)
Client Addr: 0.0.0.0 (HTTP: 8500, HTTPS: -1, DNS: 8600, RPC: 8400)
Cluster Addr: 10.201.102.198 (LAN: 8301, WAN: 8302)
Gossip encrypt: false, RPC-TLS: false, TLS-Incoming: false
Atlas: <disabled>
==> Log data will now stream in as it occurs:
2017/03/17 18:03:08 [INFO] raft: Restored from snapshot 139-352267-1489707086023
2017/03/17 18:03:08 [INFO] raft: Initial configuration (index=6982): [{Suffrage:Voter ID:10.201.102.199:8300 Address:10.201.102.199:8300} {Suffrage:Voter ID:10.201.102.200:8300 Address:10.201.102.200:8300} {Suffrage:Voter ID:10.201.102.198:8300 Address:10.201.102.198:8300}]
2017/03/17 18:03:08 [INFO] raft: Node at 10.201.102.198:8300 [Follower] entering Follower state (Leader: "")
2017/03/17 18:03:08 [INFO] serf: EventMemberJoin: s1 10.201.102.198
2017/03/17 18:03:08 [INFO] serf: Attempting re-join to previously known node: s2: 10.201.102.199:8301
2017/03/17 18:03:08 [INFO] consul: Adding LAN server s1 (Addr: tcp/10.201.102.198:8300) (DC: dc1)
2017/03/17 18:03:08 [INFO] consul: Raft data found, disabling bootstrap mode
2017/03/17 18:03:08 [INFO] serf: EventMemberJoin: s2 10.201.102.199
2017/03/17 18:03:08 [INFO] serf: EventMemberJoin: s3 10.201.102.200
2017/03/17 18:03:08 [INFO] serf: Re-joined to previously known node: s2: 10.201.102.199:8301
2017/03/17 18:03:08 [INFO] consul: Adding LAN server s2 (Addr: tcp/10.201.102.199:8300) (DC: dc1)
2017/03/17 18:03:08 [INFO] consul: Adding LAN server s3 (Addr: tcp/10.201.102.200:8300) (DC: dc1)
2017/03/17 18:03:08 [INFO] serf: EventMemberJoin: s1.dc1 10.201.102.198
2017/03/17 18:03:08 [INFO] consul: Adding WAN server s1.dc1 (Addr: tcp/10.201.102.198:8300) (DC: dc1)
2017/03/17 18:03:08 [WARN] serf: Failed to re-join any previously known node
2017/03/17 18:03:14 [INFO] agent: Synced service 'consul'
2017/03/17 18:03:14 [INFO] agent: Deregistered service 'consul01'
2017/03/17 18:03:14 [INFO] agent: Deregistered service 'consul02'
2017/03/17 18:03:14 [INFO] agent: Deregistered service 'consul03'View cluster members
Open a new terminal window and run consumer members. You can see the members of the Consul cluster
[root@dhcp-10-201-102-198 ~]# consul members Node Address Status Type Build Protocol DC s1 10.201.102.198:8301 alive server 0.7.4 2 dc1 s2 10.201.102.199:8301 alive server 0.7.4 2 dc1 s3 10.201.102.200:8301 alive server 0.7.4 2 dc1
Start Consul Client
consul agent -data-dir /tmp/consul -node=c1 -bind=10.201.102.248 -config-dir=/etc/consul.d/ -join 10.201.102.198
Run cosnul agent in client mode, - join Join the existing cluster.
[root@dhcp-10-201-102-248 ~]# consul agent -data-dir /tmp/consul -node=c1 -bind=10.201.102.248 -config-dir=/etc/consul.d/ -join 10.201.102.198
==> Starting Consul agent...
==> Starting Consul agent RPC...
==> Joining cluster...
Join completed. Synced with 1 initial agents
==> Consul agent running!
Version: 'v0.7.4'
Node ID: '564dc0c7-7f4f-7402-a301-cebe7f024294'
Node name: 'c1'
Datacenter: 'dc1'
Server: false (bootstrap: false)
Client Addr: 127.0.0.1 (HTTP: 8500, HTTPS: -1, DNS: 8600, RPC: 8400)
Cluster Addr: 10.201.102.248 (LAN: 8301, WAN: 8302)
Gossip encrypt: false, RPC-TLS: false, TLS-Incoming: false
Atlas: <disabled>
==> Log data will now stream in as it occurs:
2017/03/17 15:35:16 [INFO] serf: EventMemberJoin: c1 10.201.102.248
2017/03/17 15:35:16 [INFO] agent: (LAN) joining: [10.201.102.198]
2017/03/17 15:35:16 [INFO] serf: EventMemberJoin: s2 10.201.102.199
2017/03/17 15:35:16 [INFO] serf: EventMemberJoin: s3 10.201.102.200
2017/03/17 15:35:16 [INFO] serf: EventMemberJoin: s1 10.201.102.198
2017/03/17 15:35:16 [INFO] agent: (LAN) joined: 1 Err: <nil>
2017/03/17 15:35:16 [INFO] consul: adding server s2 (Addr: tcp/10.201.102.199:8300) (DC: dc1)
2017/03/17 15:35:16 [INFO] consul: adding server s3 (Addr: tcp/10.201.102.200:8300) (DC: dc1)
2017/03/17 15:35:16 [INFO] consul: adding server s1 (Addr: tcp/10.201.102.198:8300) (DC: dc1)
2017/03/17 15:35:16 [INFO] agent: Synced node infoView cluster members
Open a new terminal window and run consumer members. You can see the members of the Consul cluster
[root@dhcp-10-201-102-248 ~]# consul members Node Address Status Type Build Protocol DC c1 10.201.102.248:8301 alive client 0.7.4 2 dc1 s1 10.201.102.198:8301 alive server 0.7.4 2 dc1 s2 10.201.102.199:8301 alive server 0.7.4 2 dc1 s3 10.201.102.200:8301 alive server 0.7.4 2 dc1
Join cluster
[root@dhcp-10-201-102-248 ~]# consul join 10.201.102.198 Node Address Status Type Build Protocol DC c1 10.201.102.248:8301 alive client 0.7.4 2 dc1 s1 10.201.102.198:8301 alive server 0.7.4 2 dc1 s2 10.201.102.199:8301 alive server 0.7.4 2 dc1 s3 10.201.102.200:8301 alive server 0.7.4 2 dc1
Stop Agent
You can use Ctrl-C Close the Agent gracefully. After interrupting the Agent, you can see that it leaves the cluster and shuts down
During the exit, consul reminds other cluster members that the node has left. If you forcibly kill the process, other cluster members should be able to detect that the node has failed. When a member leaves, his services and detection will also be removed from the directory. When a member fails, his health status is simply marked as dangerous, However, it will not be removed from the directory. Consul will automatically attempt to reconnect the failed node. It is allowed to recover from some network conditions. The leaving node will not continue to contact
In addition, if an agent acts as a server, an elegant departure is very important to avoid the impact of potential availability failures Consistency protocol.
see here Learn about adding and removing server s
Update service
The service definition can be updated through the configuration file and sending SIGHUP to the agent. In this way, you can update the service without closing the service or keeping the service request available
consul reload
In addition, HTTP API can be used to dynamically add, remove and modify services
Registration service
After the conusl cluster is set up, users or programs can query or register services in conusl. You can register a service by providing a service definition file or calling HTTP API
First, create a directory for consul configuration. Consul will load all configuration files in the configuration folder. It is usually similar in Unix systems / etc/consul.d (. D suffix means that this path contains a set of configuration files)
mkdir /etc/consul.d
- 1
Then, we will write the service definition configuration file. Suppose we have a service named web running on port 80. In addition, we will set a label for it. In this way, we can use it as an additional query method:
echo '{"service": {"name": "web", "tags": ["rails"], "port": 80}}' >/etc/consul.d/web.jsonNow restart the agent and set the configuration directory:
$ consul agent -server -bootstrap-expect 1 -data-dir /tmp/consul -node=s1 -bind=10.201.102.198 -rejoin -config-dir=/etc/consul.d/ -client 0.0.0.0
...
[INFO] agent: Synced service 'web'
...- -Data dir: a directory is provided to store the status of agents. All agents need this directory. This directory must be stable and continue to exist after the system is restarted
You may notice that the "synced" web service is output, which means that the agent loads the service definition from the configuration file and successfully registers in the service directory
If you want to register multiple services, you should create multiple service definition files in the consult configuration directory
HTTP API registration service, curl command or postman requests consumption HTTP API in PUT mode. More details Click to view
curl -X PUT -d '{"Datacenter": "dc1", "Node": "c2", "Address": "10.155.0.106", "Service": {"Service": "MAC", "tags": ["lianglian", "Mac"], "Port": 22}}' http://127.0.0.1:8500/v1/catalog/registerQuery service
Once the agent is started and the service is synchronized, we can query the service through DNS or HTTP API
DNS API
Let's first use the DNS API to query. In the DNS API, the DNS NAME of the service is NAME.service.consumer. Although it is configurable, all DNS names will be in the consumer namespace by default. This subdomain tells Consul that we are querying the service, and NAME is the NAME of the service
For the Web service registered above, its domain name is web.service.consul :
[root@dhcp-10-201-102-198 ~]# dig @127.0.0.1 -p 8600 web.service.consul ; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.17.rc1.el6 <<>> @127.0.0.1 -p 8600 web.service.consul ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 39468 ;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0 ;; WARNING: recursion requested but not available ;; QUESTION SECTION: ;web.service.consul. IN A ;; ANSWER SECTION: web.service.consul. 0 IN A 10.201.102.198 ;; Query time: 0 msec ;; SERVER: 127.0.0.1#8600(127.0.0.1) ;; WHEN: Tue Mar 28 16:10:24 2017 ;; MSG SIZE rcvd: 52 [root@dhcp-10-201-102-198 ~]#
As you can see, an a record returns the IP address of the node where an available service is located. A record can only be set as IP address. You can also use DNS API to receive SRV records containing address and port:
[root@dhcp-10-201-102-198 ~]# dig @127.0.0.1 -p 8600 web.service.consul SRV ; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.17.rc1.el6 <<>> @127.0.0.1 -p 8600 web.service.consul SRV ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 13331 ;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; WARNING: recursion requested but not available ;; QUESTION SECTION: ;web.service.consul. IN SRV ;; ANSWER SECTION: web.service.consul. 0 IN SRV 1 1 80 s1.node.dc1.consul. ;; ADDITIONAL SECTION: s1.node.dc1.consul. 0 IN A 10.201.102.198 ;; Query time: 0 msec ;; SERVER: 127.0.0.1#8600(127.0.0.1) ;; WHEN: Tue Mar 28 16:10:56 2017 ;; MSG SIZE rcvd: 84 [root@dhcp-10-201-102-198 ~]#
The SRV record tells us that the web service runs on node dhcp-10-201-102-198 Port 80. DNS additionally returned the A record of the node
Finally, we can also use DNS API to filter services through tags. The tag based service query format is tag.name.service.consumer. In the following example, we request Consul to return Of rails Tags web service. We successfully obtained the service registered as this tag:
[root@dhcp-10-201-102-198 ~]# dig @127.0.0.1 -p 8600 rails.web.service.consul SRV ; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.17.rc1.el6 <<>> @127.0.0.1 -p 8600 rails.web.service.consul SRV ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37307 ;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; WARNING: recursion requested but not available ;; QUESTION SECTION: ;rails.web.service.consul. IN SRV ;; ANSWER SECTION: rails.web.service.consul. 0 IN SRV 1 1 80 s1.node.dc1.consul. ;; ADDITIONAL SECTION: s1.node.dc1.consul. 0 IN A 10.201.102.198 ;; Query time: 0 msec ;; SERVER: 127.0.0.1#8600(127.0.0.1) ;; WHEN: Tue Mar 28 16:11:45 2017 ;; MSG SIZE rcvd: 90 [root@dhcp-10-201-102-198 ~]#
HTTP API
In addition to DNS API, HTTP API can also be used for service query:
[root@dhcp-10-201-102-198 ~]# curl -s 127.0.0.1:8500/v1/catalog/service/web | python -m json.tool
[
{
"Address": "10.201.102.198",
"CreateIndex": 492843,
"ID": "422ec677-74ef-8f29-2f22-01effeed6334",
"ModifyIndex": 492843,
"Node": "s1",
"NodeMeta": {},
"ServiceAddress": "",
"ServiceEnableTagOverride": false,
"ServiceID": "web",
"ServiceName": "web",
"ServicePort": 80,
"ServiceTags": [
"rails"
],
"TaggedAddresses": {
"lan": "10.201.102.198",
"wan": "10.201.102.198"
}
}
]The directory API gives the services provided by all nodes. Later, we will query with health check as usual. As handled internally by DNS, this is a query method that only views healthy instances:
[root@dhcp-10-201-102-198 ~]# curl -s 127.0.0.1:8500/v1/catalog/service/web?passing | python -m json.tool
[
{
"Address": "10.201.102.198",
"CreateIndex": 492843,
"ID": "422ec677-74ef-8f29-2f22-01effeed6334",
"ModifyIndex": 492843,
"Node": "s1",
"NodeMeta": {},
"ServiceAddress": "",
"ServiceEnableTagOverride": false,
"ServiceID": "web",
"ServiceName": "web",
"ServicePort": 80,
"ServiceTags": [
"rails"
],
"TaggedAddresses": {
"lan": "10.201.102.198",
"wan": "10.201.102.198"
}
}
]WEB management interface
Consul also provides a beautiful and fully functional WEB interface out of the box. The interface can be used to view all nodes, health check and their current status. It can read and set K/V stored data. The UI automatically supports multiple data centers Click to download
Upload to the server after downloading. It is recommended that all server roles use WebUI,.
consul agent -server -bootstrap-expect 1 -data-dir /tmp/consul -node=s1 -bind=10.201.102.198 -ui-dir ./consul_ui/ -rejoin -config-dir=/etc/consul.d/ -client 0.0.0.0
- -UI dir: provides the path to store web ui resources. The directory must be readable
- -client: the listening address of the consumer service. This address provides HTTP, DNS, RPC and other services. The default is 127.0.0.1, so it does not provide external services. If you want to provide external services, change it to 0.0.0.0
Can pass http://10.201.102.198:8500 Access the WEB management interface.
health examination
We now see that the Consul runtime is so simple. Add nodes and services, query nodes and services. In this section, we will continue to add health checks to nodes and services. Health checks are a key component of service discovery. Prevent the use of unhealthy services
This step is based on the Consul cluster creation in the previous section. At present, you should have a Consul cluster with multiple nodes
Custom check
Similar to service registration, a check can be registered by checking the definition or HTTP API request
We will use and check definitions to register checks. Similar to services, this is the most common way to establish checks
Create a definition file in the configuration directory of the second node:
/etc/consul.d/web.json
- 1
{"service": {
"name": "Faceid",
"tags": ["extract", "verify", "compare", "idcard"],
"address": "10.201.102.198",
"port": 9000,
"check": {
"name": "ping",
"script": "curl -s localhost:9000",
"interval": "3s"
}
}
}or
/etc/consul.d/web.json
- 1
{"service": {
"name": "Faceid",
"tags": ["extract", "verify", "compare", "idcard"],
"address": "10.201.102.199",
"port": 9000,
"check": {
"id": "api",
"name": "HTTP API on port 9000",
"http": "http://localhost:9000",
"interval": "10s",
"timeout": "1s"
}
}
}more
Check health status
We can adapt to the HTTP API to check them. First, we check the failed checks. Use this command (Note: this command can run on any node)
[root@dhcp-10-201-102-198 ~]# curl -s http://localhost:8500/v1/health/state/critical | python -m json.tool
[
{
"CheckID": "service:Faceid",
"CreateIndex": 493398,
"ModifyIndex": 493846,
"Name": "Service 'Faceid' check",
"Node": "s1",
"Notes": "",
"Output": "",
"ServiceID": "Faceid",
"ServiceName": "Faceid",
"Status": "critical"
}
]We can see that there is only one check that our web service is in critical state
In addition, we can try to query the web service with DNS, and Consul will not return results because the service is unhealthy
[root@dhcp-10-201-102-198 ~]# dig @127.0.0.1 -p 8600 Faceid.service.consul SRV ; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.17.rc1.el6 <<>> @127.0.0.1 -p 8600 Faceid.service.consul SRV ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40884 ;; flags: qr aa rd; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 3 ;; WARNING: recursion requested but not available ;; QUESTION SECTION: ;Faceid.service.consul. IN SRV ;; ANSWER SECTION: Faceid.service.consul. 0 IN SRV 1 1 9000 s3.node.dc1.consul. Faceid.service.consul. 0 IN SRV 1 1 9000 s1.node.dc1.consul. Faceid.service.consul. 0 IN SRV 1 1 9000 s2.node.dc1.consul. ;; ADDITIONAL SECTION: s3.node.dc1.consul. 0 IN A 10.201.102.200 s1.node.dc1.consul. 0 IN A 10.201.102.198 s2.node.dc1.consul. 0 IN A 10.201.102.199 ;; Query time: 0 msec ;; SERVER: 127.0.0.1#8600(127.0.0.1) ;; WHEN: Tue Mar 28 18:20:15 2017 ;; MSG SIZE rcvd: 165
K ／V
In addition to providing the integration of service discovery and health check, Consul provides an easy-to-use key / value storage, which can be used to maintain dynamic configuration, assist in service coordination, elect leaders, and do anything developers can think of
[root@dhcp-10-201-102-198 ~]# curl -v http://localhost:8500/v1/kv/?recurse * About to connect() to localhost port 8500 (#0) * Trying ::1... connection denied * Trying 127.0.0.1... connected * Connected to localhost (127.0.0.1) port 8500 (#0) > GET /v1/kv/?recurse HTTP/1.1 > User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.21 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2 > Host: localhost:8500 > Accept: */* > < HTTP/1.1 404 Not Found < X-Consul-Index: 1 < X-Consul-Knownleader: true < X-Consul-Lastcontact: 0 < Date: Thu, 18 Aug 2016 08:21:39 GMT < Content-Length: 0 < Content-Type: text/plain; charset=utf-8 < * Connection #0 to host localhost left intact * Closing connection #0
Because there is no Key, we get a 404 response. Now let's PUT some examples of keys:
[root@dhcp-10-201-102-198 ~]# curl -X PUT -d 'test' http://localhost:8500/v1/kv/web/key1 [root@dhcp-10-201-102-198 ~]# curl -X PUT -d 'test' http://localhost:8500/v1/kv/web/key2?flags=42 [root@dhcp-10-201-102-198 ~]# curl -X PUT -d 'test' http://localhost:8500/v1/kv/web/sub/key3
We created three keys with the value "test". Note that the returned value is base64 encoded. It is used to support non UTF8 encoded characters. For Key web/key2 we set a flag value of 42. All keys support the setting of a 64 bit integer numeric flag. This value is not applicable inside Consul. However, it can be applied by the client to do some metadata
After setting, we initiate a GET request to receive the values of multiple key s, using? The recurse parameter
[root@dhcp-10-201-102-198 ~]# curl -s http://localhost:8500/v1/kv/web/?recurse | python -m json.tool
[
{
"CreateIndex": 502660,
"Flags": 0,
"Key": "web/key1",
"LockIndex": 0,
"ModifyIndex": 502660,
"Value": "dGVzdA=="
},
{
"CreateIndex": 502663,
"Flags": 42,
"Key": "web/key2",
"LockIndex": 0,
"ModifyIndex": 502663,
"Value": "dGVzdA=="
},
{
"CreateIndex": 502665,
"Flags": 0,
"Key": "web/sub/key3",
"LockIndex": 0,
"ModifyIndex": 502665,
"Value": "dGVzdA=="
}
]You can get a single key
[root@dhcp-10-201-102-198 ~]# curl -s http://localhost:8500/v1/kv/web/key1 | python -m json.tool
[
{
"CreateIndex": 502660,
"Flags": 0,
"Key": "web/key1",
"LockIndex": 0,
"ModifyIndex": 502660,
"Value": "dGVzdA=="
}
]Deleting a key is also very simple. It is done through the DELETE action. We can DELETE an individual key by specifying the full path. Or we can use? recurse recursively deletes all keys under the main path
[root@dhcp-10-201-102-198 ~]# curl -X DELETE http://localhost:8500/v1/kv/web/sub?recurse true
You can modify a Key by sending a PUT request with the same URL and different message bodies. In addition, consult provides an operation of checking and setting to realize atomic Key modification? cas = parameter plus the latest ModifyIndex in GET. For example, we want to modify "web/key1":
curl -X PUT -d 'newval' http://localhost:8500/v1/kv/web/key1?cas=502660 true curl -X PUT -d 'newval' http://localhost:8500/v1/kv/web/key1?cas=502660 false
In this case, the first CAS The update succeeded because ModifyIndex is 502660. The second failure is because ModifyIndex is no longer 502660 after the first update
We can also use ModifyIndex to wait for the key value to change. For example, we want to wait for key2 to be modified:
[root@dhcp-10-201-102-198 ~]# curl "http://localhost:8500/v1/kv/web/key2"
[{"LockIndex":0,"Key":"web/key2","Flags":42,"Value":"dGVzdA==","CreateIndex":502663,"ModifyIndex":502663}]
[root@dhcp-10-201-102-198 ~]# curl "http://localhost:8500/v1/kv/web/key2?index=502663&wait=5s"
[{"LockIndex":0,"Key":"web/key2","Flags":42,"Value":"dGVzdA==","CreateIndex":502663,"ModifyIndex":502663}]By providing ? index =, we request to wait for the key value to have a ModifyIndex greater than 502663. Although? The wait=5s parameter limits the request to 5 seconds at most, otherwise the current unchanged value will be returned. This can effectively wait for the change of keys. In addition, this function can be used to wait for a group of keys until one of them is modified
Conusl command line
I've seen the power of consumer. Consumer can be controlled through a simple CLI. Consumer has only one command line application, which is the consumer command. The consumer command can contain parameters such as agent and members for use. In this article, let's take a look at the specific usage of consumer CLI. Consumer - h can see the parameters supported by consumer CLI, and other parameters are supported in each parameter, Let's take a specific look.
[root@dhcp-10-201-102-198 ~]# consul
usage: consul [--version] [--help] <command> [<args>]
Available commands are:
agent Runs a Consul agent Run a consul agent
configtest Validate config file
event Fire a new event
exec Executes a command on Consul nodes stay consul Execute a command on the node
force-leave Forces a member of the cluster to enter the "left" state Force the state transition of node members in the cluster to left state
info Provides debugging information for operators Provides the of the operation debug Level information
join Tell Consul agent to join cluster join consul Node to cluster
keygen Generates a new encryption key Generate a new encryption key
keyring Manages gossip layer encryption keys
kv Interact with the key-value store
leave Gracefully leaves the Consul cluster and shuts down
lock Execute a command holding a lock
maint Controls node or service maintenance mode
members Lists the members of a Consul cluster List members in the cluster
monitor Stream logs from a Consul agent Print consul Log information of node
operator Provides cluster-level tools for Consul operators
reload Triggers the agent to reload configuration files Trigger node reload profile
rtt Estimates network round trip time between nodes
snapshot Saves, restores and inspects snapshots of Consul server state
version Prints the Consul version Print consul Version information for
watch Watch for changes in Consul monitor consul Change ofSee for more details Official website
Agent
Agent instruction is the core of consumer. It runs agent to maintain important information of members, run inspection, service announcement, query processing and so on.
==> Usage: consul agent [options]
Starts the Consul agent and runs until an interrupt is received. The
agent represents a single node in a cluster.
Options:
-advertise=addr Sets the advertise address to use
-advertise-wan=addr Sets address to advertise on wan instead of
advertise addr
-bootstrap Sets server to bootstrap mode
-bind=0.0.0.0 Sets the bind address for cluster
communication
-http-port=8500 Sets the HTTP API port to listen on
-bootstrap-expect=0 Sets server to expect bootstrap mode.
-client=127.0.0.1 Sets the address to bind for client access.
This includes RPC, DNS, HTTP and HTTPS (if
configured)
-config-file=foo Path to a JSON file to read configuration
from. This can be specified multiple times.
-config-dir=foo Path to a directory to read configuration
files from. This will read every file ending
in ".json" as configuration in this
directory in alphabetical order. This can be
specified multiple times.
-data-dir=path Path to a data directory to store agent
state
-dev Starts the agent in development mode.
-recursor=1.2.3.4 Address of an upstream DNS server.
Can be specified multiple times.
-dc=east-aws Datacenter of the agent (deprecated: use
'datacenter' instead).
-datacenter=east-aws Datacenter of the agent.
-encrypt=key Provides the gossip encryption key
-join=1.2.3.4 Address of an agent to join at start time.
Can be specified multiple times.
-join-wan=1.2.3.4 Address of an agent to join -wan at start
time. Can be specified multiple times.
-retry-join=1.2.3.4 Address of an agent to join at start time
with retries enabled. Can be specified
multiple times.
-retry-interval=30s Time to wait between join attempts.
-retry-max=0 Maximum number of join attempts. Defaults to
0, which will retry indefinitely.
-retry-join-ec2-region EC2 Region to use for discovering servers to
join.
-retry-join-ec2-tag-key EC2 tag key to filter on for server
discovery
-retry-join-ec2-tag-value EC2 tag value to filter on for server
discovery
-retry-join-gce-project-name Google Compute Engine project to discover
servers in
-retry-join-gce-zone-pattern Google Compute Engine region or zone to
discover servers in (regex pattern)
-retry-join-gce-tag-value Google Compute Engine tag value to filter
for server discovery
-retry-join-gce-credentials-file Path to credentials JSON file to use with
Google Compute Engine
-retry-join-wan=1.2.3.4 Address of an agent to join -wan at start
time with retries enabled. Can be specified
multiple times.
-retry-interval-wan=30s Time to wait between join -wan attempts.
-retry-max-wan=0 Maximum number of join -wan attempts.
Defaults to 0, which will retry
indefinitely.
-log-level=info Log level of the agent.
-node=hostname Name of this node. Must be unique in the
cluster
-node-meta=key:value An arbitrary metadata key/value pair for
this node.
This can be specified multiple times.
-protocol=N Sets the protocol version. Defaults to
latest.
-rejoin Ignores a previous leave and attempts to
rejoin the cluster.
-server Switches agent to server mode.
-syslog Enables logging to syslog
-ui Enables the built-in static web UI server
-ui-dir=path Path to directory containing the Web UI
resources
-pid-file=path Path to file to store agent PIDevent
The event command provides a mechanism to fire customized user events that are opaque to consumer, but they can be used to build scripts for automatic deployment, service restart, or other actions.
-http-addr: http Address of the service, agent You can link to send commands. If it is not set, the default is 127.0.0.1:8500. -datacenter: Data center. -name: Name of the event -node: A regular expression used to filter nodes -service: A regular expression used to filter matching services on nodes -tag: A regular expression used to filter matches on nodes tag Your service must be and-service Use together.
exec
The exec instruction provides a remote execution mechanism. For example, you need to execute the uptime command on all machines. The remotely executed work is specified by job and stored in KV. The agent can quickly know that a new job is generated by using the event system. The message is transmitted through the gossip protocol. Therefore, message transmission is the best, but it does not guarantee the execution of the command. Events are driven by gossip, and remote execution depends on KV storage system (just like message broker).
-http-addr: http Address of the service, agent You can link to send commands. If it is not set, the default is 127.0.0.1:8500. -datacenter: Data center. -prefix: key stay KV Prefix in the system, used to store request data. The default is_rexec -node: A regular expression used to filter nodes and evaluate events -service: A regular expression used to filter matching services on nodes -tag: A regular expression used to filter matches on nodes tag Your service must be and-service Use together. -wait: After the node has not responded for a long time, it is considered that job Already completed. -wait-repl: -verbose: Output more information
force-leave
Force leave treatment can force members in the consumer cluster to enter the left state (idle state). Remember, even if a member is active, it can still join the cluster again. The real purpose of this method is to forcibly remove the failed node. If the failed node is still part of the network, consumer will periodically relink the failed node. If after a period of time (72 hours by default), consumer will announce to stop trying to link the failed node. The force leave instruction can quickly convert the failed node to the left state.
-rpc-addr:One rpc Address, agent You can link to send commands. If not specified, the default is 127.0.0.1:8400.
info
Info instruction provides debug information that can be used in various operations. For client and server, info returns different subsystem information. At present, there are several KV information: agent (providing agent information), consumer (providing consumer library information), raft (providing raft library information), and serf_ LAN (provide LAN mission pool), serf_ Wan (provide Wan mission pool)
-rpc-addr: One rpc Address, agent You can link to send commands. If not specified, the default is 127.0.0.1:8400
- 1
join
The join instruction tells the consumer agent to join an existing cluster. A new consumer agent must join a cluster with at least one member so that it can join the existing cluster. If you do not join an existing cluster, the agent is part of its own cluster, and other agents can join. Agents can join other agents multiple times. consul join [options] address. If you want to join multiple clusters, you can write multiple addresses, and consumer will join all addresses.
-wan: agent Run in server pattern, xxxxxxx -rpc-addr: One rpc Address, agent You can link to send commands. If not specified, the default is 127.0.0.1:8400.
keygen
The keygen instruction generates an encrypted key, which can be used for consumer agent communication encryption
Generate a key
leave
The leave instruction triggers an elegant leave action and closes the agent. After the node leaves, it will not try to rejoin the cluster. Nodes running in the server state will be elegantly deleted, which is very serious. In some cases, an inelegant departure will affect the availability of the cluster.
-rpc-addr:One rpc Address, agent You can link to send commands. If not specified, the default is 127.0.0.1:8400.
members
The members instruction outputs all the members known to the consumer agent and their states. The node states are only alive, left and failed.
-detailed: Output more detailed information for each node. -rpc-addr: One rpc Address, agent You can link to send commands. If not specified, the default is 127.0.0.1:8400. -status: Filter out nodes that conform to regular rules -wan: xxxxxx
monitor
The monitor command is used to link running agents and display logs. The monitor will display the latest log and continuously display the log flow. It will not exit automatically unless you exit manually or by the remote agent itself.
-log-level: Which level of log is displayed? The default is info -rpc-addr: One rpc Address, agent You can link to send commands. If not specified, the default is 127.0.0.1:8400
reload
The reload instruction can reload the configuration file of the agent. The SIGHUP instruction is used when reloading the configuration file. Any reloading error will be written in the agent's log file and will not be printed to the screen.
-rpc-addr: One rpc Address, agent You can link to send commands. If not specified, the default is 127.0.0.1:8400
version
Print the version of the consumer
watch
The watch instruction provides a mechanism to monitor the changes of the actual data view (node list, member service, KV). If no process is specified, the current value will be dump ed
-http-addr: http Address of the service, agent You can link to send commands. If it is not set, the default is 127.0.0.1:8500. -datacenter: Data center query. -token: ACL token -key: monitor key，Only for key type -name: monitor event，Only for event type -prefix: monitor key prefix，Only for keyprefix type -service: monitor service，Only for service type -state: Guo Lue check state -tag: filter service tag -type: Monitoring types, generally key,keyprefix,service,nodes,checks,event
Consul configuration
agent has various configuration items that can be defined on the command line or configuration file. All configuration items are optional. When loading the configuration file, consumer loads the configuration from the configuration file or configuration directory. The configuration defined later will merge the configuration defined earlier, but in most cases, merging means that the configuration defined later will overwrite the configuration defined earlier. However, in some cases, such as event handle, merging is only added to the handle defined earlier. The consumer reload configuration file also supports receiving the update signal in the form of a signal.
Let's look at the command line parameters:
-advertise: The notification presentation address is used to change the addresses we present to other nodes in the cluster. Generally-bind The address is the display address -bootstrap: Used to control a server Is it bootstrap Mode, in a datacenter Only one of the server be in bootstrap Mode, when a server be in bootstrap Mode, you can elect to raft leader. -bootstrap-expect: In a datacenter Hope to provide server Number of nodes. When this value is provided, consul Wait until the specified time is reached sever The whole cluster will be booted only when the number is. This flag cannot be associated with bootstrap public -bind: This address is used for communication within the cluster. All nodes in the cluster must be reachable. The default is 0.0.0.0 -client: consul Where is the binding client On the address, this address provides HTTP,DNS,RPC And other services. The default is 127.0.0.1 -config-file: Explicitly specify which configuration file to load -config-dir: Configuration file directory, in which all.json The end of the file will be loaded -data-dir: Provide a directory to store agent Status of all agent This directory is required for all allowed. It must be stable and continue to exist after the system is restarted -dc: This flag controls agent Permissible datacenter The default is dc1 -encrypt: appoint secret key，send consul Encryption during communication, key Can pass consul keygen Generation, nodes in the same cluster must use the same key -join: Join a started agent of ip Address. You can specify multiple addresses multiple times agent Your address. If consul Cannot be added to any of the specified addresses, then agent Will fail to start, default agent No nodes will be joined at startup. -retry-join: and join Similar, but allows you to try after the first failure. -retry-interval: twice join The time interval between is 30 by default s -retry-max: Try to repeat join The default is 0, that is, infinite attempts -log-level: consul agent The level of log information displayed after startup. The default is info，Optional: trace,debug,info,warn,err. -node: The name of a node in a cluster must be unique in a cluster. The default is the host name of the node -protocol: consul Protocol version used -rejoin: send consul Ignore the previous departure and try to join the cluster after starting again. -server: definition agent Run in server Mode, each cluster has at least one server，Recommended for each cluster server No more than 5 -syslog: Enable the system log function, only in linux/osx Effective on -ui-dir:Provide storage web ui The path to the resource, which must be readable -pid-file:Provide a path to store pid File, which can be used for SIGINT/SIGHUP(close/to update)agent
In addition to the command line parameters, the configuration can also be written to the file. In some cases, the configuration file will be simpler. For example, using consumer is used to manage the system. The configuration file is in json format and is easy to write. The configuration file is not only used to set the startup of agent, but also can be used to provide the definition of health detection and service discovery. General examples of configuration files are as follows:
{
"datacenter": "dc1",
"data_dir": "/opt/consul",
"log_level": "INFO",
"node_name": "s1",
"server": true,
"bootstrap_expect": 3,
"bind_addr": "10.201.102.198",
"client_addr": "0.0.0.0",
"ui_dir": "/root/consul_ui",
"retry_join": ["10.201.102.198","10.201.102.199","10.201.102.200","10.201.102.248"],
"retry_interval": "30s",
"enable_debug": false,
"rejoin_after_leave": true,
"start_join": ["10.201.102.198","10.201.102.199","10.201.102.200","10.201.102.248"],
"enable_syslog": true,
"syslog_facility": "local5"
}Let's take a look at the detailed configuration file parameters:
acl_datacenter: Only for server，designated datacenter Authority of ACL Information, all servers and datacenter Must agree ACL datacenter acl_default_policy: The default is allow acl_down_policy: acl_master_token: acl_token: agent Will use this token and consul server Make a request acl_ttl: control TTL of cache，The default is 30 s addresses: A nested object, you can set the following key: dns,http,rpc advertise_addr: Equivalent to-advertise bootstrap: Equivalent to-bootstrap bootstrap_expect: Equivalent to-bootstrap-expect bind_addr: Equivalent to-bind ca_file: provide CA The file path is used to check the link of the client or server cert_file: Must and key_file Together check_update_interval: client_addr: Equivalent to-client datacenter: Equivalent to-dc data_dir: Equivalent to-data-dir disable_anonymous_signature: Anonymous signatures are prohibited during update checks disable_remote_exec: Remote execution is prohibited and set to true，agent All incoming remote execution requests are ignored disable_update_check: It is forbidden to automatically check the security bulletin and new version information dns_config: Is a nested object. You can set the following parameters: allow_stale,max_stale,node_ttl ,service_ttl,enable_truncate domain: By default consul In progress DNS When querying, the query is consul Field, which can be modified through this parameter enable_debug: open debug pattern enable_syslog: Equivalent to-syslog encrypt: Equivalent to-encrypt key_file: Provide the path of the private key leave_on_terminate: The default is false，If yes true，When agent Got one TERM When the signal is sent, it will send leave Information to other nodes in the cluster. log_level: Equivalent to-log-level node_name:Equivalent to-node ports: This is a nested object. You can set the following key: dns(dns Address: 8600),http(http api Address: 8500),rpc(rpc:8400),serf_lan(lan port:8301),serf_wan(wan port:8302),server(server rpc:8300) protocol: Equivalent to-protocol recursor: rejoin_after_leave: Equivalent to-rejoin retry_join: Equivalent to-retry-join retry_interval: Equivalent to-retry-interval server: Equivalent to-server server_name: Will cover TLS CA of node_name，Can be used to confirm CA name and hostname Match skip_leave_on_interrupt: and leave_on_terminate The comparison is similar, but only affects the current handle start_join: The node address provided by a character array will be added at startup statsd_addr: statsite_addr: syslog_facility: When enable_syslog After being provided, this parameter controls which level of information is sent. By default Local0 ui_dir: Equivalent to-ui-dir verify_incoming: default false，If yes true，Then all entry links need to be used TLS，Client required ca_file provide ca File, for consul server End, because client Never entered the link verify_outgoing: default false，If yes true，Then all outgoing links need to be used TLS，It needs to be used by the server ca_file provide ca Documents, consul server and client Both need to be used because both have outgoing links watches: watch A detailed list
HTTP API
The main interface of consumer is the RESTful HTTP API, which can be used to add, delete, query and modify nodes, services, checks and configuration. All endpoints are mainly divided into the following categories:
kv - Key/Value storage agent - Agent control catalog - Administration nodes and services health - Management health monitoring session - Session operation acl - ACL Create and manage event - user Events status - Consul system state
Let's look at the specific content of each module separately.
agent
agent endpoints are used to interact with local agents. They are generally used for service registration and check registration. The following interfaces are supported
/v1/agent/checks : Return to local agent All checks registered(Include configuration files and HTTP Interface) /v1/agent/services : Return to local agent All registered services /v1/agent/members : return agent In the cluster gossip pool Members seen in /v1/agent/self : Return to local agent Configuration and member information for /v1/agent/join/<address> : Trigger local agent join node /v1/agent/force-leave/<node>>: Force delete node /v1/agent/check/register : Locally agent Add a check item, use PUT Method to transfer a json Formatted data /v1/agent/check/deregister/<checkID> : Log off a local agent Check items for /v1/agent/check/pass/<checkID> : Set the status of a local check item to passing /v1/agent/check/warn/<checkID> : Set the status of a local check item to warning /v1/agent/check/fail/<checkID> : Set the status of a local check item to critical /v1/agent/service/register : Locally agent Add a new service item, use PUT Method to transfer a json Formatted data /v1/agent/service/deregister/<serviceID> : Log off a local agent Service items
catalog
catalog endpoints is used to register / unregister nodes, services, and checks
/v1/catalog/register : Registers a new node, service, or check /v1/catalog/deregister : Deregisters a node, service, or check /v1/catalog/datacenters : Lists known datacenters /v1/catalog/nodes : Lists nodes in a given DC /v1/catalog/services : Lists services in a given DC /v1/catalog/service/<service> : Lists the nodes in a given service /v1/catalog/node/<node> : Lists the services provided by a node
health
health endpoints is used to query health related information. This function is separated from the catalog
/v1/healt/node/<node>: return node Defined checks, available parameters?dc= /v1/health/checks/<service>: Returns the check associated with the service and the available parameters?dc= /v1/health/service/<service>: Return given datacenter Given in node in service /v1/health/state/<state>: Return given datacenter Services in the specified state, state Can be"any", "unknown", "passing", "warning", or "critical"，Available parameters?dc=
session
session endpoints are used to create, update, destroy, and query sessions
/v1/session/create: Creates a new session /v1/session/destroy/<session>: Destroys a given session /v1/session/info/<session>: Queries a given session /v1/session/node/<node>: Lists sessions belonging to a node /v1/session/list: Lists all the active sessions
acl
acl endpoints are used to create, update, destroy, and query ACLS
/v1/acl/create: Creates a new token with policy /v1/acl/update: Update the policy of a token /v1/acl/destroy/<id>: Destroys a given token /v1/acl/info/<id>: Queries the policy of a given token /v1/acl/clone/<id>: Creates a new token by cloning an existing token /v1/acl/list: Lists all the active tokens
event
event endpoints is used to fire new events and query existing events
/v1/event/fire/<name>: Trigger a new event，user event need name And other optional parameters, using PUT method /v1/event/list: return agent Yes events
status
status endpoints is used to or consume cluster information
/v1/status/leader : Returns the of the current cluster Raft leader /v1/status/peers : Return colleagues in the current cluster
Consul-Template
Before the advent of the consumer template, we built service discovery systems, mostly using similar systems such as zookeeper and etcd + conf. we wrote a consumer + conf article about how to dynamically generate configuration files. Now consumer has officially launched its own template system, consumer template, In this way, the dynamic configuration system can be divided into two camps: etcd + conf D and consumer + consumer template. Consumer is a system similar to etcd but stronger than etcd. You can read previous articles about etcd and consumer. The positioning of consumer template is almost the same as that of conf. the back end of conf can be etcd or consumer. I believe that consumer can play a greater role with consumer template. Consumer template provides a convenient way to obtain the stored values from consumer. The consumer template daemon will query the consumer instance to update any template specified on the system. After the update is completed, the template can choose to run some arbitrary commands.
consul template Usage scenarios: consul template Can query consul Service directory in key,key-values Wait. This powerful abstraction and query language template can make consul template It is especially suitable for dynamic creation of configuration files. For example: create apache/nginx proxy balancers,haproxy backends,varnish servers,application configurations.
Features of consumer template:
quiescence: consul template The built-in static balance function can be found intelligently consul Change information in the instance. This function can prevent frequent template updates from causing system fluctuations.
dry mode: Uncertain about the status of the current schema? Worried that the change of template will destroy the subsystem? Don't worry, because consul template also-dry pattern. stay dry pattern, consul template The results are presented in STDOUT，Therefore, the operator can check whether the output is normal to determine whether it is safe to replace the template
CLI and Config: If you like to specify everything on the command line, consul template Fine hold Stay. With built-in HCL With your support, consul template Receive a configuration file, command line parameters, or a mixture of the two. In this way, you can continue to use your existing configuration management tools and tools consul template To cooperate.
verbose debugging: Even if everything you do is almost perfect, sometimes there will be failures. consul template More detailed information can be provided debug Log information.install
You can Release page Download the release package. If you want to compile it yourself, please check it Documentation.
use
-auth=<user[:pass]> Set the basic authentication user name and password -consul-addr=<address> set up Consul Address of the instance -max-stale=<duration> The maximum frequency of query expiration. The default is 1 s -dedup Enable deduplication when many consul template Instance can be reduced when rendering a template consul Load of -ssl use https connect Consul use SSL -ssl-verify adopt SSL Check the certificate when connecting -ssl-cert SSL The client certificate is sent to the server -ssl-key Used for client authentication SSL/TLS Private key -ssl-ca-cert Verify the of the server CA Certificate list -token=<token> set up Consul API of token -syslog Redirect standard output and standard errors to syslog，syslog The default level for is local0. -syslog-facility=<f> set up syslog Level, default is local0，Must and-syslog Use together -template=<template> Add a template to be monitored in the following format:'templatePath:outputPath(:command)'，Multiple templates can be set multiple times -wait=<duration> The maximum and minimum time to wait when a new template is presented to the system and a command is triggered. If the maximum value is ignored, it defaults to 4 times the minimum value. -retry=<duration> When in and consul api The return value of the interaction is error The default waiting time is 5 s. -config=<path> Path to configuration file or configuration directory -pid-file=<path> PID Path to file -log-level=<level> Set the log level, which can be"debug","info", "warn" (default), and "err" -dry Dump The generated template is to standard output and will not be generated to disk -once function consul-template Exit once and do not run as a daemon -reap Sub process automatic harvesting
To view all options, use the following command
consul-template -h
command line
1. Query the local consl instance, generate the template, and restart nginx. If the consumer is unavailable and the api fails, try to detect the value every 30s. Run the consumer template once and exit
consul-template -retry 30s -once -consul-addr=10.201.102.198:8500 -template "test.ctmpl:test.out"
test.ctmpl
{{range service "Faceid"}}
{{.ID}} {{.Address}}:{{.Port}} check inter 5000 fall 1 rise 2 weight 2{{end}}test.out
Faceid 10.201.102.198:9000 check inter 5000 fall 1 rise 2 weight 2 Faceid 10.201.102.199:9000 check inter 5000 fall 1 rise 2 weight 2 Faceid 10.201.102.200:9000 check inter 5000 fall 1 rise 2 weight 2
2. Run consumer temperature as a service
consul-template -consul-addr=10.201.102.198:8500 -template "test.ctmpl:test.out"
3. Query an instance, render multiple templates, and restart related services
consul-template -retry 30s -once -consul-addr=10.201.102.198:8500 -template "test.ctmpl:test.out"\ -template "/tmp/redis.ctmpl:/var/redis/redis.conf:service redis restart" \ -template "/tmp/haproxy.ctmpl:/var/haproxy/haproxy.conf"
4. Query an instance, dump the template to the standard output, and the - template in the parameter will be ignored
consul-template -dry -consul-addr=10.201.102.198:8500 -template "test.ctmpl:test.out"
In addition to being used on the command line, the above parameters can also be directly configured in the file. Let's take a look at the configuration file of consul template, referred to as HCL(HashiCorp Configuration Language), which is compatible with JSON. Here's an example:
#### configuration file
```Consul-Template```Configuration files are used[HashiCorp Configuration Language (HCL)](https://github.com/hashicorp/hcl). This means that ` ` Consul Template ` ` ` is JSON compatible. For more information, please see [HCL Specification]( https://github.com/hashicorp/hcl)
The configuration file syntax supports all the above options,Unless otherwise indicated in the table.
```json
// This is the address of the Consul Agent to be connected. The default is 127.0.0.1:8500. This is the default binding address and port of Consul
// It is not recommended that you directly interact with the Consul Server. Please interact with the local Consul Agent. There are some reasons for this
// The most important thing is that the local agent can reuse the connection with the server. Reduce the number of HTTP connections. In addition, this address is better remembered
consul = "127.0.0.1:8500"
// This is the ACL token used to connect Consul. If your cluster is not enabled, it does not need to be set
//
// This option is also available through the environment variable consult_ Token to set
token = "abcd1234"
// This is the signal to listen for the starting reload event. The default value is as follows. Setting this value to blank will cause CT, so it will not listen for the reload event
reload_signal = "SIGHUP"
// This is the signal for monitoring the starting core dump event. The default value is as follows. Setting this value to null will cause CT, so the core dump signal will not be monitored
dump_signal = "SIGQUIT"
// This is the signal to listen for the starting grace stop event. The default value is as follows. Setting this value to null will cause CT, so it will not listen for the grace stop signal
kill_signal = "SIGINT"
// This is the retry time for connecting to consult. The consult template is a high error tolerance design. This means that he will not quit in case of failure. Instead, follow the
// The Convention of distributed systems is to perform exponential compensation and retry to wait for cluster recovery
retry = "10s"
// This is the maximum interval to allow "stale" data. By default, only the
// Consul leader will respond to queries; any requests to a follower will
// forward to the leader. In large clusters with many requests, this is not as
// scalable, so this option allows any follower to respond to a query, so long
// as the last-replicated data is within these bounds. Higher values result in
// less cluster load, but are more likely to have outdated data.
// This is the maximum time allowed for stale data. Consul defaults that only the leader responds to the request. All requests to followers will be forwarded to the leader
// In a large cluster with a large number of requests, this is not scalable enough, so this option allows any follower to respond to the query as long as the last copied data
// Within this range, the higher the value, the less the cluster load, but it is easier to accept expired data
max_stale = "10m"
// This is the log level. If you find a bug, please open the debug log so that we can better locate the problem. This option can also be used on the command line
log_level = "warn"
// This is the path where the PID file of Consul Template process is stored. It will be useful if you plan to send customized signals to this process
pid_file = "/path/to/pid"
// This is a static timer, which defines the minimum and maximum time to wait for the cluster to reach a consistent state before template rendering
// This is useful for some systems with large changes, which can reduce the number of template rendering
wait = "5s:10s"
// This is the beginning of the Vault configuration
// Vault is another product of HashiCorp
vault {
// This is the address of the Vault leader. The protocol (http(s)) portion
// of the address is required.
address = "https://vault.service.consul:8200"
// This is the token to use when communicating with the Vault server.
// Like other tools that integrate with Vault, Consul Template makes the
// assumption that you provide it with a Vault token; it does not have the
// incorporated logic to generate tokens via Vault's auth methods.
//
// This value can also be specified via the environment variable VAULT_TOKEN.
token = "abcd1234"
// This option tells Consul Template to automatically renew the Vault token
// given. If you are unfamiliar with Vault's architecture, Vault requires
// tokens be renewed at some regular interval or they will be revoked. Consul
// Template will automatically renew the token at half the lease duration of
// the token. The default value is true, but this option can be disabled if
// you want to renew the Vault token using an out-of-band process.
//
// Note that secrets specified in a template (using {{secret}} for example)
// are always renewed, even if this option is set to false. This option only
// applies to the top-level Vault token itself.
renew = true
// This section details the SSL options for connecting to the Vault server.
// Please see the SSL options below for more information (they are the same).
ssl {
// ...
}
}
// This section configures the basic permission verification information of the request
auth {
enabled = true
username = "test"
password = "test"
}
// This section configures the SSL information connected to the consult server
ssl {
// To use SSL, you need to turn on this switch first
enabled = true
// This enables SSL peer verification. The default value is "true", which
// will check the global CA chain to make sure the given certificates are
// valid. If you are using a self-signed certificate that you have not added
// to the CA chain, you may want to disable SSL verification. However, please
// understand this is a potential security vulnerability.
verify = false
// This is the path to the certificate to use to authenticate. If just a
// certificate is provided, it is assumed to contain both the certificate and
// the key to convert to an X509 certificate. If both the certificate and
// key are specified, Consul Template will automatically combine them into an
// X509 certificate for you.
cert = "/path/to/client/cert"
key = "/path/to/client/key"
// This is the path to the certificate authority to use as a CA. This is
// useful for self-signed certificates or for organizations using their own
// internal certificate authority.
ca_cert = "/path/to/ca"
}
// Set the configuration for connecting to the syslog server
// syslog for logging{
// Turn on the switch
enabled = true
// Equipment name
facility = "LOCAL5"
}
// This block defines the configuration for de-duplication mode. Please see the
// de-duplication mode documentation later in the README for more information
// on how de-duplication mode operates.
deduplicate {
// This enables de-duplication mode. Specifying any other options also enables
// de-duplication mode.
enabled = true
// This is the prefix to the path in Consul's KV store where de-duplication
// templates will be pre-rendered and stored.
prefix = "consul-template/dedup/"
}
// This block defines the configuration for exec mode. Please see the exec mode
// documentation at the bottom of this README for more information on how exec
// mode operates and the caveats of this mode.
exec {
// This is the command to exec as a child process. There can be only one
// command per Consul Template process.
command = "/usr/bin/app"
// This is a random splay to wait before killing the command. The default
// value is 0 (no wait), but large clusters should consider setting a splay
// value to prevent all child processes from reloading at the same time when
// data changes occur. When this value is set to non-zero, Consul Template
// will wait a random period of time up to the splay value before reloading
// or killing the child process. This can be used to prevent the thundering
// herd problem on applications that do not gracefully reload.
splay = "5s"
// This defines the signal that will be sent to the child process when a
// change occurs in a watched template. The signal will only be sent after
// the process is started, and the process will only be started after all
// dependent templates have been rendered at least once. The default value
// is "" (empty or nil), which tells Consul Template to restart the child
// process instead of sending it a signal. This is useful for legacy
// applications or applications that cannot properly reload their
// configuration without a full reload.
reload_signal = "SIGUSR1"
// This defines the signal sent to the child process when Consul Template is
// gracefully shutting down. The application should begin a graceful cleanup.
// If the application does not terminate before the `kill_timeout`, it will
// be terminated (effectively "kill -9"). The default value is "SIGTERM".
kill_signal = "SIGINT"
// This defines the amount of time to wait for the child process to gracefully
// terminate when Consul Template exits. After this specified time, the child
// process will be force-killed (effectively "kill -9"). The default value is
// "30s".
kill_timeout = "2s"
}
// This section defines the configuration of templates, which is different from other configuration blocks. This section can be configured multiple times for different templates. You can also use CLI commands
// Configure directly
template {
// This is the configuration file path of the input template. Required
source = "/path/on/disk/to/template.ctmpl"
// This is the path where the source template is stored after rendering. If the parent directory does not exist, the Consul Template will try to create it
destination = "/path/on/disk/where/template/will/render.txt"
// This is the optional command to run when the template is rendered. The
// command will only run if the resulting template changes. The command must
// return within 30s (configurable), and it must have a successful exit code.
// Consul Template is not a replacement for a process monitor or init system.
// This is an optional command to execute after the template rendering is completed. This command will only run after the template is changed. This command must be executed in 30 seconds
// A successful exit code must be returned (configurable). Consult template cannot replace process monitoring or init system
// Function of
command = "restart service foo"
// This is the maximum waiting time for the command to return. The default is 30 seconds
command_timeout = "60s"
// This is the permission of the rendered file. If it is not set, the Consul Template will match the permission of the previously existing file
// If the file does not exist, the permission is set to 0644
perms = 0600
// This option backs up the files before rendering. It keeps a backup
// This option has a rollback strategy in case of unexpected higher
backup = true
// The default separators for templates are "{{" and "}}". However, for some templates, it may be better to use other separators
// Can avoid conflict with itself
left_delimiter = "{{"
right_delimiter = "}}"
// This is the minimum and maximum waiting time to render a new template and execute commands. Use semicolons. If the maximum value is ignored, the maximum value is
// The value will be set to 4 times the minimum value. This option has no default value. This value has the highest priority relative to the global waiting time
wait = "2s:6s"
}Note: not all options are required. For example, if you don't use Vault, you don't need to set this one. Similarly, if you don't use syslog system, you don't need to specify syslog configuration
For more security, tokens can also be read from environment variables and used CONSUL_TOKEN and VAULT_TOKEN. It is strongly recommended that you do not put tokens in unencrypted text configuration files
Template syntax
Consul Template uses Go's template syntax. If you are not familiar with his syntax, you are recommended to read the document. His syntax looks similar to Mustache, Handlebars, or Liquid
In addition to the template functions provided by Go, consult template exposes the following functions:
API function
datacenters
Query all data centers in the directory. Use the following syntax:
{{datacenters}}file
Read and output the local file on the disk. If it cannot be read, an error will be generated. Use the following syntax
{{file "/path/to/local/file"}}This example will output / path/to/local/file File content to template. Note: this will not be processed in nested templates
key
Query the value of the key specified by Consul. If the value of the key cannot be converted to a string, an error will be generated. Use the following syntax:
{{key "service/redis/maxconns@east-aws"}}The above example queries the data in the East AWS data center Value of service/redis/maxconns. If the data center parameter is ignored, the value of the local data center will be queried:
{{key "service/redis/maxconns"}}The beauty of Consul key value structure is that it's entirely up to you!
key_or_default
Query the value of the key specified in consult. If the key does not exist, the default value is returned. The usage is as follows
{{key_or_default "service/redis/maxconns@east-aws" "5"}}Note that the Consul Template uses multiple stages of operations. In the first stage of operations, if Consul does not return a value, the default value will always be used. In subsequent template parsing, if the value exists, the real value will be read. This is very important. The O & M Consul temple will not block the rendering of the template because the key_or_default is not found. Even if the key exists, if Consul does not return this number on time According to, the default value will also be used for substitution
ls
View all key value pairs of Consul that begin with the specified prefix. If any value cannot be converted to a string, an error will be generated:
{{range ls "service/redis@east-aws"}}
{{.Key}} {{.Value}}{{end}}If the Consul instance has this structure service/redis in the East AWS data center, the rendered template should be similar to this:
minconns 2 maxconns 12
If you ignore the data center properties, the query results of the local data center will be returned
node
Query a node information in the directory
{{node "node1"}}If no parameters are specified, the node of the current agent will be returned:
{{node}}You can specify an optional parameter to specify the data center:
{{node "node1" "@east-aws"}}If the specified node is not found, nil will be returned. If the node exists, the information of the node and the services provided by the node will be listed
{{with node}}{{.Node.Node}} ({{.Node.Address}}){{range .Services}}
{{.Service}} {{.Port}} ({{.Tags | join ","}}){{end}}
{{end}}
nodes
Query all nodes in the directory using the following syntax
{{nodes}}This example will query the default data center of Consul. You can specify an optional parameter to specify the data center using an optional parameter:
{{nodes "@east-aws"}}This example will query all the points in the East AWS data center
secret
Query the key of the specified path in the Vault. If the specified path does not exist or the Vault Token does not have sufficient permissions to read the specified path, an error will be generated. If the path exists but the key does not exist, it will return "."
{{with secret "secret/passwords"}}{{.Data.password}}{{end}}You can use the following fields:
LeaseID - the unique lease identifier
LeaseDuration - the number of seconds the lease is valid
Renewable - if the secret is renewable
Data - the raw data - this is a map[string]interface{}, so it can be queried using Go's templating "dot notation"
If the map key has dots "." in it, you need to access the value using the index function:
{{index .Data "my.key.with.dots"}}
If additional arguments are passed to the function, then the operation is assumed to be a write operation instead of a read operation. The write operation must return data in order to be valid. This is especially useful for the PKI secret backend, for example.
{{ with secret "pki/issue/my-domain-dot-com" "common_name=foo.example.com" }}
{{ .Data.certificate }}
{{ end }}
The parameters must be key=value pairs, and each pair must be its own argument to the function:
{{ secret "path/" "a=b" "c=d" "e=f" }}
Please always consider the security implications of having the contents of a secret in plain-text on disk. If an attacker is able to get access to the file, they will have access to plain-text secrets.Please note that Vault does not support blocking queries. As a result, Consul Template will not immediately reload in the event a secret is changed as it does with Consul's key-value store. Consul Template will fetch a new secret at half the lease duration of the original secret. For example, most items in Vault's generic secret backend have a default 30 day lease. This means Consul Template will renew the secret every 15 days. As such, it is recommended that a smaller lease duration be used when generating the initial secret to force Consul Template to renew more often.
secrets
Query Vault to list the secrets at the given path. Please note this requires Vault 0.5+ and the endpoint you want to list secrets must support listing. Not all endpoints support listing. The result is the list of secret names as strings.
{{range secrets "secret/"}}{{.}}{{end}}The trailing slash is optional in the template, but the generated secret dependency will always have a trailing slash in log output.
To iterate and list over every secret in the generic secret backend in Vault, for example, you would need to do something like this:
{{range secrets "secret/"}}
{{with secret (printf "secret/%s" .)}}
{{range $k, $v := .Data}}
{{$k}}: {{$v}}
{{end}}
{{end}}
{{end}}
You should probably never do this. Please also note that Vault does not support blocking queries. To understand the implications, please read the note at the end of the secret function.
service
Query the service matching the expression in Consul. The syntax is as follows:
{{service "release.web@east-aws"}}- 1
The above example queries the health problems existing in the East AWS data center in Consul Web service. Tag and data center parameters are optional. Query the web services of all nodes from the current data center regardless of tag. The query syntax is as follows:
{{service "web"}}This function returns [] * HealthService structure. It can be applied to the template as follows:
{{range service "web@data center"}}
server {{.Name}} {{.Address}}:{{.Port}}{{end}}The following outputs are generated:
server nyc_web_01 123.456.789.10:8080 server nyc_web_02 456.789.101.213:8080
The default value will return healthy services. If you want to obtain all services, you can add the any option as follows:
{{service "web" "any"}}This will return all registered services regardless of their status
If you want to filter one or more health states specified, you can separate multiple health states by commas:
{{service "web" "passing, warning"}}This will return the services marked "passing" OR "warning" by their node AND service level inspection definition. Please note that the comma is OR rather than AND
If more than one state filter is specified and any is included, an error will be returned, because any is a more advanced filter than all States
There are some architectural differences between the latter two methods:
{{service "web"}}
{{service "web" "passing"}}The former will return all services that consul considers healthy and passing. The latter will return all services that have been registered with Consul. Then a client-side filtering will be performed. Generally, if you want to obtain healthy services, you should not use the passing parameter and just ignore the third parameter. However, the third parameter will be more useful when you want to query passing or warning services, as shown below :
{{service "web" "passing, warning"}}The status of the service is also visible. If you want to do some additional filtering yourself, the syntax is as follows:
{{range service "web" "any"}}
{{if eq .Status "critical"}}
// Critical state!{{end}}
{{if eq .Status "passing"}}
// Ok{{end}}When executing the command, set the service to maintenance mode in Consul. You only need to package Consul on your command maint Call:
#!/bin/sh set -e consul maint -enable -service web -reason "Consul Template updated" service nginx reload consul maint -disable -service web
In addition, if you do not install the consult agent, you can directly call the API request:
#!/bin/sh set -e curl -X PUT "http://$CONSUL_HTTP_ADDR/v1/agent/service/maintenance/web?enable=true&reason=Consul+Template+Updated" service nginx reload curl -X PUT "http://$CONSUL_HTTP_ADDR/v1/agent/service/maintenance/web?enable=false"
services
Query all services in the consult directory using the following syntax:
{{services}}This example will query the default data center of consult. You can specify an optional parameter to specify the data center:
{{services "@east-aws"}}- Please note: the services function is different from the service. The service accepts more parameters and queries the list of monitored services. This queries the consult directory and returns a Map of the tag of a service, as follows:
{{range services}}
{{.Name}}
{{range .Tags}}
{{.}}{{end}}
{{end}}tree
Query all key value pairs of the specified prefix. If any of the values cannot be converted to a string, an error will be raised:
{{range tree "service/redis@east-aws"}}
{{.Key}} {{.Value}}{{end}}If the Consul instance has a service/redis structure in the East AWS data center, the rendering result of the template is similar to the following:
minconns 2 maxconns 12 nested/config/value "value"
Unlike ls, the tree returns all key s under the prefix. It is similar to the Unix tree command. If the data center parameters are ignored, the local data center will be used
Haproxy instance
Create a consumer template rendering file according to the configuration file of haproxy service: haproxy.ctmpl
# Consul Haproxy configured
global
maxconn 20480
ulimit-n 65535
log 127.0.0.1 local5
uid 200
gid 200
chroot /usr/local/haproxy
nbproc 1
daemon
pidfile /usr/local/haproxy/logs/haproxy.pid
defaults
log global
mode http
option httplog
option dontlognull
option forwardfor
option abortonclose
retries 3
maxconn 3000
stats enable
stats hide-version
stats uri /admin
stats auth admin:admin
stats refresh 10s
balance roundrobin
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms
timeout check 2000ms
listen web_haproxy
bind 0.0.0.0:8080
mode http
log 127.0.0.1 local5 err
stats refresh 5s
stats uri /admin
stats realm liang lian
stats auth admin:admin
stats hide-version
stats admin if TRUE
frontend consul
bind 0.0.0.0:8500
mode http
log global
default_backend consul-cluster
backend consul-cluster
mode http
{{range service "Faceid"}}
server {{.ID}} {{.Address}}:{{.Port}} check inter 5000 fall 1 rise 2 weight 2{{end}}Run the consumer template as a service, render a haproxy.cfg configuration file through the above rendering template, and then restart the haproxy service
consul-template -consul-addr=10.201.102.185:8500 -template "/root/haproxy.ctmpl:/etc/haproxy.cfg:service haproxy restart"
- 1
10.201.102.185 read my last article Haproxy , this is the VIP of the consumer cluster. In order to avoid calling a server separately, the consumer template cannot work after the server fails.
Rendered haproxy.cfg
# Consul Haproxy configured
global
maxconn 20480
ulimit-n 65535
log 127.0.0.1 local5
uid 200
gid 200
chroot /usr/local/haproxy
nbproc 1
daemon
pidfile /usr/local/haproxy/logs/haproxy.pid
defaults
log global
mode http
option httplog
option dontlognull
option forwardfor
option abortonclose
retries 3
maxconn 3000
stats enable
stats hide-version
stats uri /admin
stats auth admin:admin
stats refresh 10s
balance roundrobin
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms
timeout check 2000ms
listen web_haproxy
bind 0.0.0.0:8080
mode http
log 127.0.0.1 local5 err
stats refresh 5s
stats uri /admin
stats realm liang lian
stats auth admin:admin
stats hide-version
stats admin if TRUE
frontend consul
bind 0.0.0.0:8500
mode http
log global
default_backend consul-cluster
backend consul-cluster
mode http
server Faceid 10.201.102.198:9000 check inter 5000 fall 1 rise 2 weight 2
server Faceid 10.201.102.199:9000 check inter 5000 fall 1 rise 2 weight 2
server Faceid 10.201.102.200:9000 check inter 5000 fall 1 rise 2 weight 2The whole is to build a consumer cluster, and the services in the platform will be registered in the consumer cluster. haproxy avoids the single point of failure when the consumer template calls the consumer. The consumer template cannot work and is highly available. The consumer template can be used in various systems and applications of the whole platform to query the consumer cluster to obtain the survival status and IP of various applications on the platform.
The whole set has realized two key points:
The central service registration query is realized
Query services and configuration files of other nodes in the platform are automatically updated
reference material
https://my.oschina.net/guol/blog/675281
https://www.consul.io/docs/guides/index.html
https://www.gitbook.com/book/vincentmi/consul-guide/details
https://my.oschina.net/abcfy2/blog/675665