In docker's various network modes, bridge's network mode is to communicate with dockers on the same host. If we want to achieve cross-node communication between dockers on multiple hosts, we need to use overlay network.

In docker swarm mode, the container created by docker service creation defaults to the overlay network mode named ingress, in which service builds containers on different nodes (hosts) and the ip of containers on different nodes will be in the same subnet.

Similarly, if multiple services are established, such as nginx and viz, then the containers under both services will be under the same subnet. As shown below, on the same node, the containers ip of serivce nginx is 10.255.0.4, and the containers ip of service viz is 10.255.0.6, both of which are in the ings network.

#docker network inspect ingress
       "Internal": false,
       "Attachable": false,
       "Ingress": true,
        "Containers":{
           "00bf0cc88d8363581b10a6a64a34cc2864d51926ecaa445fba7af0bc488d553d":{
                "Name":"nginxtest.1.5yukmeotwnl2v0smmhy26bwkg",
               "EndpointID":"064080c4efc9048bf0b0a44ab1d52d63c627f277d9d589be8cc9723c081e2616",
               "MacAddress": "02:42:0a:ff:00:04",
                "IPv4Address":"10.255.0.4/16",
               "IPv6Address": ""
            },
           "ac7ec55f931e1a4c1ece6e56a935ac0871ab6fe88e9eae35e1671513c9204b77":{
                "Name":"viz.1.zhmcw7mtvzzrma31l3letnmxp",
               "EndpointID":"0477642232e30c34c9bdc6cb8e83b0d2726a5169df8daa8c47225b8d16163ec7",
               "MacAddress": "02:42:0a:ff:00:06",
                "IPv4Address":"10.255.0.6/16",
               "IPv6Address": ""
            },
           "ingress-sbox": {
               "Name": "ingress-endpoint",
               "EndpointID":"61ae637e13284274480a1f9928bd7c627543336875a64dbdd272850285252136",
               "MacAddress": "02:42:0a:ff:00:02",
               "IPv4Address": "10.255.0.2/16",
               "IPv6Address": ""
            }
        },
       "Options": {
           "com.docker.network.driver.overlay.vxlanid_list":"256"
...........................................................

If you don't want multiple services in the same subnet, such as multi-tenant scenario, in this case, you need to create a custom overlay network to achieve different users'services in their respective subnet.

Create an overlay network called mynet

 # docker network create mynet -d overlay

7njqr6p45krfw6msq8wgxdqu3

You can also use other parameters such as -- subnet: to define the scope of the subnet

View mynet basic information

# docker network inspect mynet
[
    {
       "Name": "mynet",
       "Id": "7njqr6p45krfw6msq8wgxdqu3",
       "Created": "0001-01-01T00:00:00Z",
       "Scope": "swarm",
       "Driver": "overlay",
       "EnableIPv6": false,
       "IPAM": {
           "Driver": "default",
           "Options": null,
           "Config": []
        },
       "Internal": false,
       "Attachable": false,
       "Ingress": false,
       "Containers": null,
       "Options": {
            "com.docker.network.driver.overlay.vxlanid_list":"4096"
        },
       "Labels": null
}

As shown above, the newly created mynet network vxlan id is 4096, which is different from ingress's vxlan id 256. At the same time, mynet has not been assigned an ip address segment because no container has been added to mynet network.

Create a service using mynet network

docker service create --replicas 2 --name nginx_test01 --network mynet  nginx

After serving, check mynet again

#docker network inspect mynet
.............................................................
           "Options": null,
           "Config": [
                {
                   "Subnet": "10.0.0.0/24",
                   "Gateway": "10.0.0.1"
                }
            ]
        },
       "Internal": false,
       "Attachable": false,
       "Ingress": false,
       "Containers": {
           "a67b21bdc3d1bb144816e436f5cc5a303539ae3db8a7564236740fc46233a665":{
               "Name": "nginx_test01.1.xscom3xofubdgzp1xixt69r93",
               "EndpointID": "0dbd0fca51d0c477ee653e6f0f12048e38acb6e1a404fe1f9ae4e6506563cfce",
               "MacAddress": "02:42:0a:00:00:03",
               "IPv4Address": "10.0.0.3/24",
               "IPv6Address": ""
            }
        },
       "Options": {
           "com.docker.network.driver.overlay.vxlanid_list":"4096"
 ..........................

You can see that mynet has added a container, and its network segment becomes 10.0.0/24.

Verify the interoperability of containers under different network segments

Access to containers using mynet networks

docker exec –it  a67b21bdc3d1 bash

Containers under [root@ a67b21bdc3d1/] Ping 10.255.0.6 Ping results network

PING 10.255.0.6 (10.255.0.6) 56(84) bytes of data. 

Unable to ping, indicating that vxlan isolation is in effect, if it is connected, you may need to upgrade the system kernel