Cause, when using mirror-maker to test cluster copy data from production cluster top, the error is as follows:

[2018-10-23 10:21:47,821] FATAL [mirrormaker-thread-2] Mirror maker thread failure due to  (kafka.tools.MirrorMaker$MirrorMakerThread)
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for UMETRIP_ACTIVITY_COMPLETE_TOPIC-22: 30048 ms has passed since batch creation plus linger time
    at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:94)
    at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:64)
    at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:29)
    //.......

Combining the broker list of the test cluster on kafkaManager to display hostname instead of ip, DNS failed to parse hostname when mirror Maker connected to the test cluster (which is expected to be validated by debug log output from mirror-maker).

Comparing the registration information of two clusters on zookeeper for production and testing:

Production broker:

[zk: localhost:2181(CONNECTED) 3] get /brokers/ids/0
{"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://10.237.64.61:9092"],"jmx_port":9999,"host":"10.237.64.61","timestamp":"1537161676764","port":9092,"version":4}
cZxid = 0x40000172f
ctime = Mon Sep 17 13:21:17 CST 2018
mZxid = 0x40000172f
mtime = Mon Sep 17 13:21:17 CST 2018
pZxid = 0x40000172f
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x2009fcbb3900007
dataLength = 196
numChildren = 0

Test broker:

[zk: localhost:2181(CONNECTED) 3] get /brokers/ids/0
{"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://j1215353563-10:9092"],"jmx_port":9095,"host":"j1215353563-10","timestamp":"1539938435069","port":9092,"version":4}
cZxid = 0x1001b04e3
ctime = Fri Oct 19 16:40:20 CST 2018
mZxid = 0x1001b04e3
mtime = Fri Oct 19 16:40:20 CST 2018
pZxid = 0x1001b04e3
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x300386fd8634952
dataLength = 200
numChildren = 0 

From the information in zk, test cluster broker registers its hostname under / brokers/ids/x node. Seeing that some people on the Internet suggested putting the mapping of hostname ip under / etc/hosts, trying it did not work. In addition, there is no special host name added under the production cluster machine / etc/hosts, so it is suspected that the newly built test cluster configuration is out of order.

Put the focus back to the server.properties file and find the configuration instructions for the advertised.listeners property. It is clear that advertised.listeners need to be configured. If not configured, listeners property will be used. If listeners are not configured, get it by default: java. net. InetAddress. getCanonical HostName (), which is expected to return hostname. .

# Hostname and port the broker will advertise to producers and consumers. If not set,
# it uses the value for "listeners" if configured.  Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
#advertised.listeners=PLAINTEXT://your.host.name:9092

Comparing the configuration of the next two environments, the production broker has listeners, which was omitted from the test.

listeners=PLAINTEXT://10.237.64.61:9092

After adding configuration to the test machine, the problem was solved.