Azkaban3.X Multiple Excutors Installation Documents
1. Compiling source code
1.1 Cloned Source Code
git clone https://github.com/azkaban/azkaban.git
1.2 Compilation
# Enter the azkaban directory cd azkaban; # Compiling projects using gradle ./gradlew build installDist
After compilation, build directories will be generated in directories such as azkaban-executor-server and azkaban-db.
2.mysql configuration
2.1 Create a database of azkaban
mysql> create database azkaban;
2.2 Create a user for azkaban and authorize it
mysql> CREATE USER 'azkaban'@'%' IDENTIFIED BY 'azkaban'; mysql> GRANT SELECT,INSERT,UPDATE,DELETE ON azkaban.* to 'azkaban'@'%' WITH GRANT OPTION; mysql> flush privileges;
2.3 Import the sql of the creation table of azkaban
Go to the azkaban/azkaban-bd/build/sql directory and import the create-all-sql-3.60.0-15-gee6e229.sql file into mysql.
# Import data into mysql mysql> use azkaban; mysql> source /opt/azkaban/azkaban-db/build/sql/create-all-sql-3.60.0-15-gee6e229.sql
3. Configure keystore
Execute commands to generate keystore files:
keytool -keystore keystore -alias jetty -genkey -keyalg RSA
All passwords here are set to azkaban, which eventually generates a keystore file in the current directory and copies the file to the azkaban-web-server directory.
cp keystore /opt/azkaban/azkaban-web-server/build/install/azkaban-web-server/
4. Configure azkaban-exec-server
Enter the / opt/azkaban/azkaban-exec-server/build/install/azkaban-exec-server/conf directory and modify the azkaban.properties file
Modified file contents:
# Azkaban Personalization Settings azkaban.name=DevAzkaban azkaban.label=Dev Azkaban azkaban.color=#FF3601 azkaban.default.servlet.path=/index web.resource.dir=web/ default.timezone.id=Asia/Shanghai # Azkaban UserManager class user.manager.class=azkaban.user.XmlUserManager user.manager.xml.file=/opt/azkaban/azkaban-web-server/build/install/azkaban-web-server/conf/azkaban-users.xml # Loader for projects executor.global.properties=/opt/azkaban/azkaban-web-server/build/install/azkaban-web-server/conf/global.properties azkaban.project.dir=projects # Velocity dev mode velocity.dev.mode=false # Azkaban Jetty server properties. # jetty.use.ssl=false jetty.maxThreads=25 jetty.ssl.port=8443 jetty.port=8081 jetty.keystore=keystore jetty.password=azkaban jetty.keypassword=azkaban jetty.truststore=keystore jetty.trustpassword=azkaban # Where the Azkaban web server is located # azkaban.webserver.url=http://localhost:8081 # mail settings mail.sender= mail.host= # User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users. # enduser -> myazkabanhost:443 -> proxy -> localhost:8081 # when this parameters set then these parameters are used to generate email links. # if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used. # azkaban.webserver.external_hostname=myazkabanhost.com # azkaban.webserver.external_ssl_port=443 # azkaban.webserver.external_port=8081 job.failure.email= job.success.email= lockdown.create.projects=false cache.directory=cache # JMX stats jetty.connector.stats=true executor.connector.stats=true # Azkaban plugin settings azkaban.jobtype.plugin.dir=/opt/azkaban/azkaban-exec-server/build/install/azkaban-exec-server/plugins/jobtypes # Azkaban mysql settings by default. Users should configure their own username and password. database.type=mysql mysql.port=3306 mysql.host=dev-bg-01 mysql.database=azkaban mysql.user=azkaban mysql.password=azkaban mysql.numconnections=100 # Azkaban Executor settings executor.maxThreads=50 executor.flow.threads=30 # Azkaban Executor settings executor.port=12321 azkaban.use.multiple.executors=true azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1 azkaban.executorselector.comparator.Memory=1 azkaban.executorselector.comparator.LastDispatched=1 azkaban.executorselector.comparator.CpuUsage=1
The modifications here include:
1. Change default.timezone.id=America/Los_Angeles to default.timezone.id=Asia/Shanghai
2. Change relative path to absolute path
3. Support jetty's ssl and http
4.executor.port=12321
5.mysql information
5. Configure azkaban-web-server
# Azkaban Personalization Settings azkaban.name=DevAnkaban azkaban.label=Dev Azkaban azkaban.color=#FF3601 azkaban.default.servlet.path=/index web.resource.dir=web/ default.timezone.id=Asia/Shanghai # Azkaban UserManager class user.manager.class=azkaban.user.XmlUserManager user.manager.xml.file=/opt/azkaban/azkaban-web-server/build/install/azkaban-web-server/conf/azkaban-users.xml # Loader for projects executor.global.properties=/opt/azkaban/azkaban-web-server/build/install/azkaban-web-server/conf/global.properties azkaban.project.dir=projects # Velocity dev mode velocity.dev.mode=false # Azkaban Jetty server properties. # jetty.use.ssl=false jetty.maxThreads=25 jetty.ssl.port=8443 jetty.port=8081 jetty.keystore=keystore jetty.password=azkaban jetty.keypassword=azkaban jetty.truststore=keystore jetty.trustpassword=azkaban # Azkaban Executor settings # mail settings mail.sender= mail.host= # User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users. # enduser -> myazkabanhost:443 -> proxy -> localhost:8081 # when this parameters set then these parameters are used to generate email links. # if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used. # azkaban.webserver.external_hostname=myazkabanhost.com # azkaban.webserver.external_ssl_port=443 # azkaban.webserver.external_port=8081 job.failure.email= job.success.email= lockdown.create.projects=false cache.directory=cache # JMX stats jetty.connector.stats=true executor.connector.stats=true # Azkaban mysql settings by default. Users should configure their own username and password. database.type=mysql mysql.port=3306 mysql.host=dev-bg-01 mysql.database=azkaban mysql.user=azkaban mysql.password=azkaban mysql.numconnections=100 # Azkaban Executor settings executor.port=12321 azkaban.use.multiple.executors=true azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1 azkaban.executorselector.comparator.Memory=1 azkaban.executorselector.comparator.LastDispatched=1 azkaban.executorselector.comparator.CpuUsage=1
Same configuration as azkaban-exec-server.
6. Start azkaban-exec-server and azkaban-web-server
6.1 Start azkaban-exec-server
Go to the azkaban-exec-server directory and execute:
# Start the azkaban-exec-server service > bin/start-exec.sh # Stop it > bin/shutdown-exec.sh
Activate the executor:
curl -G "localhost:$(<./executor.port)/executor?action=activate" && echo
6.2 Start azkaban-web-server
Go to the azkaban-web-server directory and execute:
# Start the azkaban-web-server service > bin/start-web.sh # Stop it > bin/shutdown-web.sh
After successful startup, access url: https://dev-bg-01:8443 to the Azkaban landing page. The user name and password are in the azkaban-users.xml file in the azkaban-web-server/conf directory. The default user name and password are azkaban.
7. Problems encountered in Azkaban startup
The main problem with Azkaban is that there is no pairing in the configuration file:
7.1 java.lang.IllegalArgumentException
java.lang.IllegalArgumentException: azkaban.use.multiple.executors must be true. Single executor mode is not supported any more.
This is that azkaban.use.multiple.executors are not set to true in the configuration file: just add the following configuration.
azkaban.use.multiple.executors=true azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1 azkaban.executorselector.comparator.Memory=1 azkaban.executorselector.comparator.LastDispatched=1 azkaban.executorselector.comparator.CpuUsage=1
7.2 java.io.FileNotFoundException
This exception requires that the relative path in the configuration be changed to the absolute path
For example:
user.manager.xml.file=/opt/azkaban/azkaban-web-server/build/install/azkaban-web-server/conf/azkaban-users.xml executor.global.properties=/opt/azkaban/azkaban-web-server/build/install/azkaban-web-server/conf/global.properties azkaban.jobtype.plugin.dir=/opt/azkaban/azkaban-exec-server/build/install/azkaban-exec-server/plugins/jobtypes
7.3 azkaban.executor.ExecutorManagerException: No active executors found
7.4 jetty is accessed directly through http without keystore
Access directly through http can operate the interface, but can not perform tasks, through jetty through the keystore configuration ssl, can be executed.
Specific configuration:
# jetty.use.ssl=false jetty.maxThreads=25 jetty.ssl.port=8443 jetty.port=8081 jetty.keystore=keystore jetty.password=azkaban jetty.keypassword=azkaban jetty.truststore=keystore jetty.trustpassword=azkaban
Reference documents: