How to compile Ali's MNN happily on windows

Keywords: cmake Android SDK Python

At present, in-depth learning has many high-quality open source frameworks on terminal deployment, such as Baidu's Paddle-lite, Ali's MNN, Tencent's ncnn.However, after watching a lot of evaluations, I finally chose MN of Ali to study.However, the corresponding tool chain provided by Ali is not particularly satisfied.I guess the most intertwined users are those using windows.(It is strongly recommended that the official use of Unified Scripting Language to implement the corresponding auxiliary scripts.)In this article, I'll show you how to modify the source code to compile MNN happily in windows (in fact, it can be used on other operating systems without doing anything).

Also, refer to the original document: https://www.yuque.com/mnn/cn/build_windows

Before proceeding with the tutorial, make sure your operating system has at least the following environments:

  • Microsoft Visual Studio (2017 or above) Note that the author is using vs2015 and needs to replace/WX in 3rd_party/flatbuffers/CMakeLists.txt with/WX-
  • cmake (recommended version 3.10 or above)
  • android sdk (if you need to compile android sdk on win)

1. First, under the schema, a generate.py code is implemented using python.Note, I'm using Python 3.x.

#-*-coding:utf-8-*-
#coding by: yuangu(lifulinghan@aol.com)

import os
import sys
import shutil
import platform

def p():
    frozen = "not"
    if getattr(sys, 'frozen',False):
        frozen = "ever so"
        return os.path.dirname(sys.executable)

    return os.path.split(os.path.realpath(__file__))[0]

currentWorkPath = p()
os.chdir(currentWorkPath)

if '-lazy' in sys.argv and os.path.isdir("current"):
    print("*** done ***")
    exit(0)

# check is flatbuffer installed or not
FLATC = '../3rd_party/flatbuffers/tmp/flatc' + ('.exe' if "Windows" ==  platform.system() else '')
FLATC = os.path.realpath(FLATC)

if not os.path.isfile(FLATC):
    print("*** building flatc ***")
    tmpDir = os.path.realpath('../3rd_party/flatbuffers/tmp')
    
    if os.path.isdir(tmpDir):
        shutil.rmtree(tmpDir)
    
    os.mkdir(tmpDir)
    os.chdir(tmpDir)

    os.system('cmake  -DCMAKE_BUILD_TYPE=Release ..')
    if "Windows" ==  platform.system():
        os.system('cmake --build . --target flatc --config Release')
        if os.path.isfile( os.path.join(tmpDir, 'Release/flatc.exe') ):
            shutil.move(os.path.join(tmpDir, 'Release/flatc.exe'), FLATC)
    else:
        os.system('cmake --build . --target flatc')


    # dir recover
    os.chdir(currentWorkPath)

# determine directory to use
DIR='default'
if os.path.isdir('private'):
    DIR = 'private'
DIR = os.path.realpath(DIR)

# clean up
print('*** cleaning up ***')
if os.path.isdir('current'):
    shutil.rmtree('current')
os.mkdir('current')

# flatc all fbs
os.chdir('current')
listFile = os.listdir(DIR)
for fileName in listFile:
    tmpFileName = os.path.join(DIR, fileName)
    cmd = "%s -c -b --gen-object-api --reflect-names %s" %(FLATC, tmpFileName)
    os.system(cmd)

os.chdir(currentWorkPath)
print( "*** done ***")



Put original steps

powershell ./schema/generate.ps1

change into

Python. /schema/generate.py example:

If successful, you will see the following results.

2. Next, transform the cmake file.

In MNN/CMakeLists.txt

# schema generator
if(WIN32)
    add_custom_target( MNN_SCHEMA ALL
        COMMAND powershell ${CMAKE_CURRENT_SOURCE_DIR}/schema/generate.ps1 -lazy
    )
else()
    add_custom_target( MNN_SCHEMA ALL
        COMMAND bash ${CMAKE_CURRENT_SOURCE_DIR}/schema/generate.sh -lazy
    )
endif()

Replace with the following code.

find_program(PYTHON_BIN python3)

if (NOT PYTHON_BIN)
    find_program(PYTHON_BIN python)
endif()

if (NOT PYTHON_BIN)
    message(FATAL_ERROR "please install python and add it's path to system path")
endif()

# schema generator
add_custom_target( MNN_SCHEMA ALL
    COMMAND ${PYTHON_BIN} ${CMAKE_CURRENT_SOURCE_DIR}/schema/generate.py -lazy
)

3. Formally compile source code

Then you can say that on the official website.(Note that the author has proven in practice that some of the vs2015 grammar support is insufficient and some errors will be reported in the compilation stage.If you don't have the ability to refine your code, it's not a big problem to be honest with the website, using vs2017 or more, and not using the official recommended Inja.)

mkdir build && cd build
cmake  -DCMAKE_BUILD_TYPE=Release ..

As shown in the diagram:

Last used

cmake --build .

4. Last nursery-level tips:

If you prefer mingw32 or another clang compiler, or ide, you can change the contents of -G.As to how -G is filled in.The following example can be executed to prompt

5. The following is for students who need to compile the so library for Android.(Please finish the 1 and 2 above)

Original document: https://www.yuque.com/mnn/cn/build_android

Strongly spit out the documents and tools used.Documentation defaults to cmake being used by users, and tools can only be used in unix-like environments.

^^Many users don't know what the documentation says about compilation options.^^.However, the android under demo in the official source code has a corresponding so library.It also eliminates the compilation process for many users.

1. Before compiling, check to see if your android sdk has the following components

The two sdk components can be installed using the GUI in android studio as shown in the figure above.

Also command the party for reference (the latest Android SDK is not verified, note:%ANDROID_HOME% is the root directory of Android sdk):

mkdir "%ANDROID_HOME%/licenses"
echo -e "\n8933bad161af4178b1185d1a37fbf41ea5269c55" > "%ANDROID_HOME%/licenses/android-sdk-license"
echo -e "\n84831b9409646a918e30573bab4c9c91346d8abd" > "%ANDROID_HOME%/licenses/android-sdk-preview-license"
"%ANDROID_HOME%/tools/bin/sdkmanager.bat" cmake;3.10.2.4988404 ndk-bundle

2. cd to project\android (note:%ANDROID_HOME% is the root directory of android sdk). If you don't like typing commands, you can skip this step directly.

mkdir build && cd build
%ANDROID_HOME%/cmake/3.10.2.4988404/bin/cmake.exe -DANDROID_ABI=arm64-v8a   \            
                -DCMAKE_BUILD_TYPE=Release   \
                -DANDROID_NDK=%ANDROID_HOME%/ndk-bundle    \
                -DANDROID_STL=c++_static \
                -DCMAKE_CXX_FLAGS=-std=c++11 -frtti -fexceptions   \
                -DCMAKE_TOOLCHAIN_FILE=%ANDROID_HOME%/ndk-bundle\build\cmake/android.toolchain.cmake    \
                -DCMAKE_MAKE_PROGRAM=%ANDROID_HOME%/cmake/3.10.2.4988404/bin/ninja.exe -G "Ninja"    \
                -DMNN_BUILD_FOR_ANDROID_COMMAND=true   \
                -DMNN_DEBUG=false -DNATIVE_LIBRARY_OUTPUT=. ../../..
cmake --build .

Note:

In the above command, ANDROID_ABI can select parameters other than arm64-v8a.There are also armeabi-v7a,x86,x86_64.If you need to support armeabi, mips, mips64, you need to do so by lowering the corresponding ndk and sdk versions.That's because google officially removed support for these platforms.

3. Below is the script implemented by the author in 2 steps.(There are actually andrid SDK and NDK detection and installation capabilities, but estimates make the script more complex, so I've removed it.) Please place it in the project/android directory and execute it

#-*-coding:utf-8-*
# code by yuangu(lifulinghan@aol.com)

import os
import platform
import shutil
import sys
import getopt

def p():
    frozen = "not"
    if getattr(sys, 'frozen',False):
        frozen = "ever so"
        return os.path.dirname(sys.executable)

    return os.path.split(os.path.realpath(__file__))[0]

def checkPath(path):
    if not os.path.isdir(path):      
        parent = os.path.dirname(path)
        if os.path.isdir(parent):
            os.mkdir(path)
        else:
            checkPath(parent)

SUPPER_ABI_LIST = [
    'armeabi-v7a',
    "arm64-v8a",
    "x86",
    'x86_64'
] 

class MNN_Builder:
    def __init__(self, argv):
        try:
            opts, args = getopt.getopt(argv,"hs:o:a:",['help', 'sdk=','out=','abi='])
        except getopt.GetoptError:
            self.usage()
            sys.exit(-1)

        androidSDKPath = None
        outDir = os.path.join(p(), 'out')
        abiList = [abi for abi in SUPPER_ABI_LIST]
        extOption = []

        for opt, arg in opts:
            if opt in ("-h","--help"):
                self.usage()
                sys.exit()
           
            elif opt in ("-s","--sdk"):
                androidSDKPath = arg
            
            elif opt in ("-o","--out"):
                outDir =  arg
            elif opt in ('-a', '--abi'):
                if arg == 'all':
                    pass
                elif arg in SUPPER_ABI_LIST:
                    abiList.append(arg)
                else:
                    print('abi not support')
                    self.usage()
                    sys.exit(-1)
            elif opt in ('-c', '--core'):
                if arg == 'opencl':
                    extOption.append("-DMNN_OPENCL=true")
                    extOption.append("-DANDROID_PLATFORM=android-16")
                elif arg == 'opengl':
                    extOption.append("-DMNN_OPENGL=true")
                    extOption.append("-DANDROID_PLATFORM=android-21")
                elif arg == 'vulkan':
                    extOption.append("-DMNN_VULKAN=true")
                    extOption.append("-DANDROID_PLATFORM=android-21")
                elif arg == 'cpu':
                    pass
                else:
                    print('the core not support')
                    self.usage()
                    sys.exit(-1)
            elif opt in ('-d', '--enable_debug'):
                extOption.append('-DMNN_DEBUG=true')
        
        if not '-DMNN_DEBUG=true' in extOption:
            extOption.append('-DMNN_DEBUG=false')


        if androidSDKPath == None:
            androidSDKPath = self.getAndroidSDKPath()
        
        #check android sdk
        if androidSDKPath == None or not os.path.isdir(androidSDKPath):
            print("not found android sdk")
            sys.exit(-1)

        androidNDKPath = self.getNDKPath(androidSDKPath)
        # check android ndk
        if androidNDKPath == None:
            print('not found android ndk')
            sys.exit(-1)
        
        cmakeDir = self. getCmakeDir(androidSDKPath)
        if cmakeDir == None:
            print("please install cmake in android sdk")
            exit(-1)

        outDir = os.path.realpath(outDir)
        checkPath(outDir)

        cmakeBin = os.path.join(cmakeDir,'bin/cmake')   + ( '.exe' if  'Windows' ==  platform.system() else '' )
        ninjaBin = os.path.join(cmakeDir,'bin/ninja')  + ( '.exe' if  'Windows' ==  platform.system() else '' )
        print(abiList)
        for abi in  abiList:
            build_path = self.build(abi, androidNDKPath, cmakeBin, ninjaBin, extOption)
            self.copySoToOut(build_path, abi, outDir)
        
        print('****done****')
        
    def usage(self):
        print('usage: python build_mnn.py [-s <Android SDK>] [-o <*.so out dir>] [-a <aib name>]')
        print("-h, --help  print this message")
        print("-s, -sdk  Android SDK dir path, default from system variables of ANDROID_HOME or ANDROID_SDK_ROOT ")
        print("-o, --out  *.so out dir default './out' ")
        print("-a, --abi  all,armeabi-v7a,arm64-v8a,x86,x86_64, default all")
        print("-c, --core cpu, opencl,opengl,vulkan, default cpu")
        print("-d, --enable_debug, default close")

    def getAndroidSDKPath(self):
        environ_names = [
            'ANDROID_HOME', 
            'ANDROID_SDK_ROOT'
        ]

        for name in environ_names:            
            #Not present in environment variable
            if name not  in os.environ.keys():
                continue

            android_sdk_path = os.environ[name]
            #Verify that this directory does not exist 
            if not  os.path.isdir(android_sdk_path):
                continue
            
            return android_sdk_path
        
        #No corresponding sdk path was found
        return None
    
    def getCmakeDir(self, androidSDKPath):
        ndk_cmake_dir  = os.path.join(androidSDKPath,  "cmake")
        if  not  os.path.isdir(ndk_cmake_dir):
            return None
        
        cmake_dir_list = os.listdir(ndk_cmake_dir)
        list_len = len(cmake_dir_list)
        if list_len <= 0:
            return  None
        
        return os.path.join(ndk_cmake_dir, cmake_dir_list[0] )


    # get ndk path from android sdk or NDK_ROOT or ANDROID_NDK
    def  getNDKPath(self, androidSDKPath):
        #Find through system variables
        environ_names = [
            'NDK_ROOT', 
            'ANDROID_NDK'
        ]

        for name in environ_names:            
            #Not present in environment variable
            if name not  in os.environ.keys():
                continue

            android_ndk_path = os.environ[name]
            #Verify that this directory does not exist 
            if not  os.path.isdir(android_ndk_path):
                continue
            
            return android_ndk_path
        
        ndk_bundle_dir  = os.path.join(androidSDKPath,  "ndk-bundle/toolchains")
        if os.path.isdir(ndk_bundle_dir):
            return  os.path.join(androidSDKPath, "ndk-bundle")
    
    def build(self, abi, androidNDKPath,cmakeBin ,ninjaBin, extOption):
        rootPath = p()
        build_path = os.path.join(rootPath, 'build/' + abi)
        checkPath(build_path)

        if os.path.isdir(build_path):
            shutil.rmtree(build_path)

        os.mkdir(build_path)
        os.chdir(build_path)
        
        cmd = '''%s -DANDROID_ABI=%s   \
                %s \
                -DCMAKE_BUILD_TYPE=Release   \
                -DANDROID_NDK=%s    \
                -DANDROID_STL=c++_static \
                -DCMAKE_CXX_FLAGS=-std=c++11 -frtti -fexceptions   \
                -DCMAKE_TOOLCHAIN_FILE=%s/build/cmake/android.toolchain.cmake    \
                -DCMAKE_MAKE_PROGRAM=%s -G "Ninja"    \
                -DMNN_BUILD_FOR_ANDROID_COMMAND=true   \
                -DMNN_DEBUG=false -DNATIVE_LIBRARY_OUTPUT=. ../../../../'''%(cmakeBin,abi, ' '.join(extOption), androidNDKPath,androidNDKPath,ninjaBin) 

        if (os.system(cmd) != 0 or  os.system("%s --build ."%(cmakeBin, )) != 0):
            print("build failed")
            sys.exit(-1)
        
        os.chdir(rootPath)
        return build_path

    def copySoToOut(self, build_path, abi, outDir):
        copyList = ['libMNN.so']
        for v in copyList:
            if os.path.exists(os.path.join( build_path, v )):
                checkPath(os.path.join(outDir, abi))
                shutil.copy( os.path.join( build_path, v ), os.path.join(outDir, abi, v ))



if __name__ == "__main__":
    MNN_Builder(sys.argv[1:])

    

4. Turn on the compilation option in the document.Take the MNN_OPENCL compilation option that compiles the OpenCL section for example.Simply add -DMNN_OPENCL=true or -DMNN_OPENCL=1 to the cmake command above.If you have this requirement and use my python script, please modify it yourself.The complete example of Command 2 is shown below.:

mkdir build && cd build
%ANDROID_HOME%/cmake/3.10.2.4988404/bin/cmake.exe -DANDROID_ABI=arm64-v8a   \            
                -DCMAKE_BUILD_TYPE=Release   \
                -DANDROID_NDK=%ANDROID_HOME%/ndk-bundle    \
                -DANDROID_STL=c++_static \
                -DCMAKE_CXX_FLAGS=-std=c++11 -frtti -fexceptions   \
                -DCMAKE_TOOLCHAIN_FILE=%ANDROID_HOME%/ndk-bundle\build\cmake/android.toolchain.cmake    \
                -DCMAKE_MAKE_PROGRAM=%ANDROID_HOME%/cmake/3.10.2.4988404/bin/ninja.exe -G "Ninja"    \
                -DMNN_BUILD_FOR_ANDROID_COMMAND=true   \
                -DMNN_DEBUG=false -DMNN_OPENCL=true -DNATIVE_LIBRARY_OUTPUT=. ../../..
cmake --build .

5. Also, the tools/script command has not been python modified.But at this point, it has basically not affected everyone's use.

If you're interested in the underlying principles of frameworks, I can write some thoughts about what I've read about this set of framework codes.

***************************************************End*********************************

Advertising time:

1. If you have a passion for in-depth learning algorithms, or are engaged in corresponding work and research.You can join me in WeChat to research and make progress together.Please note: Deep learning

2. I am good at cross-platform c++ development and python.If you have in-depth study of the corresponding work, do not disregard me as an unknown school (985, 211) and no SCI can also add the above micro-signals to contact me.

Posted by fellixombc on Tue, 10 Dec 2019 09:55:53 -0800