In the process of using the Android SDK for face recognition, face frames usually need to be drawn when previewing, but different from the PC platform camera application, the application development of Android platform camera also needs to consider the situation of front and back camera switching, device horizontal and vertical screen switching, etc., so in the development process of face recognition project, the realization of face frame rendering adaptation is relatively difficult. To solve this problem, this paper will introduce the solution through the following contents:

• Relationship between camera original frame data and preview image
• The process of drawing face frame to View
• Specific scenario adaptation scheme introduction
• Deal with the situation of multiple scenes and realize the adaptation function
• Draw the adapted face frame on the View

The Rect instructions used below:

I. relationship between camera original frame data and preview image

Android devices are generally hand-held devices. The camera is integrated on the device. The rotation of the device will also cause the camera to rotate, so the imaging will also rotate. In order to solve this problem and enable users to see the normal imaging, Android provides the API for setting the rotation angle when the camera preview data is drawn to the control. Developers can set different settings according to the display direction of the Activity. The rotation angle of is described in the following article:
Android uses Camera2 to get preview data
Convert the previewed YUV data to NV21, then to Bitmap and display it on the control. At the same time, convert the Bitmap to the camera preview effect Bitmap and display it on the control, so as to understand the relationship between the original data and the preview screen.

2. Flow of drawing face frame to View

Overall process

• Step 1, zoom

• Step 2, rotate

It is necessary to select the corresponding rotation scheme according to the rotation angle relationship between image data and preview screen.

• Rear camera (preview not mirrored)

Rear camera, 0 degree rotation

Rear camera, rotate 90 degrees

Rear camera, rotate 180 degrees

Rear camera, rotate 270 degrees

• Front camera (preview will mirror)

Front camera, 0 degree rotation

Front camera, rotate 90 degrees

Front camera, rotate 180 degrees

Front camera, rotate 270 degrees

III. introduction of adaptation scheme in specific scenarios

Take the following scenario as an example to introduce the human face frame adaptation scheme:

Screen resolution	Camera preview size	Camera ID	Screen orientation	Raw data	Preview effect
1080x1920	1280x720	Rear camera	Vertical screen	Raw data	Preview effect

It can be seen that in the case of vertical screen, the original data can be rotated 90 degrees clockwise and zoomed to achieve the effect of preview screen. Since the image data is rotated and zoomed, the face frame also needs to rotate and zoom with the image. We can rotate first and then scale, or scale first and then rotate. Here we take scale first and then rotate as an example to introduce the steps of adaptation.

Step 1, zoom

Step 2, rotate

• Step 1: Zoom
  Assuming that the location information of the face detection result is originalRect: (left, top, right, bottom) (relative to the location of 1280x720 image), we enlarge it to the location of 1920x1080 image:
  scaledRect:(originalRect.left * 1.5, originalRect.top * 1.5, originalRect.right * 1.5, originalRect.bottom * 1.5)

• Step 2: rotate
  After the size modification, we can rotate the face frame to get the target face frame. The rotation process is as follows:

  1. Obtain the rotation angle of the original data and preview screen (90 degrees in the above case)

  2. According to the rotation angle, the face frame is adjusted to the face frame needed by View. For the face frame needed for rendering, we analyze the following calculation method:

     • drawRect.left
       Draw the left value of the required Rect, that is, the distance from the lower boundary of scaledRect to the lower boundary of the image, that is, 1080 - scaledRect.bottom
     • drawRect.top
       The value of top for drawing the required Rect is the distance from the left boundary of scaledRect to the left boundary of the image, which is scaledRect.left.
     • drawRect.right
       Draw the right value of the required Rect, that is, the distance from the upper boundary of scaledRect to the lower boundary of the image, that is, 1080 - scaledRect.top
     • drawRect.bottom
       The value of the bottom for drawing the required Rect is the distance from the right boundary of scaledRect to the upper boundary of the image, which is scaledRect.right.

Finally, the drawRect needed for drawing when the rotation angle is 90 degrees is obtained.

4. Deal with a variety of scenarios to realize the adaptation function

Through the above analysis, it can be concluded that the drawing parameters needed for frame drawing are as follows. The last two parameters of the constructor are added additionally for manual correction of special scenes:

• previewWidth & previewHeight
  Preview width and height, the face frame of face tracking is based on this size
• canvasWidth & canvasHeight
  The width and height of the drawn control, that is, the mapped target size
• cameraDisplayOrientation
  Preview rotation angle of data and source data
• cameraId
  Camera ID, the system does default image processing for the front camera, while the rear camera does not
• isMirror
  Whether the preview screen is displayed horizontally. For example, if we manually set the preview screen to be mirrored again, we need to mirror the final result as well.
• mirrorHorizontal
  For use with some compatible devices, mirror the adjusted frame horizontally again
• mirrorVertical
  For compatibility with some devices, mirror the adjusted frame vertically again

/**
 * Create a drawing helper class object and set the drawing related parameters
 *
 * @param previewWidth             Preview width
 * @param previewHeight            Preview height
 * @param canvasWidth              Draw the width of the control
 * @param canvasHeight             Draw the height of the control
 * @param cameraDisplayOrientation Rotation angle
 * @param cameraId                 Camera ID
 * @param isMirror                 Whether to mirror horizontally (set to true for correction if the camera is manually mirrored)
 * @param mirrorHorizontal         For use with some compatible devices, mirror horizontally again
 * @param mirrorVertical           For use with some compatible devices, mirror vertically again
 */
public DrawHelper(int previewWidth, int previewHeight, int canvasWidth,
                  int canvasHeight, int cameraDisplayOrientation, int cameraId,
                  boolean isMirror, boolean mirrorHorizontal, boolean mirrorVertical) {
    this.previewWidth = previewWidth;
    this.previewHeight = previewHeight;
    this.canvasWidth = canvasWidth;
    this.canvasHeight = canvasHeight;
    this.cameraDisplayOrientation = cameraDisplayOrientation;
    this.cameraId = cameraId;
    this.isMirror = isMirror;
    this.mirrorHorizontal = mirrorHorizontal;
    this.mirrorVertical = mirrorVertical;
}

The realization of face frame mapping

/**
     * Adjust the face frame to draw
     *
     * @param ftRect FT Face frame
     * @return The adjusted needs are drawn to rect on the View
     */
    public Rect adjustRect(Rect ftRect) {
        // Preview height
        int previewWidth = this.previewWidth;
        int previewHeight = this.previewHeight;

        // The width and height of the canvas, that is, the width and height of the View
        int canvasWidth = this.canvasWidth;
        int canvasHeight = this.canvasHeight;

        // Camera preview shows rotation angle
        int cameraDisplayOrientation = this.cameraDisplayOrientation;

        // Camera Id, the front camera will be mirrored by default when it is displayed
        int cameraId = this.cameraId;

        // Preview image or not
        boolean isMirror = this.isMirror;

        // For some special scenes, do additional face frame image operation.
        // For example, when the camera with cameraId of camera? Facing? Front is turned on, there is no image.
        // Or when the camera with cameraId of camera? Facing? Back is turned on
        boolean mirrorHorizontal = this.mirrorHorizontal;
        boolean mirrorVertical = this.mirrorVertical;

        if (ftRect == null) {
            return null;
        }

        Rect rect = new Rect(ftRect);
        float horizontalRatio;
        float verticalRatio;

        // When cameraDisplayOrientation is 0 or 180, i.e. landscape or reverse landscape
        // or
        // When cameraDisplayOrientation is 90 or 270, i.e. portrait or reverse portrait
        // Calculate the horizontal scale ratio and vertical scale ratio respectively
        if (cameraDisplayOrientation % 180 == 0) {
            horizontalRatio = (float) canvasWidth / (float) previewWidth;
            verticalRatio = (float) canvasHeight / (float) previewHeight;
        } else {
            horizontalRatio = (float) canvasHeight / (float) previewWidth;
            verticalRatio = (float) canvasWidth / (float) previewHeight;
        }
        rect.left *= horizontalRatio;
        rect.right *= horizontalRatio;
        rect.top *= verticalRatio;
        rect.bottom *= verticalRatio;

        Rect newRect = new Rect();
        // In the key part, the face frame is rotated and mirrored according to the rotation angle and camera ID.
        switch (cameraDisplayOrientation) {
            case 0:
                if (cameraId == Camera.CameraInfo.CAMERA_FACING_FRONT) {
                    newRect.left = canvasWidth - rect.right;
                    newRect.right = canvasWidth - rect.left;
                } else {
                    newRect.left = rect.left;
                    newRect.right = rect.right;
                }
                newRect.top = rect.top;
                newRect.bottom = rect.bottom;
                break;
            case 90:
                newRect.right = canvasWidth - rect.top;
                newRect.left = canvasWidth - rect.bottom;
                if (cameraId == Camera.CameraInfo.CAMERA_FACING_FRONT) {
                    newRect.top = canvasHeight - rect.right;
                    newRect.bottom = canvasHeight - rect.left;
                } else {
                    newRect.top = rect.left;
                    newRect.bottom = rect.right;
                }
                break;
            case 180:
                newRect.top = canvasHeight - rect.bottom;
                newRect.bottom = canvasHeight - rect.top;
                if (cameraId == Camera.CameraInfo.CAMERA_FACING_FRONT) {
                    newRect.left = rect.left;
                    newRect.right = rect.right;
                } else {
                    newRect.left = canvasWidth - rect.right;
                    newRect.right = canvasWidth - rect.left;
                }
                break;
            case 270:
                newRect.left = rect.top;
                newRect.right = rect.bottom;
                if (cameraId == Camera.CameraInfo.CAMERA_FACING_FRONT) {
                    newRect.top = rect.left;
                    newRect.bottom = rect.right;
                } else {
                    newRect.top = canvasHeight - rect.right;
                    newRect.bottom = canvasHeight - rect.left;
                }
                break;
            default:
                break;
        }

        /**
         * isMirror mirrorHorizontal finalIsMirrorHorizontal
         * true         true                false
         * false        false               false
         * true         false               true
         * false        true                true
         *
         * XOR
         */
        if (isMirror ^ mirrorHorizontal) {
            int left = newRect.left;
            int right = newRect.right;
            newRect.left = canvasWidth - right;
            newRect.right = canvasWidth - left;
        }
        if (mirrorVertical) {
            int top = newRect.top;
            int bottom = newRect.bottom;
            newRect.top = canvasHeight - bottom;
            newRect.bottom = canvasHeight - top;
        }
        return newRect;
    }

V. draw the adapted face frame on the View

• Implement a custom View

/**
 * Controls for displaying face information
 */
public class FaceRectView extends View {
    private static final String TAG = "FaceRectView";
    private CopyOnWriteArrayList<DrawInfo> drawInfoList = new CopyOnWriteArrayList<>();
    private Paint paint;

    public FaceRectView(Context context) {
        this(context, null);
    }

    public FaceRectView(Context context, @Nullable AttributeSet attrs) {
        super(context, attrs);
        paint = new Paint();
    }

    // Main drawing operations
    @Override
    protected void onDraw(Canvas canvas) {
        super.onDraw(canvas);
        if (drawInfoList != null && drawInfoList.size() > 0) {
            for (int i = 0; i < drawInfoList.size(); i++) {
                DrawHelper.drawFaceRect(canvas, drawInfoList.get(i), 4, paint);
            }
        }
    }
    // Empty the face in the picture
    public void clearFaceInfo() {
        drawInfoList.clear();
        postInvalidate();
    }

    public void addFaceInfo(DrawInfo faceInfo) {
        drawInfoList.add(faceInfo);
        postInvalidate();
    }

    public void addFaceInfo(List<DrawInfo> faceInfoList) {
        drawInfoList.addAll(faceInfoList);
        postInvalidate();
    }
}

• Specific operation of drawing, drawing face frame

/**
     * Draw the data information to the view. If {@ link drawinfo ා getname()} is not null, draw {@ link drawinfo ා getname()}.
     *
     * @param canvas            canvas of the view to be drawn
     * @param drawInfo          Drawing information
     * @param faceRectThickness Face frame thickness
     * @param paint             Paint brush
     */
    public static void drawFaceRect(Canvas canvas, DrawInfo drawInfo, int faceRectThickness, Paint paint) {
        if (canvas == null || drawInfo == null) {
            return;
        }
        paint.setStyle(Paint.Style.STROKE);
        paint.setStrokeWidth(faceRectThickness);
        paint.setColor(drawInfo.getColor());
        paint.setAntiAlias(true);

        Path mPath = new Path();
        //Left upper
        Rect rect = drawInfo.getRect();
        mPath.moveTo(rect.left, rect.top + rect.height() / 4);
        mPath.lineTo(rect.left, rect.top);
        mPath.lineTo(rect.left + rect.width() / 4, rect.top);
        //Right upper
        mPath.moveTo(rect.right - rect.width() / 4, rect.top);
        mPath.lineTo(rect.right, rect.top);
        mPath.lineTo(rect.right, rect.top + rect.height() / 4);
        //lower right
        mPath.moveTo(rect.right, rect.bottom - rect.height() / 4);
        mPath.lineTo(rect.right, rect.bottom);
        mPath.lineTo(rect.right - rect.width() / 4, rect.bottom);
        //Left lower
        mPath.moveTo(rect.left + rect.width() / 4, rect.bottom);
        mPath.lineTo(rect.left, rect.bottom);
        mPath.lineTo(rect.left, rect.bottom - rect.height() / 4);
        canvas.drawPath(mPath, paint);

        // It should be noted that x is the starting point of the horizontal direction.
        // y is BaseLine, and the text will be drawn above BaseLine
        if (drawInfo.getName() == null) {
            paint.setStyle(Paint.Style.FILL_AND_STROKE);
            paint.setTextSize(rect.width() / 8);

            String str = (drawInfo.getSex() == GenderInfo.MALE ? "MALE" : (drawInfo.getSex() == GenderInfo.FEMALE ? "FEMALE" : "UNKNOWN"))
                    + ","
                    + (drawInfo.getAge() == AgeInfo.UNKNOWN_AGE ? "UNKNWON" : drawInfo.getAge())
                    + ","
                    + (drawInfo.getLiveness() == LivenessInfo.ALIVE ? "ALIVE" : (drawInfo.getLiveness() == LivenessInfo.NOT_ALIVE ? "NOT_ALIVE" : "UNKNOWN"));
            canvas.drawText(str, rect.left, rect.top - 10, paint);
        } else {
            paint.setStyle(Paint.Style.FILL_AND_STROKE);
            paint.setTextSize(rect.width() / 8);
            canvas.drawText(drawInfo.getName(), rect.left, rect.top - 10, paint);
        }
    }

reminder:
After studying for a long time, I found that the adaptation scheme has been given in the Android Demo of hongsoft face recognition. The above code also comes from the official Demo. Through studying the Demo, I found that there are many other optimization strategies that may be used when accessing the hongsoft face recognition SDK, such as:
1. Multi face recognition by asynchronous face feature extraction
2. Use faceId to optimize recognition logic
3. Frame adaptation scheme in recognition
4. Turn on the dual camera for infrared live detection

Android Demo is available in [hongruan face recognition open platform] download