The camera architecture we are familiar with is the following picture:

The bottom layer is camera driver, which is strongly related to hardware; the top layer of camera driver is the operation-driven camera HAL layer, which is also the core code of each manufacturer's camera. The manufacturer encapsulates its own code well without observing the open source conditions. The camera HAL layer is also the focus of modification and optimization, which runs in the process of camera provider. The interview will be analyzed in detail. camera service is a bridge between camera provider and upper communication, just as an intermediate layer. camera framework is a well-known call interface for developers.

camera Architecture (1).jpg

 

The camera framework and camera service layers may be familiar to everyone, but the camera provider, the Camera HAL layer we are familiar with, is not yet clear. Our main purpose is to help you understand the relationship between camera service and camera provider.

I've talked a lot about the connection and relationship between the camera framework and the camera service. I don't want to describe it here. The camera HAL layer has always been mysterious, because google really built a camera HAL on the basis of camera driver in order to ensure that every manufacturer can protect their hardware source code completely. Because each manufacturer encapsulates its own interface, the camera HAL layer of each manufacturer is actually not open source, which is also the focus of each manufacturer's camera optimization.

1. When was the camera provider registered?

Just as aidl is the communication mode of IPC between framework and service, in fact, through Binder IPC, the communication between camera provider and camera service will also rely on a carrier Hal - > hardware abstract layer (hardware Abstract layer).

Let's look at a specific hal code example.

image.png

 

The. h file will be generated automatically when compiled. The interface included in this file can ensure that the camera provider and camera service can communicate directly with each other.

./out/soong/.intermediates/hardware/interfaces/camera/provider/2.4/android.hardware.camera.provider@2.4_genc++_headers/gen/android/hardware/camera/provider/2.4/ICameraProvider.h
./out/soong/.intermediates/hardware/interfaces/camera/provider/2.4/android.hardware.camera.provider@2.4_genc++_headers/gen/android/hardware/camera/provider/2.4/ICameraProviderCallback.h

Back to the point, we know that camera provider mainly deals with the function of camera HAL. camera service communicates with camera provider. It is also a cross-process call. Its principle is similar to that of aidl, and it also needs to register service. The specific registration process is as follows:

The figure below shows that when cameraprovider started, two providers were registered, one named legacy/0 and the other external/0.

camera provider registration process. jpg

 

The specific registration process can be seen according to the code path suggested by the above figure, which will not be repeated here.

2. How does camera service call camera provider?

camera provider registration place already knows, there must be calls, calls in the camera service, in order to facilitate your understanding, directly paste a sequence diagram, you can check the code yourself.

camera provider calls process. jpg

 

From the above invocation process, camera service initialization will also establish a link between camera service and camera provider, the established providerInfo will be placed in memory, and then directly fetch the memory providerInfo, which can be invoked from camera service to camera provider.

But here I want to talk about the specific process of contact:

status_t CameraProviderManager::initialize(wp<CameraProviderManager::StatusListener> listener,
        ServiceInteractionProxy* proxy) {
    std::lock_guard<std::mutex> lock(mInterfaceMutex);
    if (proxy == nullptr) {
        ALOGE("%s: No valid service interaction proxy provided", __FUNCTION__);
        return BAD_VALUE;
    }
    mListener = listener;
    mServiceProxy = proxy;
 
    // Registering will trigger notifications for all already-known providers
    bool success = mServiceProxy->registerForNotifications(
        /* instance name, empty means no filter */ "",
        this);
    if (!success) {
        ALOGE("%s: Unable to register with hardware service manager for notifications "
                "about camera providers", __FUNCTION__);
        return INVALID_OPERATION;
    }
 
    // See if there's a passthrough HAL, but let's not complain if there's not
    addProviderLocked(kLegacyProviderName, /*expected*/ false);
    addProviderLocked(kExternalProviderName, /*expected*/ false);
 
    return OK;
}

Above is the initialization process of Camera Provider Manager. Camera Provider Manager is an engineering management class that manages the communication between camera Service and camera provider. There are two parameters, the second of which is the remote agent class. This parameter is already the default assignment. Believe it or not, you can see the definition of Camera Provider Manager:: initialize.

status_t initialize(wp<StatusListener> listener,
            ServiceInteractionProxy *proxy = &sHardwareServiceInteractionProxy);

This is the definition. The default value is --> sHardware Service Interaction Proxy. This sHardware Service Interaction Proxy is an example of --> Hardware Service Interaction Proxy. It can be seen that ICameraProvider - > getService (...) has been directly invoked in the definition of Hardware Service Interaction Proxy.

// Standard use case - call into the normal generated static methods which invoke
// the real hardware service manager
struct HardwareServiceInteractionProxy : public ServiceInteractionProxy {
    virtual bool registerForNotifications(
            const std::string &serviceName,
            const sp<hidl::manager::V1_0::IServiceNotification>
            &notification) override {
        return hardware::camera::provider::V2_4::ICameraProvider::registerForNotifications(
                serviceName, notification);
    }
    virtual sp<hardware::camera::provider::V2_4::ICameraProvider> getService(
            const std::string &serviceName) override {
        return hardware::camera::provider::V2_4::ICameraProvider::getService(serviceName);
    }
};

Camera Provider Manager:: initialize adds two Provider Names, the two service names of our camera provider - --> (legacy/0 extrenal/0)
addProviderLocked(kLegacyProviderName, /expected/ false);
addProviderLocked(kExternalProviderName, /expected/ false);
The bridge between camera service and camera provider is realized here.

3. Camera Provider Manager ProviderInfo Data Structure

Camera Provider Manager provides a Provider Info to save Camera Provider information. It is convenient to manage camera service to call camera provider. Now let's analyze how Provider Info is like. Let's cut out the code in the analysis camera provider from this.

struct ProviderInfo :
        virtual public hardware::camera::provider::V2_4::ICameraProviderCallback,
        virtual public hardware::hidl_death_recipient
{
    const std::string mProviderName;
    const sp<hardware::camera::provider::V2_4::ICameraProvider> mInterface;
 
    ProviderInfo(const std::string &providerName,
            sp<hardware::camera::provider::V2_4::ICameraProvider>& interface,
            CameraProviderManager *manager);
 
 
    status_t initialize();
    status_t addDevice(const std::string& name,
            hardware::camera::common::V1_0::CameraDeviceStatus initialStatus =
            hardware::camera::common::V1_0::CameraDeviceStatus::PRESENT,
            /*out*/ std::string *parsedId = nullptr);
 
 
    // ICameraProviderCallbacks interface - these lock the parent mInterfaceMutex
    virtual hardware::Return<void> cameraDeviceStatusChange(
            const hardware::hidl_string& cameraDeviceName,
            hardware::camera::common::V1_0::CameraDeviceStatus newStatus) override;
    virtual hardware::Return<void> torchModeStatusChange(
            const hardware::hidl_string& cameraDeviceName,
            hardware::camera::common::V1_0::TorchModeStatus newStatus) override;
 
 
    // hidl_death_recipient interface - this locks the parent mInterfaceMutex
    virtual void serviceDied(uint64_t cookie, const wp<hidl::base::V1_0::IBase>& who) override;
 
 
    //......
};

ProviderInfo inherits hardware:: camera:: provider:: V2_4:: ICamera Provider Callback and hardware::hidl_death_recipient. The second parameter of ProviderInfo is the IPC interface between camera service and cameraprovider, which ensures smooth communication between the two layers.

ICamera Provider Callback is the callback interface of camera provider and can communicate between IPC s. hardware::hidl_death_recipient is the hal layer's death callback interface, which is convenient to notify the upper layer when the bottom layer dies.

The main function of initialize () --> initialize function is to initialize the camera provider, and IPC calls the camera provider to get the camera device information, and then calls the addDevice interface to save the acquired camera device in memory.

Add Device - > saves the camera device information acquired at the bottom in the camera service to prevent multiple cross-process calls.

Camera Device StatusChange and torchModeStatusChange are callback functions of ICamera Provider Callback, which need to be notified to the upper level when the camera provider changes.

Service Died is a callback function of hardware::hidl_death_recipient, which notifies the upper layer of changes when problems occur at the bottom.

In camera service, we use ProviderInfo to communicate with the underlying camera provider and save the camera device to run smoothly.

4. ProviderInfo - > DeviceInfo3 Data Structure

There are several DeviceInfo-type data structures in ProviderInfo, but the latest ones are all DeviceInfo3 data structures, so here we focus on DeviceInfo3 only.

// HALv3-specific camera fields, including the actual device interface
struct DeviceInfo3 : public DeviceInfo {
    typedef hardware::camera::device::V3_2::ICameraDevice InterfaceT;
    const sp<InterfaceT> mInterface;
 
    virtual status_t setTorchMode(bool enabled) override;
    virtual status_t getCameraInfo(hardware::CameraInfo *info) const override;
    virtual bool isAPI1Compatible() const override;
    virtual status_t dumpState(int fd) const override;
    virtual status_t getCameraCharacteristics(
            CameraMetadata *characteristics) const override;
 
    DeviceInfo3(const std::string& name, const metadata_vendor_id_t tagId,
            const std::string &id, uint16_t minorVersion,
            const hardware::camera::common::V1_0::CameraResourceCost& resourceCost,
            sp<InterfaceT> interface);
    virtual ~DeviceInfo3();
private:
    CameraMetadata mCameraCharacteristics;
};

InterfaceT defined in DeviceInfo3 is hardware::camera::device::V3_2::ICameraDevice type.

./hardware/interfaces/camera/device/3.2/ICameraDevice.hal ----> ./out/soong/.intermediates/hardware/interfaces/camera/device/3.2/android.hardware.camera.device@3.2_genc++_headers/gen/android/hardware/camera/device/3.2/ICameraDevice.h

Operating the underlying camera device hardware is called through this interface.