Simple Face Detection

This example shows the smallest useful Qt Edge AI application: a QML scene that displays live camera video and, at the same time, feeds the frames into an on-device inference pipeline. The pipeline is declared entirely in QML with the InferenceFlow, InferenceVideoSource, VisionInference, and InferenceResultParser types; no C++ inference code is needed. Study it to see how a capture session from Qt Multimedia is shared between the on-screen video preview and an inference flow, and how an application receives inference results.

Running the example

The example needs a camera and the GStreamer backend of Qt Edge AI; see Platform notes below.

The example takes two command line options, which choose the step provider that runs the model:

Platform notes below lists the providers that the example ships a configuration for, and how to run each of them.

What to look for

One capture session for preview and inference

CaptureSession {
    id: session
    camera: Camera {}
    videoOutput: videoOutput
}

VideoOutput {
    id: videoOutput
    anchors.fill: parent
}

A CaptureSession owns the Camera and renders its frames to the VideoOutput that fills the window. The same session is assigned to the flow's InferenceVideoSource below, so the preview and the inference pipeline work from one camera source.

The inference pipeline

InferenceFlow {
    id: pipeline

    InferenceVideoSource {
        id: videoSource
        captureSession: session
    }

    VisionInference {
        id: detectionStep
        provider: window.provider
        configFile: window.configFile
    }

    InferenceResultParser {
        id: jsonParser
        onDataAvailable: data => console.log(data)
    }
}

The steps are declared as children of InferenceFlow. None of them declares an explicit DataPortConnection, so the flow chains them in declaration order by matching output and input port types: camera video flows from the InferenceVideoSource into the VisionInference step, and the model output continues into the InferenceResultParser.

On the VisionInference step, provider names the step provider, which main.cpp sets from the --provider command line option. configFile names the provider's configuration file: the one that --config gives, or by default configs/<provider>/inference.json relative to the directory of the executable. The file holds what its provider reads; see Qt Edge AI Configuration Files.

The parser's dataAvailable signal delivers each parsed result, which this example simply logs to the console.

Starting the flow

Component.onCompleted: {
    session.camera.start()
    pipeline.start(window)
}

The camera and the flow only start once the scene has loaded: the handler starts the camera and calls start(), passing the window so that the pipeline starts on the render thread.

Platform notes

The inference steps are provided by the GStreamer backend plugin of Qt Edge AI, which is built only when GStreamer 1.20 or later is available. The example also sets the QT_MEDIA_BACKEND environment variable to gstreamer in main.cpp, so it requires a Qt Multimedia GStreamer media backend.

The example ships a configuration for each of the providers below. Without the --provider option, it uses the default step provider of the target: the first of the providers that it has when they are ordered alphabetically by the name in the metadata of their plugins, so Edge Impulse on a Qualcomm device and NVIDIA DeepStream on an NVIDIA Jetson. The models are not shipped with the example; what each provider needs is described in its own section.

Edge Impulse

The Edge Impulse provider runs a model that Edge Impulse compiles into a dynamic library. To use it, start the example with --provider edgeimpulse.

Its configuration file, configs/edgeimpulse/inference.json, is installed relative to the directory of the executable. Its pluginPath key names the model library for the provider to load, relative to the directory of the executable. Keep the file name that Edge Impulse gives the library, for example libgstedgeimpulse-898869-face-detection-fomo.so, and set pluginPath to it. A renamed library fails to load. See Edge Impulse in the configuration file documentation for the details.

NVIDIA DeepStream

The NVIDIA DeepStream provider runs a model with the NVIDIA DeepStream SDK, on NVIDIA Jetson AGX Orin and Jetson Thor. To use it, start the example with --provider deepstream --config configs/deepstream/dsdetect1_config.yml.

The example installs two DeepStream configuration files in configs/deepstream: dsdetect1_config.yml, a DeepStream YAML configuration that sets up nvstreammux and chooses nvinfer, and dsdetect1_pgie_config.yml, the nvinfer configuration that it names. The latter runs the primary detector model of the DeepStream samples, from the DeepStream SDK's installation directory; change it to run another model. See NVIDIA DeepStream in the configuration file documentation for the details.

The Qt Edge AI DeepStream backend plugin is built only when the DeepStream SDK 9.0 or later and a matching CUDA toolkit are found. It is tested with DeepStream 9.1.

Qualcomm IM SDK

The Qualcomm Intelligent Multimedia SDK (IM SDK) provider runs a TensorFlow Lite model on Qualcomm devices. To use it, start the example with --provider imsdk.

Its configuration file, configs/imsdk/inference.json, is installed relative to the directory of the executable. Its model and labels keys name the TensorFlow Lite model and its labels, relative to the directory of the executable. The example ships the labels, models/face_detection_plain.labels, but not the model. Download Lightweight-Face-Detection_w8a8.tflite for the Dragonwing IQ-9075 EVK from AI Hub, and copy it to models. The file sets the qfd post-processing module, and names no delegate, so the provider runs the model on the NPU through the QNN delegate. The delegate needs ADSP_LIBRARY_PATH to name the DSP library directories of the device. See Qualcomm IM SDK in the configuration file documentation for the details.

The device needs the IM SDK GStreamer elements, and Qt Multimedia built with -DFEATURE_gstreamer_qt_api=ON, so that the frames reach the backend in DMA buffers.

Files:

• simplefacedetection/CMakeLists.txt
• simplefacedetection/Main.qml
• simplefacedetection/main.cpp