Qt Edge AI Configuration Files

A VisionInference step runs its model with a step provider, which a backend plugin provides for the GStreamer backend. The step's required provider property names the step provider, and its required configFile property names the file that tells the step provider what to run and how:

VisionInference {
    provider: "<providername>"
    configFile: "configs/<providername>/inference.json"
}

What the file contains, and in which format, depends entirely on the step provider, and can change from one version of a step provider to the next. Each step provider is described in its own section below.

Locating the File

The configFile property is a URL, but only local files are supported, because a step provider needs a file that it can open. A qrc: URL, or any other scheme than file:, makes the step fail to initialize. The file can have any name.

A relative configFile is resolved against the directory that contains the application's executable, not against the QML file that sets it. An absolute path is used as it is.

If the file is missing, cannot be read, or the step provider finds an error in it, the step logs a warning that names the file and the problem, and never becomes ready.

Edge Impulse

The edgeimpulse step provider runs a model that Edge Impulse compiles, together with its labels, into a dynamic library. It loads the library into GStreamer and hands the model the step's frames in RGB, the format that Edge Impulse models take.

Edge Impulse Configuration File

The configuration file is a JSON object with the following keys. Keys that the step provider does not know are ignored.

If pluginPath is missing, empty, or not a string, or the library cannot be loaded, the step logs a warning and never becomes ready.

Edge Impulse Configuration Example

The following configuration file, saved as configs/edgeimpulse/inference.json relative to the directory of the application's executable, runs the model that is compiled into libgstedgeimpulse-898869-face-detection-fomo.so in that directory:

{
    "pluginPath": "libgstedgeimpulse-898869-face-detection-fomo.so"
}

Keep the file name that Edge Impulse gives the library. A renamed library fails to load.

A vision step uses it like this:

VisionInference {
    provider: "edgeimpulse"
    configFile: "configs/edgeimpulse/inference.json"
}

NVIDIA DeepStream

The deepstream step provider runs a model with the NVIDIA DeepStream SDK, on NVIDIA Jetson AGX Orin and Jetson Thor. It builds a pipeline like the one of the DeepStream deepstream-test1 sample: it uploads the frames to GPU memory, batches them with nvstreammux, and runs the model with nvinfer. The deepstream backend plugin is built only when the DeepStream SDK 9.0 or later and a matching CUDA toolkit are found. It is tested with DeepStream 9.1.

DeepStream Configuration File

The configuration file is DeepStream's own, and the step provider hands it to DeepStream as it is. Its format depends on the file name:

• A file whose name ends in .yml or .yaml is a DeepStream YAML application configuration. The step provider reads two of its groups with DeepStream's YAML parser: streammux, with the properties of nvstreammux, and primary-gie, with the nvinfer inference element. Its config-file-path names the element's own configuration file, which describes the model. Other groups, such as the source and sink, are not used: the Qt Edge AI flow provides them.
• Any other file is the configuration file of nvinfer, in DeepStream's key-file format, and the step provider runs nvinfer with it.

The keys of these files, and how DeepStream resolves the paths in them, are described in the NVIDIA DeepStream SDK documentation.

If the file is missing or DeepStream cannot parse it, the step logs a warning and never becomes ready.

DeepStream Configuration Example

The following YAML configuration file, saved as configs/deepstream/dsdetect1_config.yml relative to the directory of the application's executable, batches the frames at 1920 x 1080 and runs nvinfer with the model that dsdetect1_pgie_config.yml describes:

streammux:
  batch-size: 1
  batched-push-timeout: 40000
  width: 1920
  height: 1080

primary-gie:
  plugin-type: 0
  config-file-path: dsdetect1_pgie_config.yml

dsdetect1_pgie_config.yml is the configuration file of nvinfer. For example, for a detection model from the DeepStream samples:

property:
  onnx-file: /opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/resnet18_trafficcamnet_pruned.onnx
  labelfile-path: /opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/labels.txt
  batch-size: 1
  network-mode: 2
  num-detected-classes: 4

A vision step uses it like this:

VisionInference {
    provider: "deepstream"
    configFile: "configs/deepstream/dsdetect1_config.yml"
}

Qualcomm IM SDK

The imsdk step provider runs TensorFlow Lite models with the Qualcomm Intelligent Multimedia SDK (IM SDK) on Qualcomm devices, by default on the NPU through the QNN delegate. It builds a GStreamer pipeline from the IM SDK elements: qtimlvconverter prepares the frames for the model, qtimltflite runs the model, and qtimlpostprocess interprets the model's output. The IM SDK GStreamer elements must be installed on the device.

qtimlvconverter takes the frames only in DMA buffers, which Qt Multimedia passes on only when it is built with -DFEATURE_gstreamer_qt_api=ON and runs with QT_MEDIA_BACKEND set to gstreamer. Otherwise the step logs a warning and never becomes ready.

IM SDK Configuration File

The configuration file is a JSON object with the following keys. Only model, labels and module are required; a key that is not set uses its default. Keys that the step provider does not know are ignored.

The values of mode, module and delegate are the names of the element's enumeration values, which gst-inspect-1.0 lists for the element.

A relative path in model, labels, or a delegatePath that contains a / is resolved against the directory that contains the application's executable, and an absolute path is used as it is.

If model, labels or module is missing, a key has the wrong type, or maxResults is not a positive integer, the step logs a warning and never becomes ready.

IM SDK Configuration Example

The following configuration file, saved as configs/imsdk/inference.json relative to the directory of the application's executable, runs a face detection model with the default delegate, on the NPU if the IM SDK provides the QNN delegate. The Simple Face Detection example ships it:

{
    "module": "qfd",
    "model": "models/Lightweight-Face-Detection_w8a8.tflite",
    "labels": "models/face_detection_plain.labels",
    "threshold": 70,
    "maxResults": 5
}

A file only needs the keys that differ from the defaults. The following one runs the same model on the CPU, with the default threshold and number of results:

{
    "module": "qfd",
    "model": "models/Lightweight-Face-Detection_w8a8.tflite",
    "labels": "models/face_detection_plain.labels",
    "delegate": "none"
}

A vision step uses it like this:

VisionInference {
    provider: "imsdk"
    configFile: "configs/imsdk/inference.json"
}