Qt Edge AI backends

Qt Edge AI does not run inference itself. The steps of a flow describe what to do, and a backend does it. The module ships one backend, gstreamer. A backend gets the parts that do the work from backend plugins on the target, so the same application code runs on any platform that has the backend plugins for it.

Flow Providers and Step Providers

A backend plugin provides one or both of the following for a backend:

• A flow provider, a QInferenceFlowBackendProvider, creates the QInferenceBackendFlow that owns the pipeline and runs it.
• A step provider, a QInferenceBackendStepProvider, creates the QInferenceBackendStep that does the work for a step of a given category.

QInferenceBackendFactory loads the backend plugins and hands out what they provide. It reads plugins from the edgeai subdirectory of the application's library paths.

Selecting the Backend

The backend is selected by name. The QT_INFERENCE_FLOW_BACKEND environment variable sets the name; when it is unset or empty, the default, gstreamer, is used. The name is read once, when the factory is first used, so every flow and step in the process uses the same backend, and changing the variable later has no effect.

If no backend plugin provides the selected backend, flows cannot run, and a warning names the backends that are available.

Selecting a Step Provider

For each step, the factory looks for a step provider within the selected backend. A VisionInference step uses the provider that its provider property names. The other steps, such as InferenceVideoSource and InferenceResultParser, use the provider registered for their category.

If no provider is found, including when a VisionInference step gives an empty or unknown provider name, no implementation is created, the step never becomes ready, and a warning is logged.

A VisionInference step must name its provider, because its configuration file is specific to the provider that it names. To choose a provider without hard-coding its name, call QInferenceBackendFactory::defaultStepProvider(). It returns the name of the first provider, in alphabetical order, of the selected backend that creates steps of a category, vision by default, or an empty string if there is none.

The GStreamer Backend

The gstreamer backend gets its flow provider and step providers from the following backend plugins, depending on the target platform. Each plugin has a provider name and lists the step categories that it provides. The step type of each category is listed in Qt Edge AI steps and ports:

The gstflowbackend plugin also provides the flow provider of the gstreamer backend. The other plugins provide only step providers.

A VisionInference step selects its step provider by the provider name, with VisionInference::provider, and the step's configFile is written for that step provider.

For what each plugin needs on the target, see the overview and Qt Edge AI prerequisites. Each provider reads its settings from the step's configuration file; see Qt Edge AI Configuration Files.

Writing a Backend Plugin

To provide a flow provider or step providers for a backend, implement a Qt plugin with one or both of the provider interfaces, QInferenceFlowBackendProvider and QInferenceBackendStepProvider, and derive the pipeline and step classes from QInferenceBackendFlow and QInferenceBackendStep. The plugin's JSON metadata tells the factory what it provides:

{ "backend": "mybackend",
  "name": "myprovider",
  "provided_categories": ["vision"]
}

A flow provider plugin needs only the backend key. A step provider plugin needs all three keys: one that lacks a key, or has an empty value, is skipped. If two step provider plugins of a backend have the same name, the one that is loaded first is used, and the other is skipped with a warning.