Qt Edge AI steps and ports

An inference flow is made of steps. A step does one job, such as reading camera frames or running a model, and passes its result on to the next step through ports.

Steps

Every step is a QInferenceStep in C++, or an InferenceStep in QML. A step has a category that tells the backend what kind of step to create. The following step types are provided:

A step is created in two parts. The step object you use is a thin front end; the work is done by a backend implementation, a QInferenceBackendStep, that a backend plugin creates for the step's category. A step is ready once its implementation has finished initializing. If no provider is available for the category, the step has no implementation and never becomes ready.

Ports

A port is a QInferenceDataPort with a name, a type, and a direction. Output ports send data and input ports receive it. A step can have several of each, and inputPorts and outputPorts list them.

Port Types

The type of a port, QInferenceDataPort::Type, tells what the port carries and how the data reaches the application; see the enum for the details of each type. Two ports can be connected only if they carry the same type, and a connection that cannot carry the data is rejected.

Zero-copy Video Frames

Qt Edge AI enables zero-copy operation for video frames, which means that unnecessary memory copying of video frames can be avoided to achieve better performance. Video ports link the backend pipeline's elements directly, so the frames are not copied from one step to the next, unless a step converts their format. With the GStreamer backend, the video source step also hands the buffer of a camera frame that Qt Multimedia already holds to the pipeline, rather than copying the frame.

Zero-copy operation still has requirements, and a flow that does not meet them works, but copies frames and uses more CPU and memory bandwidth:

• Qt Multimedia must be built with the Qt GStreamer API (-DFEATURE_gstreamer_qt_api=ON) and use the GStreamer media backend, by setting QT_MEDIA_BACKEND to gstreamer. Otherwise the video source step cannot pass the camera frame's GStreamer buffer on. It maps each frame and hands the mapped memory to the pipeline instead, which copies the frame when its data is in GPU memory, and decodes JPEG frames.
• The frames must be in a video format that the next step accepts. A step that needs another format converts the frames, which copies them. For example, the Edge Impulse step converts its input to RGB.

To keep a flow zero-copy, choose a camera format that the inference step takes directly. If the video source step cannot pass the camera frames' GStreamer buffers on, it logs a warning when it is created. The qt.edgeai.gst.videosource logging category reports how the video source step handles frames.

Ports of the Built-in Steps

The ports depend on the backend. A port name tells the type of data the port carries and its direction: <type>-in for an input port and <type>-out for an output port, where <type> is video, text, or json. The GStreamer backend provides the following:

Receiving Data

When a step has something to deliver to the application, it emits the dataAvailable() signal. In this release only the result parsers use it for delivering the inference results. The signal carries the data and the name of the output port it came from.

In QML, each step type forwards the signal of the C++ step it wraps, so you only add an onDataAvailable handler to the step, as in the following example:

InferenceResultParser {
    onDataAvailable: (data, portName) => console.log(data)
}

In C++, connect to dataAvailable() on the step.

How data moves between steps, such as the video frames, depends on the AI runtime, and the application does not receive it. What the results hold depends on the model and the backend that produced them.