Adding a custom step

The built-in steps cover the common flow, but an application can add steps of its own to make the flow fit its needs, for example a step that converts the video frames to grayscale before they reach the model. A custom step is a C++ class made of three parts:

1. A QInferenceStep subclass, the front end that the application creates and puts in the flow. It sets the step's category and asks the factory for its implementation, as the built-in steps do.
2. A QInferenceBackendStep subclass, the implementation that does the work and declares the step's ports. For the GStreamer backend, derive from QGstInferenceBackendStep of the QtGstEdgeAi module, which holds the GStreamer bin of the step. This module is not documented, and its API may change during the Technical Preview.
3. A QInferenceBackendStepProvider that creates the implementation for the category.

The following step turns the frames into grayscale. The front end:

class GrayscaleStep : public QInferenceStep
{
public:
    explicit GrayscaleStep(QObject *parent = nullptr) : QInferenceStep(parent)
    {
        setCategory(QStringLiteral("grayscale"));
        auto *factory = QInferenceBackendFactory::instance();
        if (QInferenceBackendStep *impl = factory->createBackendStep(this, factory->defaultBackend())) {
            impl->initialize();
            setImplementation(impl);
        }
    }
};

The implementation builds a GStreamer bin of videoconvert and a capsfilter that asks for the GRAY8 format, and exposes the ends of the bin as a video input port and a video output port. This is how the backend implements every step: a step is a GStreamer bin of elements, and its ports wrap the pads at the ends of the bin. The flow adds the bin of each step to one GStreamer pipeline and connects the output port of a step to the input port of the next step by linking their pads. The bin has no name, so that GStreamer gives it a unique one: a flow can have more than one step of this kind, and bins with the same name cannot be in the same pipeline.

class GrayscaleBackendStep : public QGstInferenceBackendStep
{
public:
    using QGstInferenceBackendStep::QGstInferenceBackendStep;

    void initialize() override
    {
        QGstElementHandle bin{ gst_bin_new(nullptr) };
        QGstElementHandle convert{ gst_element_factory_make("videoconvert", "grayscale-convert") };
        QGstElementHandle filter{ gst_element_factory_make("capsfilter", "grayscale-caps") };
        if (!bin || !convert || !filter)
            return; // the step stays unready

        GstCaps *gray = gst_caps_new_simple("video/x-raw", "format", G_TYPE_STRING, "GRAY8", nullptr);
        g_object_set(filter.get(), "caps", gray, nullptr);
        gst_caps_unref(gray);
        gst_bin_add_many(GST_BIN(bin.get()), convert.get(), filter.get(), nullptr);
        if (!gst_element_link(convert.get(), filter.get()))
            return;

        QGstPadHandle in{ gst_element_get_static_pad(convert.get(), "sink") };
        QGstPadHandle out{ gst_element_get_static_pad(filter.get(), "src") };
        QGstPadHandle ghostIn{ gst_ghost_pad_new("step-in", in.get()) };
        QGstPadHandle ghostOut{ gst_ghost_pad_new("step-out", out.get()) };
        gst_element_add_pad(bin.get(), ghostIn.get());
        gst_element_add_pad(bin.get(), ghostOut.get());

        addInputPort(std::make_unique<QGstInferenceBackendDataPort>(
                std::move(ghostIn), QStringLiteral("video-in"), QInferenceDataPort::Type::Video));
        addOutputPort(std::make_unique<QGstInferenceBackendDataPort>(
                std::move(ghostOut), QStringLiteral("video-out"), QInferenceDataPort::Type::Video));

        setBin(std::move(bin)); // last: a step that returned early stays unready
        emit ready();
    }
};

The provider creates the implementation:

class GrayscaleProvider : public QInferenceBackendStepProvider
{
public:
    QInferenceBackendStep *createBackendStep(QInferenceStep *step, const QString &) override
    {
        return new GrayscaleBackendStep(step);
    }
};

The application registers the provider for the gstreamer backend before it creates the first GrayscaleStep, with grayscaleProvider an object of this class:

QInferenceBackendFactory::instance()->registerStepProvider(
        QStringLiteral("gstreamer"), QStringLiteral("grayscale"), &grayscaleProvider);

The factory is a singleton that lives as long as the process, and it does not take ownership of the provider. It keeps the pointer until the provider is unregistered, and every GrayscaleStep that is created in the meantime, in any flow, uses it. The provider has to stay alive while it is registered, so unregister it before you destroy it, for example before the application exits:

QInferenceBackendFactory::instance()->unregisterStepProvider(
        QStringLiteral("gstreamer"), QStringLiteral("grayscale"));

The factory does not synchronize access to its registrations. Register and unregister the provider, and create the steps, in the same thread, such as the main thread.

The step then goes in the flow like any other. The flow chains it by port type, so the frames from the video source go in and the grayscale frames come out, to the next step:

QInferenceFlow *flow = QInferenceFlow::fromSteps(
        { videoSource, new GrayscaleStep, detection, parser });

The step that follows must accept the format. A model step that expects RGB or a DMA buffer may not accept GRAY8, and converting the frames produces new buffers, so a step like this ends any zero-copy handling of the frames.

A custom step does not have to be registered by the application. A plugin that provides the grayscale category, with the JSON metadata described in Writing a Backend Plugin, makes the step available to every application on the device.