Building and deploying the example

See specific steps relating to building and deploying Qt for Android Automotive examples.

Overview

The Surface Streaming example shows how to use a QAndroidRaaSApplication to render Qt Quick3D content offscreen within an Android Service. The service renders two separate outputs:

• One Qt Quick3D view rendered to an Android Surface provided by an activity (using an ObservableSurfaceView).
• Another view rendered to an ANativeWindow obtained from a QAndroidSurfaceStreamer, which encodes the content into an H.264 stream and serves it via an RTSP/RTP connection.

The RTSP server listens on a TCP port (8554 in this example). After a successful RTSP negotiation, the video player client listens on two UDP ports used for RTP and RTCP. The network connection must provide the necessary visibility of both the TCP and UDP ports between the Android device and the client machine. This is usually not possible with Android emulators (e.g., QEMU-based emulators) because they have limited virtual networking capabilities.

This example demonstrates how to combine Qt's offscreen rendering capabilities with Android's native surface management and media encoding APIs.

How It Works

The architecture consists of:

• Activity and Surface: The Android activity manages an ObservableSurfaceView, a small helper that simplifies surface lifecycle handling. Its Android Surface is passed to the service through an AIDL interface, allowing Qt to render into it.
• Qt Service: The service runs a QAndroidRaaSApplication with a QAndroidSurfaceRenderEngine. It manages multiple offscreen render targets, one bound to the activity’s surface, another to an ANativeWindow created by the QAndroidSurfaceStreamer. A QtRaaSApplication Java instance in the RenderingService will start and communicate with the Qt Quick application.
• Streaming Output: The streamer uses Android MediaCodec to set up an encoder. QAndroidSurfaceStreamer encode frames rendered on its surface into H.264 format, then transmits them over an RTSP/RTP connection.
• QML Scene: Both render targets share the same QML engine but can display independent View3D components or cameras, representing different viewpoints of the same 3D content.

C++ Entry Point

The main.cpp file creates a QAndroidRaaSApplication and initializes both the QAndroidSurfaceRenderEngine and the QAndroidSurfaceStreamer. The streamer's surface is used to encode the second Qt Quick3D view for network streaming.

QAndroidRaaSApplication app(argc, argv);

QAndroidSurfaceRenderEngine *renderEngine = new QAndroidSurfaceRenderEngine(&app);

Create and start the streamer.

QAndroidMediaFormat mediaFormat;
mediaFormat.setCodecType(QAndroidMediaFormat::CodecType::H264);
mediaFormat.setSize(QSize(1080, 720));
mediaFormat.setBitrate(8000000);
mediaFormat.setFramerate(60.0f);
mediaFormat.setIFrameInterval(1);
mediaFormat.setOperatingRate(60.0f);
mediaFormat.setAvcProfile(AVCProfileHigh);
mediaFormat.setAvcLevel(AVC_LEVEL_4_2);

QAndroidSurfaceStreamer *streamer = new QAndroidSurfaceStreamer(mediaFormat, &app);

QObject::connect(
    streamer, &QAndroidSurfaceStreamer::error,
    [](QAndroidSurfaceStreamer::Error error, const QString &details) {
      qWarning() << "Stream error:"
                 << QAndroidSurfaceStreamer::errorString(error) << details;
    });

QHostAddress host = QHostAddress::Any; // Listen on all interfaces
quint16 port = 8554;                   // RTSP port

if (!streamer->start(host, port)) {
    qFatal("Failed to start surface streamer");
    return -1;
}

Pass the ANativeWindow instance to QAndroidSurfaceRenderEngine to start rendering the Quick item on it.

QObject::connect(
    renderEngine, &QAndroidSurfaceRenderEngine::objectCreated, renderEngine,
    [streamer, renderEngine]() {
      ANativeWindow *surface = streamer->surface();
      if (surface)
        renderEngine->setNativeWindowForItem(surface, "orbitingCameraView");
      else
        qWarning(
            "The surface is invalid. Failed to start surface streamer!");
    });

renderEngine->load(QLatin1String("qrc:/qt/qml/streaming_module/Main.qml"));

QML Scene

The QML scene is defined in Main.qml and uses an Item as its root object. Only direct children of this root Item can be referenced by their IDs from the Android side. Each surface registered through the AIDL interface is associated with one of these direct child items.

Item {
    id: root

    Item {
        id: stationaryCameraView

        View3D {
            id: stationaryView3d
            anchors.fill: parent
            importScene: standAloneScene
        }
    }

    Item {
        id: orbitingCameraView

        View3D {
            id: orbitingView3d
            anchors.fill: parent
            importScene: standAloneScene
            camera: cameraPerspectiveOrb
        }
    }
}

Android Integration

The Android service is implemented in Java and defined in the manifest as a service. It instantiates a QtRaaSApplication that loads the Qt runtime and starts the Qt Quick application. It implements the AIDL-defined IRenderingService interface to bridge the bound client(s) to the rendering engine.

interface IRenderingService {

    void setSurface(in Surface surface, String itemId);
    void unsetSurface(String itemId);
    void motionEvent(in MotionEvent event, String itemId);
}

The activity creates its own ObservableSurfaceView instances and binds to the service. When the surface becomes available, the activity passes it together with a QML item ID to the service. When the surface becomes invalid or is destroyed, it notifies the service to unset the surface for that particular item, so that the QtRaaSApplication stops rendering that item.

Video Stream Playback

After running the example, you will be able to play back the stream in any video player that supports the RTSP protocol by using a URL like the one shown below. You need to replace ip.of.android with the IP address of the Android device.

rtsp://ip.of.android:8554/

Here is an example a command line usage of mpv player:

mpv --rtsp-transport=udp --profile=low-latency --untimed --vd-lavc-threads=1 --cache=no -hwdec=auto rtsp://ip.of.android:8554