QStringTokenizer Class

template <typename Haystack, typename Needle> class QStringTokenizer

The QStringTokenizer class splits strings into tokens along given separators. More...

Header: #include <QStringTokenizer>
CMake: find_package(Qt6 COMPONENT Core)
target_link_libraries(mytarget PUBLIC Qt::Core)
qmake: QT += core
Since: Qt 6.0
Inherits: QtPrivate::Tok::HaystackPinning (private), QtPrivate::Tok::NeedlePinning (private), and

This class was introduced in Qt 6.0.

Note: All functions in this class are reentrant.

Public Types

(alias) value_type

Detailed Description

Splits a string into substrings wherever a given separator occurs, returning a (lazily constructed) list of those strings. If the separator does not match anywhere in the string, produces a single-element list containing this string. If the separator is empty, QStringTokenizer produces an empty string, followed by each of the string's characters, followed by another empty string. The two enumerations Qt::SplitBehavior and Qt::CaseSensitivity further control the output.

QStringTokenizer drives QStringView::tokenize(), but, at least with a recent compiler, you can use it directly, too:

for (auto it : QStringTokenizer{string, separator})
    use(*it);

Note: You should never, ever, name the template arguments of a QStringTokenizer explicitly. If you can use C++17 Class Template Argument Deduction (CTAD), you may write QStringTokenizer{string, separator} (without template arguments). If you can't use C++17 CTAD, you must use the QStringView::split() or QLatin1String::split() member functions and store the return value only in auto variables:

auto result = string.split(sep);

This is because the template arguments of QStringTokenizer have a very subtle dependency on the specific string and separator types from with which they are constructed, and they don't usually correspond to the actual types passed.

Lazy Sequences

QStringTokenizer acts as a so-called lazy sequence, that is, each next element is only computed once you ask for it. Lazy sequences have the advantage that they only require O(1) memory. They have the disadvantage that, at least for QStringTokenizer, they only allow forward, not random-access, iteration.

The intended use-case is that you just plug it into a ranged for loop:

for (auto it : QStringTokenizer{string, separator})
    use(*it);

or a C++20 ranged algorithm:

std::ranges::for_each(QStringTokenizer{string, separator},
                      [] (auto token) { use(token); });

End Sentinel

The QStringTokenizer iterators cannot be used with classical STL algorithms, because those require iterator/iterator pairs, while QStringTokenizer uses sentinels. That is, it uses a different type, QStringTokenizer::sentinel, to mark the end of the range. This improves performance, because the sentinel is an empty type. Sentinels are supported from C++17 (for ranged for) and C++20 (for algorithms using the new ranges library).

Temporaries

QStringTokenizer is very carefully designed to avoid dangling references. If you construct a tokenizer from a temporary string (an rvalue), that argument is stored internally, so the referenced data isn't deleted before it is tokenized:

auto tok = QStringTokenizer{widget.text(), u','};
// return value of `widget.text()` is destroyed, but content was moved into `tok`
for (auto e : tok)
   use(e);

If you pass named objects (lvalues), then QStringTokenizer does not store a copy. You are responsible to keep the named object's data around for longer than the tokenizer operates on it:

auto text = widget.text();
auto tok = QStringTokenizer{text, u','};
text.clear();      // destroy content of `text`
for (auto e : tok) // ERROR: `tok` references deleted data!
    use(e);

See also QStringView::split(), QLatin1String::split(), and QRegularExpression.

Member Type Documentation

[alias] QStringTokenizer::value_type

This is a type alias for typename Base::value_type.

Alias for const QStringView or const QLatin1String, depending on the tokenizer's Haystack template argument.

© 2020 The Qt Company Ltd. Documentation contributions included herein are the copyrights of their respective owners. The documentation provided herein is licensed under the terms of the GNU Free Documentation License version 1.3 as published by the Free Software Foundation. Qt and respective logos are trademarks of The Qt Company Ltd. in Finland and/or other countries worldwide. All other trademarks are property of their respective owners.