Skip to content

lexilla design decisions

The decisions behind lexilla's technical approach. The project's overall purpose -- including why lexilla is a separate package from pyside6-scintilla -- is in the mission. Paths are relative to packages/lexilla/ unless they start at the repo root.

Binding technology: nanobind

Lexilla's public surface is plain C++ (CreateLexer, a handful of free functions, and the ILexer5 abstract interface) with no Qt types involved. shiboken6 -- used by pyside6-scintilla -- is built around Qt's object model and would drag in the Qt toolchain and the PySide6 version pin for no benefit. nanobind has no Qt/PySide6 dependency and fits a small set of free functions plus one abstract interface class. See How the packages are built for the full comparison.

Cross-binding integration: raw pointer, plus convenience glue

CreateLexer returns an ILexer5*. The Scintilla side expects that same pointer value via SCI_SETILEXER (an sptr_t message parameter). The default, zero-coupling approach is to expose the lexer's pointer as a plain Python int (uintptr_t) — callers pass it to whatever binding's send/message API they're using themselves:

editor.send(SCI_SETILEXER, 0, lexer.pointer)

For ergonomics, convenience glue that knows about pyside6-scintilla's API directly (e.g. a set_lexer(editor, lexer) helper) is also provided — this is the only place the two packages are allowed to depend on each other.

Scope: minimal first, full API later

The first usable version covers CreateLexer(name), lexer discovery (GetLexerCount, GetLexerName), and the core ILexer4/ILexer5 property and word-list methods (PropertyGet/PropertySet, WordListSet, and the introspection methods used to back them). Lex and Fold are deferred: both take an IDocument*, which only a Scintilla editor instance provides in normal use (Scintilla calls them itself once a lexer is wired up via SCI_SETILEXER) — binding them as Python-callable would mean also binding IDocument as a trampoline class Python code can implement, a much bigger surface for unclear benefit. A follow-up should investigate whether something like Pygments or tree-sitter could back an IDocument implementation usefully, or whether exposing IDocument at all is worth it — see borco/lexilla-py#6. The deprecated CreateLexerLibrary path and the full property/word-list introspection API are also deferred.

No bare ints/strings for "magic" values: typed enums for property types, language identifiers, and lexer names

Lexer.property_type(), Lexer.identifier, and create_lexer(name) all deal in values that are really enumerations dressed up as a bare int or str: Scintilla's SC_TYPE_* (boolean/integer/string), Scintilla's SCLEX_* language identifiers (declared in Lexilla's own vendored include/SciLexer.h), and Lexilla's ~139-entry lexer-name catalogue respectively. Left as bare values there's no IDE autocomplete or hover documentation, and a typo silently does the wrong thing instead of failing loudly. The fix, applied consistently to all three:

  • Both PropertyType and LanguageIdentifier are registered with nb::is_arithmetic(), so e.g. lexer.identifier == 3 works directly against the raw Scintilla constant, without requiring .value or an explicit int() cast — they still wrap Scintilla's own SC_TYPE_*/ SCLEX_* integers, so comparing against the documented numeric constant should just work.
  • PropertyType (SC_TYPE_*) is a small, hand-written nb::enum_ in _binding.cpp — only 3 values, no codegen needed.
  • LanguageIdentifier (SCLEX_*, ~142 values) is generated, not hand-typed, by tools/generate_language_enums.py (in the package's tools/, like pyside6-scintilla's generators), mirroring Lexilla's own convention for its large generated lists (src/lexilla_vendor/src/Lexilla.cxx's //++Autogenerated -- run scripts/LexillaGen.py to regenerate markers): a one-off script, run manually, output spliced into _binding.cpp and checked into git — not wired into the CMake/uv sync build, so Python stays out of the C++ compile step beyond what scikit-build-core already needs. The generator reuses the vendored, unmodified src/lexilla_vendor/scripts/LexillaData.py (Lexilla's own lexer-catalogue parser) rather than re-deriving its parsing logic. Four SCLEX_* macros have no associated lexer (CONTAINER, NULL, AUTOMATIC, and XCODE — confirmed via grep that none has a LexerModule); all four are kept in the enum rather than excluded, since dropping any would make the enum lossy and risk a runtime nanobind error if GetIdentifier() ever legitimately returns one. SCLEX_NULL's enum member is named NullValue in C++ (it would otherwise collide with the <cstddef> NULL macro, which the preprocessor rewrites before the compiler ever sees an identifier); its Python-facing name is still NULL, via the nb::enum_<>::value() registration string. Every generated value gets its own short docstring, derived from the same name data already being parsed, so the per-value documentation bar is the same for 3 values or 142.
  • Language (the lexer-name strings) is a Python enum.StrEnum in a generated src/lexilla/_languages.py — no C++ change needed, since nanobind's const char* parameter for create_lexer accepts any str subclass, and StrEnum members are str instances (create_lexer(Language.CPP) works with zero binding changes). There's no natural C++ representation for opaque name strings (unlike SCLEX_*, a real Scintilla enum), so this stays pure Python. The one catalogue name unsafe as a Python identifier as-is, "PL/M", becomes member name PLM (not PL_M) — matching every other C identifier for that lexer across the vendored source (SCLEX_PLM, SCE_PLM_*, lmPLM), none of which use an underscore for the dropped slash.

Checked https://www.scintilla.org/LexillaDoc.html for official wording to reuse in these docstrings: it documents only the library-level functions (CreateLexer, GetLexerCount, GetLexerName, etc.), nothing at the SC_TYPE_*/SCLEX_*/ILexer4/ILexer5 granularity — so all docstrings here are original, not adapted from upstream text.

Because the generated enums are derived from whatever Lexilla version is currently vendored, they go stale silently at compile time (a static_cast always succeeds) but loudly at the Python boundary (nanobind raises if GetIdentifier() ever returns a value with no matching enum member) — see auditing.md's re-vendoring checklist for the regeneration step this requires.

ILexer5's declaration: a copy of Scintilla's interface headers, not all of Scintilla

ILexer5 (along with ILexer4, IDocument, and the Sci_Position/ Sci_PositionU types) is declared in Scintilla's ILexer.h/Sci_Position.h, not in Lexilla's own tarball -- Lexilla.h assumes the caller has already included ILexer.h. The individual lexer implementations also need a handful of fold-level flag constants from Scintilla's Scintilla.h (the per-language SCE_* style constants already live in Lexilla's own vendored include/SciLexer.h). Depending on all of Scintilla for these would reintroduce the Scintilla/Qt coupling lexilla deliberately avoids, so only those headers are kept, unmodified, under src/scintilla_interface/include/.

Since the move into the monorepo they are a byte-identical copy of the Scintilla release pyside6-scintilla vendors, kept in sync by tools/sync_scintilla_headers.py (checked in CI), so the two packages can't disagree about ILexer5's layout within a commit. lexilla keeps a copy at all -- rather than including pyside6-scintilla's tree -- because its sdist must build on its own. (Before the move, lexilla-py vendored them from the Scintilla release matching its Lexilla version, 5.5.0.) See Verifying vendored Lexilla.

Naming: lexilla package, src/lexilla_vendor/ for vendored source

The PyPI package and Python import name are both plain lexilla. Within the package, the vendored upstream source lives at src/lexilla_vendor/, not src/lexilla/: Lexilla's own source directory and the Python package directory would otherwise have the same name, colliding on case-insensitive filesystems (Windows, default macOS). pyside6-scintilla has no such clash (src/scintilla/ next to src/pyside6_scintilla/).

(The standalone repo was named lexilla-py rather than lexilla for the same reason, one level up; since issue #19 the package lives in the scintilla-py monorepo as packages/lexilla/.)

Versioning

The same scheme as pyside6-scintilla: <Lexilla version>.<binding revision> (e.g. Lexilla 5.5.0 → package 5.5.0.0). The binding revision increments for releases of this package that don't correspond to a new Lexilla version, and resets to 0 when Lexilla itself releases a new version.

Wheel builds: one cp312-abi3 wheel per platform

nanobind's STABLE_ABI mode compiles against Python's limited API, so one build works on every later Python without a rebuild -- but nanobind hard-requires CPython 3.12 for it (Py_LIMITED_API fixed to 0x030c0000). lexilla therefore requires Python 3.12 and ships a single cp312-abi3 wheel per platform, like pyside6-scintilla's single cp311-abi3 wheel (shiboken6's FORCE_LIMITED_API can go down to 3.11).

Before the monorepo, lexilla-py kept Python 3.11 with a second, version-specific cp311 wheel per platform, doubling the wheel count for one Python version; that was dropped with the move (issue #19). lexilla 5.5.0.1 remains available for 3.11.