From 6be9ba5d53e18e9b8eb565dfd6f211802fe5747e Mon Sep 17 00:00:00 2001 From: Marco Casaroli Date: Thu, 30 Jul 2026 14:36:59 +0200 Subject: [PATCH] Documentation: Add a developer guide for FDPIC modules. Documentation/components/fdpic.rst covers what someone building or loading an FDPIC module needs: how the format works and what the loader does with it, how it differs from NXFLAT and from the ELF loader's own PIC/XIP path, what the target and toolchain have to provide, how to build a module, a shared library and a C++ module, and what a firmware entry point must do to accept a module callback. The comparison is the part worth stating plainly, because two of the three properties are shared. All three formats run position-independent code from flash with no MMU and give several instances of one module a shared .text with private .data. NXFLAT needs its own tools and cannot export symbols, so it has no shared libraries. ELF PIC needs no extra tools at all, but one base register per task means a shared object is loaded as a single allocation -- its text cannot stay in flash -- and there is no DT_NEEDED walk. FDPIC costs an arm-uclinuxfdpiceabi linker and buys a pointer that carries its own data base, which is what makes shared libraries and callbacks on a thread the module never created possible. The reference material at the end is the part that is expensive to rediscover: the FDPIC marker being in EI_OSABI rather than e_flags, the three relocation types that survive a static link, why both relocation tables are bound eagerly, why .rofixup is skipped, and the two binfmt contract details that fail as a panic or a bare -EINVAL. Also adds the fdpic and reject sections to the xipfs test suite document, which apps/testing/fs/xipfs grows in the matching apps change. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Marco Casaroli --- .../applications/testing/xipfs/index.rst | 14 + Documentation/components/fdpic.rst | 490 ++++++++++++++++++ Documentation/components/index.rst | 1 + 3 files changed, 505 insertions(+) create mode 100644 Documentation/components/fdpic.rst diff --git a/Documentation/applications/testing/xipfs/index.rst b/Documentation/applications/testing/xipfs/index.rst index 7b68753c03f..3511a3525bb 100644 --- a/Documentation/applications/testing/xipfs/index.rst +++ b/Documentation/applications/testing/xipfs/index.rst @@ -79,6 +79,20 @@ Sections power loss mid-program leaves behind. These need ``CONFIG_FS_XIPFS_FAULT_INJECT``. +``fdpic`` + That a module staged into the volume loads and runs from flash: global + constructors in dependency order, per-instance data, manufactured + descriptors, both relocation tables, the GOT fallback for a library with no + PLT, and every firmware entry point that has to resolve a module callback. + Each module reports its own invariants through its exit status. Needs + :doc:`CONFIG_FDPIC `. + +``reject`` + That the loader refuses a module it cannot honour rather than loading + something broken -- a mutated header, an absent dependency, a missing + import, more ``DT_NEEDED`` entries than the walk will follow. Needs + ``CONFIG_FDPIC``. + Configuration ============= diff --git a/Documentation/components/fdpic.rst b/Documentation/components/fdpic.rst new file mode 100644 index 00000000000..aab37b1860b --- /dev/null +++ b/Documentation/components/fdpic.rst @@ -0,0 +1,490 @@ +============= +FDPIC Modules +============= + +FDPIC modules are ELF shared objects that execute **in place** out of +memory-mapped flash. The read-only segment is mapped where it already sits on +the media and never copied to RAM; only the writable segment is copied, once +per running instance. Shared libraries work, and so does C++ with global +constructors. + +The loader is ``binfmt/fdpic.c``, enabled by ``CONFIG_FDPIC``. Modules are +built out of tree with the tooling in ``tools/fdpic``. + +How it works +============ + +An FDPIC object has two ``PT_LOAD`` segments that may be placed independently +of one another. Nothing in the code depends on the distance between them, +which is what lets the loader leave the read-only segment in flash and put the +writable one wherever RAM is available. + +Code reaches its own data through a base register -- **r9** on ARM -- holding +the address of that object's GOT. A function pointer is therefore not enough +to call a function: the callee needs its data base too. FDPIC represents a +function pointer as a two-word **descriptor**: + +=========== ============================================================== +Word Contents +=========== ============================================================== +``entry`` code address, including its Thumb bit +``got`` data base to install in r9 before branching +=========== ============================================================== + +Building those descriptors is most of what the loader does. Because each one +names its own base, two objects can be in use at once with separate writable +segments, and a pointer handed to the firmware carries everything needed to +call back into the module later. + +A module links against nothing. libc and everything else are undefined +imports, resolved at load time against other loaded objects first and the +firmware's exported symbol table second. + +Load sequence +------------- + +``binfmt/fdpic.c``, in order: + +#. Parse the ELF and program headers and check the FDPIC marker. Anything + else is declined quietly so another binfmt handler can try. + +#. Read ``PT_DYNAMIC`` from the file before either segment is placed: the + relocation count bounds the descriptor pool the writable allocation needs. + +#. Pin the read-only segment on the media with ``XIPFSIOC_PIN``. The returned + pointer *is* the execution address. The pin stops the filesystem + relocating code that is executing. + +#. Allocate the writable segment, copy ``.data`` from flash, zero ``.bss``. + +#. Walk ``DT_NEEDED`` and load each shared library the same way, depth capped + by ``FDPIC_MAX_DEPTH``. + +#. Apply relocations for every object, from both ``DT_REL`` and ``DT_JMPREL``. + +#. Run ``DT_INIT_ARRAY`` for every object, dependencies first. + +#. Hand the scheduler a ``struct dspace_s`` holding the module's GOT, which + ``up_initial_state()`` installs into r9. + +On unload the loader runs ``DT_FINI_ARRAY`` -- module first, then its +libraries -- while their writable segments still exist, drops the pin, and +frees the allocation. The pin release does not depend on the module calling +``munmap``: ``mm_map_destroy()`` invokes each mapping's callback during group +release, so a module that faults or is killed still drops its pin. + +Differences from NXFLAT and ELF PIC +=================================== + +All three run position-independent code from flash on a target with no MMU, +and all three give several instances of one module a shared ``.text`` with +private ``.data``. They differ in what a *pointer* can express and in what +the toolchain has to provide. + +========================= ============== ============== ============ +Property NXFLAT ELF PIC (XIP) FDPIC +========================= ============== ============== ============ +Format NuttX only ELF ELF +Extra build tools yes none linker only +Base register r10 r10 r9 +Data base per task task object +Shared libraries no no yes +Foreign-thread callback no no yes +``.rodata`` in flash linker script automatic automatic +Cores Cortex-M3/M4 CONFIG_PIC Thumb-2 +========================= ============== ============== ============ + +**NXFLAT** is a NuttX-specific format. A module imports symbols from the base +firmware but cannot export any, so shared libraries are not possible, and the +build needs ``mknxflat`` to generate a thunk, ``ldnxflat`` to link, and one of +the ``binfmt/libnxflat`` linker scripts to place the sections -- with which +script depending on how the compiler addresses ``.rodata``. + +**ELF PIC** needs no extra tools at all. With ``CONFIG_PIC`` the loader +allocates the writable sections separately and, when the filesystem answers +``FIOC_XIPBASE``, leaves the read-only ones on the media +(``libs/libc/elf/elf_load.c``); ``mps3-an547:picostest`` runs ``ostest`` that +way. Two limits follow from having one base register per task. A shared +object is loaded as a single allocation, because "the relative positions +between text and data must be maintained due to references to the GOT", so its +text cannot stay in flash. And ``tcb->dspace`` is installed into ``REG_PIC`` +once per task, so every object in a task shares one data base and there is no +``DT_NEEDED`` walk to load a second one. + +**FDPIC** pays for its descriptors with an ``arm-uclinuxfdpiceabi`` linker, and +gets back the two things a single register cannot express: + +* **A module can be called back on a thread it never created.** A task or + pthread the module starts inherits its data base through + ``nxtask_dup_dspace()``, so a register is enough there. A work-queue worker + is not: it was created at boot and carries no module base. A descriptor + supplies one, which is how ``SIGEV_THREAD`` notifications from + ``timer_create()`` and ``mq_notify()`` reach module code. + +* **Two objects hold distinct data bases at once.** A module and the + libraries it imports each have their own writable segment and still call one + another, so two instances of a module share a library's text while each gets + its own copy of the library's data. + +Requirements +============ + +**An ARM Thumb-2 core.** The boundary is the instruction set, not the core +profile: GCC rejects FDPIC in Thumb-1 mode. + +========================= ========================== ===== +Core Architecture FDPIC +========================= ========================== ===== +Cortex-M3 / M4 / M7 ARMv7-M / ARMv7E-M yes +Cortex-M33 ARMv8-M Mainline yes +Cortex-M0 / M0+ / M23 ARMv6-M / ARMv8-M Baseline no +========================= ========================== ===== + +If the toolchain's multilib list includes Thumb-1 variants, the Thumb-1 error +appears even when the target is a Thumb-2 core. Configure the multilib set to +exclude them. + +RISC-V has no FDPIC ABI -- the psABI addendum is an unmerged RFC and no +``EI_OSABI`` value is assigned -- so a RISC-V target cannot use this loader. + +**Flash that is memory mapped and executable**, exposed by a filesystem that +answers ``BIOC_XIPBASE``: :doc:`XIPFS ` or ROMFS. + +**The linker.** A stock ``arm-none-eabi`` GCC compiles correct FDPIC objects +for both C and C++. What it cannot do is link them: ``arm-none-eabi-ld`` is +configured with the ``armelf`` emulation alone, and rather than failing it +marks the output ``UNIX - System V``, which the loader refuses at run time. +Only ``arm-uclinuxfdpiceabi`` carries ``armelf_linux_fdpiceabi``. + +No distribution packages it, so build it -- binutils alone, about a minute:: + + tools/fdpic/build-binutils.sh ~/fdpic + export PATH=~/fdpic/toolchain/bin:$PATH + +An FDPIC GCC is not needed and nothing here uses one. + +GNU ld's ARM FDPIC support has open defects, notably a failure on +``R_ARM_GOTOFFFUNCDESC`` against a hidden function symbol (binutils PR31407, +PR31408, PR31409). + +Configuration +------------- + +``CONFIG_FDPIC`` selects ``BINFMT_LOADABLE`` and ``PIC``, and depends on +``CONFIG_ARCH_ARM``. A working configuration also needs a symbol table for +modules to import from and a filesystem that can expose its media:: + + CONFIG_FDPIC=y + CONFIG_LIBC_EXECFUNCS=y + CONFIG_EXECFUNCS_HAVE_SYMTAB=y + CONFIG_EXECFUNCS_SYSTEM_SYMTAB=y + CONFIG_FS_XIPFS=y + +The options under ``if FDPIC``: + +===================== =========== ===================================== +Option Default Meaning +===================== =========== ===================================== +``FDPIC_LIBPATH`` /mnt/xipfs Where ``DT_NEEDED`` objects are found +``FDPIC_STACKSIZE`` 2048 Stack for a task made from a module +``FDPIC_ALLOW_COPY`` n Copy text to RAM when the media + cannot be mapped +===================== =========== ===================================== + +``FDPIC_ALLOW_COPY`` is for bring-up. Left off, a module that cannot execute +in place fails to load instead of silently costing the RAM this mechanism +exists to save. + +**The base firmware must reserve r9.** It is not enough for the module to be +well behaved: a firmware routine calling back into module code arrives with +the module's GOT in r9 only if the compiler was never free to allocate that +register elsewhere. ``arch/arm/src/common/Toolchain.defs`` adds +``--fixed-r9`` to ``ARCHCPUFLAGS`` under ``CONFIG_FDPIC`` -- not to ``CFLAGS``, +which most board ``Make.defs`` files assign with ``:=`` afterwards and would +discard. A lost flag is silent, so check it arrived:: + + make V=1 2>&1 | grep -m1 fixed-r9 + +.. _fdpic_tooling: + +Building a module +================= + +A whole module is three lines of makefile beside the source. Taking +``apps/examples/fdpicxip/modules/qsorter.c`` as the source:: + + MODULE = qsorter + SRCS = qsorter.c + + include /path/to/nuttx/tools/fdpic/nuttx-fdpic.mk + +:: + + make NUTTX_DIR=/path/to/nuttx + CC qsorter.c + LD qsorter.fdpic + OK qsorter.fdpic: FDPIC, entry 0x2a1, 4 imports resolved + +``NUTTX_DIR`` must be a configured, **built** tree: the compile needs its +headers and the verification step needs ``libs/libc/exec_symtab.c``, which the +build generates. + +C++ sources go in ``CXXSRCS``. ``ARM_TOOLCHAIN=`` and ``FDPIC_TOOLCHAIN=`` +override the two toolchain prefixes, and ``BINDNOW=`` empties the default +``-z now``. ``make`` will run the verification step alone against a stale +``.fdpic`` and report ``OK``, so ``make clean`` when changing toolchains. + +The commands it runs are:: + + arm-none-eabi-gcc -mcpu=cortex-m33 -mthumb -mfdpic -fPIC -Os \ + -fno-builtin -I$NUTTX/include -D__NuttX__ -c qsorter.c -o qsorter.o + + arm-uclinuxfdpiceabi-ld -m armelf_linux_fdpiceabi -shared -z now \ + -e main -o qsorter.fdpic qsorter.o + +Four flags carry weight: + +* **-mfdpic** is stated rather than assumed, so a mis-set toolchain fails + loudly instead of producing a plain ELF the loader refuses. + +* **-fPIC** is not implied by ``-mfdpic`` on the bare-metal target, and + without it the link emits ``TEXTREL``. Text relocations cannot work + against text executed from read-only flash. + +* **-shared** preserves the ``R_ARM_FUNCDESC_VALUE`` relocations for imported + symbols. A PIE link with ``--unresolved-symbols=ignore-all`` appears to + work but degrades every import to ``R_ARM_NONE``, and the module branches to + zero on its first call into the firmware. + +* **-m armelf_linux_fdpiceabi** is required: this linker supports four + emulations and will not guess. + +Verifying imports +----------------- + +A module links with ``-shared``, so undefined symbols are permitted: importing +something the firmware does not export links cleanly and fails on the target +as a bare ``-ENOENT`` naming no symbol. ``fdpic-verify.sh`` runs as part of the +default target and turns that into a build failure:: + + FAIL imports the firmware does not export: + this_symbol_does_not_exist + +The export list comes from ``CONFIG_EXECFUNCS_SYSTEM_SYMTAB``, which generates +``libs/libc/exec_symtab.c``. Most entries there sit behind +``#if defined(CONFIG_...)`` guards, so the file has to go through the +preprocessor before its names mean anything -- read textually it offers every +symbol that *could* be exported rather than the ones that were:: + + tools/fdpic/nuttx-exports.sh /path/to/nuttx # the list + make exports NUTTX_DIR=/path/to/nuttx # the count + +Shared libraries +---------------- + +A library is built the same way with a soname and no entry point, and the +module names it on the link line:: + + arm-uclinuxfdpiceabi-ld ... -shared -soname libcounter.so \ + -e 0 -o libcounter.so libcounter.o + + arm-uclinuxfdpiceabi-ld ... -shared -e main \ + -o user.fdpic user.o libcounter.so + +At run time the library must be present at ``CONFIG_FDPIC_LIBPATH`` under the +name in the module's ``DT_NEEDED`` entry. Each object gets its own GOT and its +own writable data, so two modules using one library see independent library +state while sharing one copy of its code in flash. + +A leaf library -- one that calls nothing outside itself -- has no +``DT_PLTGOT``, because with no external calls there is no PLT. It still has a +GOT, and the loader falls back to ``PT_DYNAMIC`` vaddr plus memsz. + +C++ modules +----------- + +Three compile flags are not optional: + +* ``-fno-use-cxa-atexit``. The default registers each static object's + destructor with ``__cxa_atexit(dtor, obj, &__dso_handle)``, and + ``__dso_handle`` comes from ``crtbegin``, which ``-nostartfiles`` excludes; + the link fails outright. Turning it off also puts destructors in + ``.fini_array``, which is what the loader walks on unload. + +* ``-fno-exceptions -fno-rtti``. Both need ``libsupc++``, which a module + linking ``-nostdlib`` cannot reach. + +C++ is compiled with ``arm-none-eabi-g++`` and linked by the FDPIC linker, +exactly as C is. + +``DT_INIT_ARRAY`` runs after relocations and before the module is entered, +``DT_FINI_ARRAY`` on unload, both per object and in dependency order. +Constructors run in the task that called the loader, not the task that will +run the module, so a constructor sees the loading task's identity. + +Getting this wrong is quiet: a loader that ignores ``DT_INIT_ARRAY`` still +loads a C++ module, resolves every symbol and runs it, with every global left +zero. + +Calling back into a module +========================== + +A module's function pointer is the address of a descriptor in its **writable +segment**. Firmware that stores one and later branches to it jumps into RAM +data. ``fdpic_callback()`` resolves such a pointer to a code address, +deciding whether it is a descriptor by testing r9: a module task runs with its +GOT there, a kernel task with zero. + +It is applied at every entry point a module can reach today: ``qsort``, +``bsearch``, ``pthread_create``, ``signal``, ``sigaction``, +``task_create``/``task_create_with_stack``, ``task_spawn``, ``pthread_once``, +``scandir`` (its filter), and ``mq_notify``/``timer_create`` with +``SIGEV_THREAD``. + +A new entry point that takes a module callback must resolve it too, under +three rules: + +* **Resolve once, in the innermost common routine.** Resolving twice treats a + code address as a descriptor. ``qsort`` recurses, so its wrapper resolves + and the recursive body does not; ``signal()`` does not resolve because + ``nxsig_action()`` does it for both paths; ``scandir`` resolves its filter + but not the comparison function it hands to ``qsort``. + +* **Exclude sentinel values by hand.** ``fdpic_callback()`` declines to + dereference NULL and nothing else. ``sigaction`` excludes ``SIG_IGN``, + ``SIG_DFL``, ``SIG_HOLD`` and ``SIG_ERR`` -- the integers 0, 1, 2 and -1. + +* **A callback on a shared thread needs its base installed.** A + ``SIGEV_THREAD`` notification runs on a work-queue worker with no module + base, so resolving the entry is not enough. Capture the base at + registration with ``fdpic_base()``, in the module's context, and install it + around the call with ``fdpic_invoke()``. + +Everywhere else the callback runs in a task that inherited the module's +D-Space, so only the code address needs resolving. + +Reference +========= + +Object layout +------------- + +The FDPIC marker is ``e_ident[EI_OSABI] == ELFOSABI_ARM_FDPIC`` (65), which +``readelf -h`` shows as "OS/ABI: ARM FDPIC". It is *not* in ``e_flags``, +which read an ordinary ``0x5000000, Version5 EABI``. Validate the OS/ABI byte +and nothing else. + +A linked module has the shape execute-in-place needs with no linker script:: + + LOAD vaddr 0x00000000 R E .text .rodata .hash .dynsym .dynstr + LOAD vaddr 0x000012b4 RW .dynamic .got .data .bss + DYNAMIC DT_PLTGOT -> .got + +``.rodata`` lands in the RX segment on its own, resolved PC-relative or +GOT-indirect. That matters: in the RW segment it would be copied to RAM with +``.data`` and most of the saving would evaporate, silently and with everything +still working. + +``DT_PLTGOT`` gives the GOT base, which is the value r9 must hold. + +Relocations +----------- + +The static link resolves ``R_ARM_GOT_BREL`` and ``R_ARM_GOTFUNCDESC`` into the +GOT already, so only three types carry work into a linked module. +``R_ARM_NONE`` is accepted and skipped. + +``R_ARM_RELATIVE`` + An address needing its segment's base added. + +``R_ARM_FUNCDESC_VALUE`` + A descriptor the linker has laid out, for the loader to fill in with + {resolved code, GOT base}. This is what a *call* to an imported function + produces, and where the firmware's symbol table is consulted. + +``R_ARM_FUNCDESC`` + A pointer to a descriptor that does not exist yet, which the loader + manufactures from a pool behind the writable segment. This is what *taking + the address* of an imported function produces -- a different thing from + calling one, and both can appear for the same symbol. Unlike NXFLAT, a + pointer to a ``static`` function needs no workaround. + +Constants, from binutils ``include/elf/arm.h`` and now in +``arch/arm/include/elf.h``: ``R_ARM_GOTFUNCDESC`` 161, +``R_ARM_GOTOFFFUNCDESC`` 162, ``R_ARM_FUNCDESC`` 163, +``R_ARM_FUNCDESC_VALUE`` 164. + +Relocation tables +----------------- + +Which table an imported function's descriptor lands in is the linker's +decision. With ``-z now`` imports stay in ``DT_REL``; without it they go to +``DT_JMPREL`` and a module may have no ``DT_REL`` at all. + +The loader binds **both**, eagerly. There is no lazy resolver, so an unwalked +table would not be deferred work -- the descriptors would stay unrelocated +until the module branched through one. Nothing is lost by binding early: a +module carries a handful of relocations. + +The two are not walked identically. In ``DT_REL`` the word being overwritten +is the addend and is added to the resolved symbol value. In ``DT_JMPREL`` it +is the lazy-binding bootstrap -- the address of the entry's own PLT stub, with +a GOT half of ``-1`` -- and the binder overwrites the descriptor outright. + +``DT_PLTREL`` must say ``REL``. Nothing reads ``RELA``, whose entries are +twelve bytes rather than eight, so an object declaring that layout is refused +rather than misread. + +``.rofixup`` is skipped. It is the self-relocation list a *static* +executable's ``crt0`` walks to find its own GOT; a module links ``-shared +-nostartfiles``, so no ``crt0`` runs, and the section holds only the GOT +self-pointer that the loader supplies from r9 instead. Everything else +load-dependent is in ``.rel.dyn``/``.rel.plt``, and a PIC object cannot keep an +absolute pointer in its read-only segment. + +binfmt integration +------------------ + +The loader registers a ``struct binfmt_s``, so modules load through the +standard dispatch alongside ELF and NXFLAT. Two details of that contract are +easy to get wrong: + +* ``binp->picbase`` is a ``struct dspace_s *``, not an address. + ``up_initial_state()`` dereferences ``->region``, so a bare GOT puts a kernel + address in r9 and panics. + +* ``binp->stacksize`` must be set, or ``binfmt_execmodule`` rejects the load + with ``-EINVAL``. + +``binfmt`` reports any loader error as ``-ENOENT``, because at the dispatch +layer "refused it" and "not my format" are the same answer. The reason +reaches the syslog only, so do not read ``-ENOENT`` from ``exec`` as "file not +found". + +Testing +======= + +``apps/testing/fs/xipfs`` stages modules out of the filesystem and asserts on +them. ``xipfs_test fdpic`` runs that part in about twenty seconds, and +``xipfs_test reject`` the loader's refusal paths. + +It covers what fails silently: constructors in dependency order, per-instance +data, manufactured descriptors, both relocation tables, the GOT fallback for a +library with no PLT, and every callback entry point. Each check has been +verified against a deliberate one-line breakage of the loader. + +**A run that ends without its summary line is a failure, not a hang.** These +tests call into modules whose relocations must be correct; when they are not, +the module branches to an unrelocated address and the HardFault takes the +system down mid-test, because on Cortex-M a HardFault is system-wide rather +than per-task. + +The module sources are in ``apps/examples/fdpicxip/modules``: ``libshape.cpp`` +for the init/fini arrays, ``funcdesc.c`` for the descriptor pool, ``lazymod.c`` +for ``DT_JMPREL``, ``libcounter.c`` for a leaf library with no ``DT_PLTGOT``. +An opt-in ``make regen`` there rebuilds the headers both apps embed; it is +never part of the application build, so the tree stays buildable with a plain +toolchain. + +``apps/examples/fdpicxip`` is a runnable demonstration of the same modules -- +see :doc:`/applications/examples/fdpicxip/index`. diff --git a/Documentation/components/index.rst b/Documentation/components/index.rst index fe60af7b985..7b041920dce 100644 --- a/Documentation/components/index.rst +++ b/Documentation/components/index.rst @@ -14,6 +14,7 @@ case, you can head to the :doc:`reference <../reference/index>`. binfmt.rst concurrency/index.rst drivers/index.rst + fdpic.rst nxflat.rst nxgraphics/index.rst paging.rst