nuttx/Documentation/guides/fork_vfork_migration.rst

=========================================
Migrating to separate ``fork``/``vfork``
=========================================

What changed
============

NuttX used to implement ``fork()`` and ``vfork()`` as the same function.  Both
were thin libc wrappers around a single ``up_fork()`` syscall; ``vfork()``
differed only by a trailing ``waitpid()``.  Underneath, the child joined the
parent's address environment -- the same ``addrenv_join()`` that
``pthread_create()`` uses -- and got a private *copy of the stack*.  So the
child shared ``.data``, ``.bss`` and the heap with its parent, and ran
concurrently with it.

That was not ``fork()``.  It was ``vfork()``-with-a-private-stack published
under ``fork()``'s name.  The history says so plainly: the ``fork()`` this
replaces was NuttX's old ``vfork()``, renamed in 2023 without any change of
behaviour.  And the consequence was silent: a program written against POSIX
``fork()`` compiled and ran, and its child's writes quietly landed in the
parent's variables.

There are now two distinct primitives, and each means what its name says:

.. list-table::
   :header-rows: 1
   :widths: 14 30 26 30

   * - API
     - Memory
     - Parent
     - Availability
   * - ``fork()``
     - child gets **its own copy** at the same virtual addresses
     - runs concurrently
     - ``CONFIG_ARCH_HAVE_FORK`` -- only where an address environment can be
       duplicated
   * - ``vfork()``
     - child **shares** the parent's memory
     - **suspended** until the child ``_exit()``\ s or ``exec()``\ s
     - ``CONFIG_ARCH_HAVE_VFORK`` -- no address environment needed

.. note::

   ``fork()`` is provided only where the architecture implements
   ``up_addrenv_fork()`` and therefore selects ``CONFIG_ARCH_HAVE_FORK``; it
   becomes available architecture by architecture as that hook lands.  Check
   ``CONFIG_ARCH_HAVE_FORK`` in your own configuration rather than assuming
   either way.  Where it is unset, ``vfork()`` is what the configuration
   offers.

This is a breaking change
=========================

Two things break, and they break loudly rather than quietly:

**Code calling** ``fork()`` **on a target without a duplicable address
environment no longer builds.**  ``fork()`` is not declared in ``unistd.h``
there, so you get a compile error naming the function.  That is the intended
outcome:  a build error is strictly better than the silent wrongness it
replaces.  Today that is every in-tree architecture, so every caller of
``fork()`` has to be looked at.

**Code calling** ``fork()`` **on a target that does have real** ``fork()``
**changes behaviour** -- from sharing to copying.  Code that (perhaps
unknowingly) relied on the sharing will now see the parent and child diverge.

Which replacement do I want?
============================

Answer the question "why did I call ``fork()``?".

*I want the child to run a different program.*
   Use :c:func:`posix_spawn` or ``task_spawn()``.  This is the single most
   common reason to call ``fork()``, NuttX has always provided a better answer
   for it, and that answer does not have the pid discontinuity that
   ``fork()``\ +\ ``exec()`` has.  If you must keep the two-step idiom, use
   ``vfork()`` + ``exec*()``:  that is exactly what ``vfork()`` is for, and
   unlike ``fork()`` it needs no duplicable address environment.

   .. code-block:: c

      #include <unistd.h>

      pid = vfork();         /* was: pid = fork(); */
      if (pid == 0)
        {
          execv(path, argv); /* or _exit() on failure */
          _exit(EXIT_FAILURE);
        }

   Note the restriction that comes with it:  between the ``vfork()`` and the
   ``exec()`` the child shares the parent's memory and runs on the parent's
   behalf, so it must not modify anything, must not return from the calling
   function, and must not call anything other than ``_exit()`` or an ``exec``
   family function.

*I want a second flow of control that shares my memory.*
   Use ``pthread_create()``.  That is the same memory relationship the old
   ``fork()`` gave you, spelled clearly, with a normal entry point instead of a
   function that returns twice.  There is no longer a returns-twice primitive
   with concurrent sharing semantics:  the old behaviour was not POSIX, and
   ``vfork()`` is not a drop-in for it -- the parent is suspended, so parent and
   child never run concurrently.

*I want a genuinely independent copy of this process.*
   Keep calling ``fork()``, and make sure your configuration selects
   ``CONFIG_ARCH_HAVE_FORK``.  Be aware there is no copy-on-write: the copy is
   eager, so forking a large process needs as much free memory as the process
   occupies and fails with ``ENOMEM`` otherwise.

Configuration symbols
=====================

``CONFIG_ARCH_HAVE_VFORK``
   Hidden.  The architecture can implement POSIX ``vfork()``.  Selected exactly
   where ``ARCH_HAVE_FORK`` used to be, so every configuration that had the old
   ``fork()`` has ``vfork()``.

``CONFIG_ARCH_HAVE_FORK``
   Hidden, ``depends on ARCH_ADDRENV``.  It no longer means "``fork()``
   exists"; it means "this configuration can provide POSIX ``fork()``
   semantics", which requires an address environment and an
   ``up_addrenv_fork()`` to duplicate it with.

Notes for architecture maintainers
==================================

The register/stack snapshot machinery is common to both primitives.  Each
architecture exposes one entry point, ``up_fork(bool vfork)``, whose argument
says which primitive the caller used and is handed to ``nxtask_setup_fork()``.
That is where the memory semantics are decided: ``addrenv_join()`` for
``vfork()``, ``addrenv_fork()`` for ``fork()``.

Adding real ``fork()`` to an architecture
-----------------------------------------

Two things are needed, and the second is the one that is easy to miss.

**Implement** ``up_addrenv_fork()``.  It duplicates an address environment:
allocate fresh pages, copy the parent's contents into them, and map them at the
*same* virtual addresses.  ``up_addrenv_clone()`` is not this -- it copies only
the representation and leaves both processes pointing at one set of page
tables.  Then give ``ARCH_HAVE_FORK`` a ``default y if <arch>`` line in
``arch/Kconfig``.

**Build the child from the caller's saved system call frame.**  In a kernel
build ``fork()`` is reached through a system call, so the return address and
stack pointer the architecture's fork entry point can observe for itself belong
to the *kernel*, not to the caller; a child built from those resumes at a
kernel address on a kernel stack.  The architecture must record the caller's
exception frame when it traps -- ``xcp.sregs`` is the field that exists for
this -- and build the child from that instead, while a kernel thread that calls
the entry point directly still takes the ordinary path.

Four architectures do it, and they are worth copying:

* RISC-V: ``riscv_swint.c`` stores the frame in ``xcp.sregs``, and
  ``riscv_fork.c`` rebuilds the child from it.
* arm64: ``arm64_vectors.S`` hands the frame to ``dispatch_syscall()``, which
  stores it in ``xcp.sregs``; ``arm64_fork()`` then dispatches to
  ``arm64_fork_syscall()`` or ``arm64_fork_direct()`` according to whether
  ``TCB_FLAG_SYSCALL`` is set, so a kernel thread that calls the entry point
  directly still works.
* armv7-a: ``arm_syscall.c`` stores the frame in ``xcp.sregs``, and
  ``arm_fork()`` dispatches to ``arm_fork_syscall()`` or
  ``arm_fork_direct()``.  The discriminator here is a saved user stack
  pointer, ``xcp.ustkptr``, rather than ``TCB_FLAG_SYSCALL``:  armv7-a
  dispatches a system call by re-pointing the caller's own exception frame at
  ``dispatch_syscall()``, so the caller *is* the task that runs the kernel
  side of the call.  What makes its snapshot useless is not the system call
  as such but the switch to the kernel stack, which leaves the kernel-side
  frames on a stack the child gets no copy of.  A build without a kernel
  stack dispatches on the caller's own stack, so there the frames are copied
  along with the caller's and ``arm_fork_direct()`` remains correct.  Because
  ``arm_syscall()`` has already re-pointed the frame by the time
  ``arm_fork()`` runs, its PC, CPSR and SP are the kernel's; the caller's are
  read from where ``arm_syscall()`` put them -- ``syscall[0].sysreturn``,
  ``syscall[0].cpsr`` and ``ustkptr``.
* x86_64: ``x86_64_syscall()`` stores the frame in ``xcp.sregs``, and
  ``x86_64_fork()`` dispatches to ``x86_64_fork_syscall()`` or
  ``x86_64_fork_direct()``.  The discriminator here is ``xcp.sregs`` itself
  being non-NULL, because raising ``TCB_FLAG_SYSCALL`` would also defer signal
  actions -- something x86_64 has never done and its kernel-build signal path
  does not currently survive.  Two properties of ``SYSCALL``/``SYSRET`` shape
  the child's frame:  the instruction leaves the caller's RIP and RFLAGS in
  RCX and R11 rather than on a stack, so they have to be moved into the RIP
  and RFLAGS slots of the interrupt frame the child is resumed from; and the
  hardware never records the caller's CS and SS at all -- ``SYSRETQ``
  reconstructs them from ``IA32_STAR`` -- so the child's have to be filled in
  with the user code and data selectors at RPL 3.  For the same reason the
  saved frame is not copied wholesale:  only the extended state and the
  general registers are inherited, and the segment registers and thread
  pointer come from the frame ``up_initial_state()`` built for the child.

Nothing else is required:  the ``up_fork()`` entry point and the libc wrapper
are already there and become live automatically.

Note that ``ARCH_HAVE_FORK`` is about a *per-process* address environment.  A
protected build has one address space carved up once at boot, whether the
boundaries are drawn by an MPU or by a fixed set of MMU mappings; its
``up_addrenv_*()`` are stubs, and there is no mapping to duplicate at the same
virtual addresses.  ``CONFIG_ARCH_ADDRENV`` being set is therefore not by
itself evidence that ``fork()`` can be provided.  ``vfork()``, which shares the
parent's memory, works there as everywhere else.

Known gaps
==========

``fork()`` **is gained one architecture at a time.**  The generic machinery is
complete -- ``addrenv_fork()``, the ``up_addrenv_fork()`` hook, the syscall, the
libc wrapper and the ``ostest`` case -- so an architecture provides ``fork()``
by implementing ``up_addrenv_fork()`` and selecting ``CONFIG_ARCH_HAVE_FORK``,
with no further generic work.

**A windowed ABI needs its stack rebased, not just copied.**  On Xtensa,
giving a child a relocated copy of the parent's stack takes more than the copy:
the register-window save areas embedded in the stack hold absolute stack
pointers, so each one has to be rebased along with the copy, or the child
reloads a pointer into the *parent's* stack on its very first window underflow.
That rebasing is architecture-specific and belongs with the Xtensa entry points
rather than here.

Note also on ``waitpid()`` after ``vfork()``
============================================

The ``vfork()`` parent is now resumed when the child's TCB is torn down, so by
the time it runs the child is completely gone.  Where the child called
``exec()`` this makes no difference -- ``exec_swap()`` has already given the
loaded program the child's pid, and that program is still running, so
``waitpid()`` behaves normally.  Where the child called ``_exit()``,
``waitpid()`` can only return its status if ``CONFIG_SCHED_CHILD_STATUS`` is
enabled; otherwise it returns ``ECHILD``, because NuttX does not retain the
status of a task that no longer exists.  That is a pre-existing property of
that configuration, not a change:  the previous implementation blocked in a
libc ``waitpid(WNOWAIT)`` and an application's own ``waitpid()`` afterwards hit
the same wall.