Aksal/nuttx-apps
1
0
Fork 1
mirror of https://github.com/apache/nuttx-apps.git synced 2026-08-01 20:29:00 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
nuttx-apps/interpreters/python/Makefile
Marco Casaroli 7aac6c5cb8 testing/ostest: split the fork test into task_fork, vfork and fork 
nuttx implements fork() and vfork() as the same function, and is gaining the
three separate primitives its issue #19540 describes:  task_fork() (shares
memory, private stack copy, both running), vfork() (shares memory, parent
suspended) and POSIX fork() (child gets its own copy).  This is the apps side
of that, and it lands first:  it works against nuttx with or without the
split, so the tests keep running across the transition rather than silently
compiling out.

ostest's "vfork" test was never testing vfork().  It has the child write a
global and the parent observe the write -- which is the defining property of
*sharing*, not of vfork(), whose defining property is that the parent is
suspended and whose contract forbids the child to write anything at all.  It
is renamed to task_fork.c, unchanged, because that is the primitive it has
always described.

vfork.c is rewritten to test what vfork() promises.  The child does only what
POSIX permits -- it calls _exit(42), and nothing else, not even exit(), which
would run atexit handlers and flush stdio in the parent's address space.  The
observable is therefore the child's exit status rather than a memory write.
Where child status is not retained -- ostest_main() sets SA_NOCLDWAIT for the
whole run, deliberately -- waitpid() returning ECHILD is accepted as equally
good evidence:  it says the child was already gone when the parent asked.

fork.c is new and tests POSIX fork():  the child's writes to .data, .bss and
the heap are invisible to the parent and vice versa, a pointer to a stack
local taken before the fork names the same object in both, and the child does
everything a vfork() child may not -- calls malloc() and printf(), and returns
from the function that called fork().

All three run at the top of user_main() rather than in the middle.  They
exercise the lowest-level machinery in the suite -- address environments,
stack setup, the architecture's register context -- so a fault in one takes
the process down instead of reporting a failure, and finding that out in
seconds rather than after everything else has passed is the difference
between a usable iteration and a coffee break when a port is being brought
up.

The other in-tree callers are audited for which primitive they actually
meant.  nand_sim wants a daemon that outlives its caller and shares its
memory, which is task_fork().  bas's SHELL and EDIT statements, python's
_posixsubprocess and libwebsockets' feature macros want the fork-then-exec
path, which vfork() serves; python's os.fork() and libwebsockets'
LWS_HAVE_FORK stay on fork() proper.  fdsantest's vfork case follows vfork().

Two third-party suites need their source lists narrowed, because they call
fork() from code that is compiled unconditionally:

* system/libuv -- test-fork.c and test-pipe-close-stdout-read-stdin.c are
  filtered out of the test-*.c glob.  Every test they define is already
  excluded from the task list on NuttX by 0001-libuv-port-for-nuttx.patch --
  the nine fork_* entries and pipe_close_stdout_read_stdin -- so they were
  dead code being compiled only because fork() happened to be declared.
* testing/ltp -- the open_posix_testsuite is filtered through the existing
  BLACKWORDS mechanism, which already drops tests for absent features and is
  already conditioned on configuration symbols.  Where fork() is not
  provided this drops 278 of 1943 test files; the pattern is written to spare
  vfork() and task_fork(), which remain available.  Where fork() is provided
  -- which today is everywhere -- nothing is dropped.

Compatibility: the nuttx symbols this keys on do not exist yet.  task_fork.c
is built where CONFIG_TASK_FORK says task_fork() was built and, on a nuttx that
has no CONFIG_ARCH_HAVE_TASK_FORK at all -- which is the pre-split one -- where
CONFIG_ARCH_HAVE_FORK does.  Today's fork() *is* task_fork(), so the test that
has always covered that primitive keeps running, under its own name, and no
coverage is lost across the transition.  Both spellings are needed because
CONFIG_TASK_FORK is optional on the nuttx side:  ARCH_HAVE_TASK_FORK says the
architecture can clone a task, TASK_FORK says this build asked for it.  A
follow-up removes the fallback once the split has landed.

vfork_test() and fork_test() deliberately have no such fallback.  Both check
semantics a pre-split nuttx does not describe -- the parent suspension and the
private copy -- and ARCH_HAVE_VFORK is the evidence that the split has landed.
Mapping them onto ARCH_HAVE_FORK would also add a test to configurations that
never had one, which is not free:  vfork.c costs about 470 bytes of .text on
armv7-m at -Os, and that is what put lm3s6965-ek:qemu-protected over its 128
KiB user flash region.

Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-31 14:35:25 +02:00

296 lines
13 KiB
Makefile
Raw Blame History

############################################################################
# apps/interpreters/python/Makefile
#
# SPDX-License-Identifier: Apache-2.0
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership. The
# ASF licenses this file to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance with the
# License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
############################################################################
include $(APPDIR)/Make.defs
CPYTHON_URL ?= "https://github.com/python/cpython/archive"
CPYTHON_VERSION = $(patsubst "%",%,$(strip $(CONFIG_INTERPRETERS_CPYTHON_VERSION)))
CPYTHON_VERSION_MINOR=$(basename $(CPYTHON_VERSION))
CPYTHON_ZIP = v$(CPYTHON_VERSION).zip
CPYTHON_UNPACKNAME = Python
UNPACK ?= unzip -q -o
MACHDEP=nuttx
CONFIG_SITE=${CURDIR}/config.site
SETUP_LOCAL=${CURDIR}/Setup.local
CPYTHON_PATH=$(CURDIR)/$(CPYTHON_UNPACKNAME)
BUILDIR=$(CURDIR)/build
INSTALLDIR=$(CURDIR)/install
HOSTBUILD=$(BUILDIR)/host
HOSTINSTALL=$(INSTALLDIR)/host
HOSTPYTHON=$(HOSTINSTALL)/bin/python3
TARGETBUILD=$(BUILDIR)/target
TARGETINSTALL=$(INSTALLDIR)/target
TARGETLIBPYTHON=$(TARGETINSTALL)/libpython$(CPYTHON_VERSION_MINOR).a
TARGETMODULESPACK=$(TARGETBUILD)/lib/python$(shell echo $(CPYTHON_VERSION_MINOR) | tr -d .).zip
TARGETMODULES=$(TARGETINSTALL)/lib/
CFLAGS += ${INCDIR_PREFIX}$(CPYTHON_PATH)$(DELIM)Include
CFLAGS += ${INCDIR_PREFIX}$(CPYTHON_PATH)$(DELIM)Test
CFLAGS += ${INCDIR_PREFIX}$(CPYTHON_PATH)$(DELIM)Include$(DELIM)internal
CFLAGS += ${INCDIR_PREFIX}$(APPDIR)$(DELIM)system
CFLAGS += ${INCDIR_PREFIX}$(APPDIR)$(DELIM)system$(DELIM)zlib$(DELIM)zlib
CFLAGS += ${INCDIR_PREFIX}$(TARGETBUILD)
CFLAGS += -Wno-shadow -Wno-undef -Wno-format -Wno-builtin-macro-redefined
CFLAGS += -Wno-type-limits -Wno-implicit-fallthrough -Wno-char-subscripts
CFLAGS += -Wno-sign-compare -Wno-unused-const-variable -Wno-unused-function
CFLAGS += -Wno-unused-variable -Wno-overflow -Wno-unused-but-set-variable
CFLAGS += -Wno-strict-prototypes -Wno-maybe-uninitialized -nostdlib
DEPPATH += --dep-path $(CPYTHON_UNPACKNAME)$(DELIM)Programs
VPATH += :$(CPYTHON_UNPACKNAME)$(DELIM)Programs
$(CPYTHON_ZIP):
@echo "Downloading: $(CPYTHON_URL)/$(CPYTHON_ZIP)"
$(Q) $(call DOWNLOAD,$(CPYTHON_URL),$(CPYTHON_ZIP))
$(CPYTHON_UNPACKNAME): $(CPYTHON_ZIP)
@echo "Unpacking: $(CPYTHON_ZIP) -> $(CPYTHON_UNPACKNAME)"
$(Q) $(UNPACK) $(CPYTHON_ZIP)
$(Q) mv cpython-$(CPYTHON_VERSION) $(CPYTHON_UNPACKNAME)
@echo "Patching $(CPYTHON_UNPACKNAME)"
$(Q) patch -p1 -d $(CPYTHON_UNPACKNAME) < patch$(DELIM)0001-workaround-newlib-resource.h-limitations.patch
$(Q) patch -p1 -d $(CPYTHON_UNPACKNAME) < patch$(DELIM)0002-fix-various-uint32_t-unsigned-int-type-mismatch-issu.patch
$(Q) patch -p1 -d $(CPYTHON_UNPACKNAME) < patch$(DELIM)0003-reuse-wasm_assets.py-for-generating-an-archive-of-py.patch
$(Q) patch -p1 -d $(CPYTHON_UNPACKNAME) < patch$(DELIM)0004-recognize-nuttx-as-a-supported-OS.patch
$(Q) patch -p1 -d $(CPYTHON_UNPACKNAME) < patch$(DELIM)0005-gh-122907-Fix-Builds-Without-HAVE_DYNAMIC_LOADING-Se.patch
$(Q) patch -p1 -d $(CPYTHON_UNPACKNAME) < patch$(DELIM)0006-change-var-name-to-avoid-conflict-with-nuttx-unused_.patch
$(Q) patch -p1 -d $(CPYTHON_UNPACKNAME) < patch$(DELIM)0007-undef-atexit_register.patch
$(Q) patch -p1 -d $(CPYTHON_UNPACKNAME) < patch$(DELIM)0008-declare-struct-timeval.patch
$(Q) patch -p1 -d $(CPYTHON_UNPACKNAME) < patch$(DELIM)0009-include-nuttx-sys-select-header-to-define-FD_SETSIZE.patch
$(Q) patch -p1 -d $(CPYTHON_UNPACKNAME) < patch$(DELIM)0010-check-for-the-d_ino-member-of-the-structure-dirent.patch
$(Q) patch -p1 -d $(CPYTHON_UNPACKNAME) < patch$(DELIM)0011-avoid-redefinition-warning-if-UNUSED-is-already-defi.patch
$(Q) patch -p1 -d $(CPYTHON_UNPACKNAME) < patch$(DELIM)0012-hack-place-_PyRuntime-structure-into-PSRAM-bss-regio.patch
$(Q) patch -p1 -d $(CPYTHON_UNPACKNAME) < patch$(DELIM)0013-transform-functions-used-by-NuttX-to-lowercase.patch
$(Q) patch -p1 -d $(CPYTHON_UNPACKNAME) < patch$(DELIM)0014-insert-prefix-to-list_length-to-avoid-symbol-collisi.patch
$(Q) patch -p1 -d $(CPYTHON_UNPACKNAME) < patch$(DELIM)0015-keep-ensurepip-in-stdlib-archive.patch
$(Q) patch -p1 -d $(CPYTHON_UNPACKNAME) < patch$(DELIM)0016-fix-timezone-offset-check-when-time-t-is-unsigned.patch
$(Q) patch -p1 -d $(CPYTHON_UNPACKNAME) < patch$(DELIM)0017-stdlib-zip-keep-pydecimal-and-trim-tooling-extras.patch
$(Q) patch -p1 -d $(CPYTHON_UNPACKNAME) < patch$(DELIM)0018-ignore-chmod-on-nuttx-like-wasi.patch
$(Q) patch -p1 -d $(CPYTHON_UNPACKNAME) < patch$(DELIM)0019-undef-get_errno-set_errno-to-avoid-nuttx-macro-coll.patch
$(Q) patch -p1 -d $(CPYTHON_UNPACKNAME) < patch$(DELIM)0020-ctypes-skip-pythonapi-on-nuttx.patch
$(Q) patch -p1 -d $(CPYTHON_UNPACKNAME) < patch$(DELIM)0021-posixmodule-ignore-utime-enosys-on-nuttx.patch
$(HOSTPYTHON):
mkdir -p $(HOSTBUILD)
mkdir -p $(HOSTINSTALL)
$(Q) ( \
cd $(HOSTBUILD) && $(CPYTHON_PATH)/configure \
--with-pydebug \
--prefix=$(HOSTINSTALL) \
--disable-test-modules \
)
$(MAKE) -C $(HOSTBUILD) install
# The `config.site` file contains settings that override the configuration
# settings provided by the `configure` script. Depending on the features
# enabled on NuttX, this file may need to be adjusted.
$(CONFIG_SITE):
$(Q) ( cp $(CONFIG_SITE).in $(CONFIG_SITE))
ifeq ($(CONFIG_ARCH_HAVE_FORK),y)
@echo "export ac_cv_func_fork=\"yes\"" >> $@
else
@echo "export ac_cv_func_fork=\"no\"" >> $@
endif
ifneq ($(CONFIG_ARCH_HAVE_VFORK)$(CONFIG_ARCH_HAVE_FORK),)
@echo "export ac_cv_func_vfork=\"yes\"" >> $@
else
@echo "export ac_cv_func_vfork=\"no\"" >> $@
endif
ifeq ($(CONFIG_SYSTEM_SYSTEM),y)
@echo "export ac_cv_func_system=\"yes\"" >> $@
else
@echo "export ac_cv_func_system=\"no\"" >> $@
endif
ifeq ($(CONFIG_NET),y)
@echo "export ac_cv_func_getaddrinfo=\"yes\"" >> $@
@echo "export ac_cv_func_getnameinfo=\"yes\"" >> $@
else
@echo "export ac_cv_func_getaddrinfo=\"no\"" >> $@
@echo "export ac_cv_func_getnameinfo=\"no\"" >> $@
endif
# The `Setup.local` file enables or disables Python modules.
# Depending on the features enabled on NuttX, this file may need to be
# adjusted. Please note that the base `Setup.local.in` file only contains
# a section to disable Python modules. Inserting lines to it will disable
# such modules.
$(SETUP_LOCAL):
$(Q) ( cp $(SETUP_LOCAL).in $(SETUP_LOCAL))
# _posixsubprocess is the fork-then-exec path, so vfork() is enough for it;
# os.fork() itself needs a real fork() and is governed by ac_cv_func_fork
# above.
ifeq ($(CONFIG_ARCH_HAVE_FORK)$(CONFIG_ARCH_HAVE_VFORK),)
@echo "_posixsubprocess" >> $@
endif
ifneq ($(CONFIG_LIBC_DLFCN),y)
@echo "_ctypes" >> $@
endif
ifneq ($(CONFIG_NET),y)
@echo "_socket" >> $@
endif
# For the Python's `configure` script, please consider the following
# when building for NuttX:
#
# Use sed to remove optimization flags from NuttX's CFLAGS because
# Python's configure script requires them in OPT. Having the flags in
# both places causes a conflict.
#
# Also, use -O0 for OPT because -Os is causing problems in
# Python/Modules/getpath.c (issue will be filed soon to track this
# problem).
ifneq ($(CONFIG_NET),y)
PYTHON_CONFIGURE_EXTRAS = --disable-ipv6
else
PYTHON_CONFIGURE_EXTRAS =
endif
$(TARGETBUILD)/Makefile: $(HOSTPYTHON) $(CONFIG_SITE) $(SETUP_LOCAL)
$(Q) mkdir -p $(TARGETBUILD)/Modules
$(Q) mkdir -p $(TARGETMODULES)/python$(CPYTHON_VERSION_MINOR)
$(Q) ( cp Setup.local $(TARGETBUILD)/Modules/Setup.local )
$(Q) ( \
cd $(TARGETBUILD); \
CFLAGS="$(CFLAGS)"; \
ARCH=$(CONFIG_ARCH); \
ARCH_CHIP=$(CONFIG_ARCH_CHIP); \
ARCH="$${ARCH//-/}"; \
ARCH_CHIP="$${ARCH_CHIP//-/}"; \
CFLAGS="$$(echo "$${CFLAGS}" | sed 's/-Os //')" \
CC="$(CC)" \
CXX="$(CXX)" \
AR="$(AR)" \
ARFLAGS=" " \
MACHDEP="$(MACHDEP)" \
OPT="-Os" \
CONFIG_SITE="$(CONFIG_SITE)" \
$(CPYTHON_PATH)/configure \
--prefix=${TARGETINSTALL} \
--disable-shared \
--host=$${ARCH}-$${ARCH_CHIP}-nuttx \
--build=$(shell $(CPYTHON_PATH)/config.guess) \
--with-build-python=${HOSTPYTHON} \
--without-mimalloc \
--without-pymalloc \
--disable-test-modules \
--with-ensurepip=no \
$(PYTHON_CONFIGURE_EXTRAS) \
)
$(Q) sed -i 's/^#define HAVE_LIBB2 1/\/* #undef HAVE_LIBB2 *\//' $(TARGETBUILD)/pyconfig.h
$(Q) sed -i 's/-lb2//g' $(TARGETBUILD)/Makefile
BUNDLED_WHEELS_DIR = $(CPYTHON_PATH)/Lib/ensurepip/_bundled
# PyPI downloads use the host system pip, not $(HOSTPYTHON)'s ensurepip bundle:
# pip 24.x vendors urllib3 that breaks on Python 3.13 (zlib.Decompress.unused_data).
HOSTPIP = python3 -m pip
$(TARGETLIBPYTHON): $(TARGETBUILD)/Makefile
ifneq ($(or $(filter y,$(CONFIG_INTERPRETERS_CPYTHON_ENABLE_PIP)),$(filter y,$(CONFIG_INTERPRETERS_CPYTHON_INSTALL_NUTTX_PACKAGE))),)
$(Q) mkdir -p $(BUNDLED_WHEELS_DIR)
endif
ifeq ($(CONFIG_INTERPRETERS_CPYTHON_ENABLE_PIP),y)
$(Q) ( \
PIP_WHEEL_VERSION=$$($(HOSTPYTHON) -c "import ensurepip; print(ensurepip._PIP_VERSION)"); \
PIP_WHEEL=$(BUNDLED_WHEELS_DIR)/pip-$${PIP_WHEEL_VERSION}-py3-none-any.whl; \
if [ ! -f "$${PIP_WHEEL}" ]; then \
echo "Fetching pip wheel $${PIP_WHEEL_VERSION} for ensurepip bundle"; \
$(HOSTPIP) download --only-binary=:all: --no-deps --dest $(BUNDLED_WHEELS_DIR) pip==$${PIP_WHEEL_VERSION}; \
fi; \
echo "Pre-compiling pip wheel with build Python (must match embedded CPython version)"; \
$(HOSTPYTHON) $(CURDIR)/repack_wheel_add_pyc.py --package pip "$${PIP_WHEEL}"; \
)
endif
ifeq ($(CONFIG_INTERPRETERS_CPYTHON_INSTALL_NUTTX_PACKAGE),y)
$(Q) ( \
set -e; \
NUTTX_WHEEL=$$(ls $(BUNDLED_WHEELS_DIR)/nuttx_periphery-*-py3-none-any.whl 2>/dev/null | head -1 || true); \
if [ -z "$${NUTTX_WHEEL}" ]; then \
echo "Fetching latest nuttx-periphery wheel for ensurepip bundle"; \
$(HOSTPIP) download --only-binary=:all: --no-deps --dest $(BUNDLED_WHEELS_DIR) nuttx-periphery; \
NUTTX_WHEEL=$$(ls $(BUNDLED_WHEELS_DIR)/nuttx_periphery-*-py3-none-any.whl); \
fi; \
echo "Pre-compiling nuttx-periphery wheel with build Python (must match embedded CPython version)"; \
$(HOSTPYTHON) $(CURDIR)/repack_wheel_add_pyc.py --package nuttx_periphery "$${NUTTX_WHEEL}"; \
)
endif
$(MAKE) -C $(TARGETBUILD) regen-frozen
$(MAKE) -C $(TARGETBUILD) libpython$(CPYTHON_VERSION_MINOR).a wasm_stdlib
$(Q) ( cp $(TARGETBUILD)/libpython$(CPYTHON_VERSION_MINOR).a $(TARGETLIBPYTHON) )
$(Q) $(UNPACK) $(TARGETMODULESPACK) -d $(TARGETMODULES)/python$(CPYTHON_VERSION_MINOR)
ifneq ($(or $(filter y,$(CONFIG_INTERPRETERS_CPYTHON_ENABLE_PIP)),$(filter y,$(CONFIG_INTERPRETERS_CPYTHON_INSTALL_NUTTX_PACKAGE))),)
$(Q) mkdir -p $(TARGETMODULES)/python$(CPYTHON_VERSION_MINOR)/site-packages
$(Q) ( \
set -e; \
BUNDLED_DIR=$(TARGETMODULES)/python$(CPYTHON_VERSION_MINOR)/ensurepip/_bundled; \
SITE_PACKAGES=$(TARGETMODULES)/python$(CPYTHON_VERSION_MINOR)/site-packages; \
: > "$${SITE_PACKAGES}/bundled_wheels.pth"; \
for wheel in $${BUNDLED_DIR}/*.whl; do \
[ -f "$${wheel}" ] || continue; \
whl_name=$$(basename "$${wheel}"); \
echo "Pre-installing wheel via zipimport into target sys.path: $${whl_name}"; \
echo "../ensurepip/_bundled/$${whl_name}" >> "$${SITE_PACKAGES}/bundled_wheels.pth"; \
done; \
)
endif
MODULE = $(CONFIG_INTERPRETERS_CPYTHON)
PROGNAME += $(CONFIG_INTERPRETERS_CPYTHON_PROGNAME)
PRIORITY += $(CONFIG_INTERPRETERS_CPYTHON_PRIORITY)
STACKSIZE += $(CONFIG_INTERPRETERS_CPYTHON_STACKSIZE)
MAINSRC += python_wrapper.c
checkgenromfs:
@genromfs -h 1>/dev/null 2>&1 || { \
echo "Host executable genromfs not available in PATH"; \
echo "You may need to download in from https://romfs.sourceforge.net/"; \
exit 1; \
}
romfs_cpython_modules.img : $(TARGETLIBPYTHON) checkgenromfs
@genromfs -f $@ -d $(TARGETMODULES) -V "ROMFS_Test" || { echo "genromfs failed" ; exit 1 ; }
romfs_cpython_modules.h : romfs_cpython_modules.img
@xxd -i $< | sed -e "s/^unsigned/static const unsigned/g" >$@ || { echo "xxd of $< failed" ; exit 1 ; }
context:: $(CPYTHON_UNPACKNAME)
depend:: romfs_cpython_modules.h
distclean::
$(call DELDIR, $(BUILDIR))
$(call DELDIR, $(INSTALLDIR))
$(call DELDIR, $(CPYTHON_UNPACKNAME))
$(call DELFILE, $(CPYTHON_ZIP))
$(call DELFILE, romfs_cpython_modules.img)
$(call DELFILE, romfs_cpython_modules.h)
$(call DELFILE, config.site)
$(call DELFILE, Setup.local)
include $(APPDIR)/Application.mk
Reference in a new issue View git blame Copy permalink