diff --git a/Documentation/components/filesystem/index.rst b/Documentation/components/filesystem/index.rst index a384354e693..3e40bdf42cf 100644 --- a/Documentation/components/filesystem/index.rst +++ b/Documentation/components/filesystem/index.rst @@ -1,3 +1,5 @@ +.. _nuttx-filesystem: + ================= NuttX File System ================= diff --git a/Documentation/components/filesystem/pseudofs.rst b/Documentation/components/filesystem/pseudofs.rst index 8f716ada971..c8e13cd4007 100644 --- a/Documentation/components/filesystem/pseudofs.rst +++ b/Documentation/components/filesystem/pseudofs.rst @@ -1,3 +1,5 @@ +.. _nuttx-pseudofs: + ================== Pseudo File System ================== @@ -208,4 +210,4 @@ will also fail. You cannot do this, for example: nsh> See also NxFileSystem in -`Porting Guide `_ \ No newline at end of file +`Porting Guide `_ diff --git a/Documentation/components/filesystem/smartfs.rst b/Documentation/components/filesystem/smartfs.rst index 03531f06d91..8ac92cde2d9 100644 --- a/Documentation/components/filesystem/smartfs.rst +++ b/Documentation/components/filesystem/smartfs.rst @@ -478,3 +478,637 @@ Things to Do - Add reporting of actual FLASH usage for directories (each directory occupies one or more physical sectors, yet the size is reported as zero for directories). + + + + +Using SmartFS +============= + +.. warning:: This section is a copy-paste from old wiki documentation. + It needs review and probably an update. + +What is SmartFS +--------------- + +SmartFS stands for **Sector Mapped Allocation for Really Tiny (SMART) flash**. +It is a filesystem that has been designed to work primary with small, +serial NOR type flash parts that are 1M byte to 16M byte in size +(though this is not a limitation). +The filesystem operates by segmenting the flash (or flash partition) +into "logical sectors" of equal size and then managing them (allocating, +mapping, chaining, releasing, etc.) to build files and directories. + +SmartFS Code Layering +--------------------- + +The system consists of two layers built on top of a standard NuttX MTD driver +layer (with it's associated hardware abstraction layer). + +The code directly above the NuttX MTD driver is the SMART MTD layer. +This interfaces with the MTD (flash driver) layer and handles low-level media +operations such as logical sector allocation, freeing and management, +erase block management, low-level formatting, wear leveling, etc. + +On top of the SMART MTD layer is the Smart Filesystem code. +The SmartFS code uses the logical sector services of the SMART MTD layer +to provide file and directory level management, such as creating new files, +chaining logical sectors together to create files, creating directories +and file / directory search and management routines. + ++------------+------------------------------------+ +| Smart FS | fs / smartfs / * | ++============+====================================+ +| SMART MTD | ``drivers/mtd/smart.c`` | ++------------+---------+---------+---------+------+ +| MTD Driver | m25px | sst25 | filemtd | etc. | ++------------+---------+---------+---------+------+ +| HW Driver | spi dev | spi dev | VFS | … | ++------------+---------+---------+---------+------+ + +Example SmartFS Device Setup +---------------------------- + +Setting up a device for use with SmartFS is typically done in the config +specific source initialization files and would look something like:: + + int board_app_initialize(uintptr_t arg) + { + FAR struct spi_dev_s *spi; + FAR struct mtd_dev_s *mtd; + int minor = 0; + + /* Initialize the SPI bus #3 with an M25P FLASH driver */ + + spi = stm32_spibus_initialize(3); + mtd = m25p_initialize(spi); + + /* Initialize SMART MTD to work with M25P FLASH device */ + + smart_initialize(minor, mtd, NULL); + } + +Upon successful initialization of the code above, +the NuttX Virtual File System (VFS) will contain a new entry +called ``/dev/smart0`` to represent the SMART MTD device. +Note that this is not a filesystem, but rather a raw block device +(which may or may not already be formatted for use with SmartFS). + +To use the ``/dev/smart0`` as a filesystem, it must be initialized +and mounted to the VFS as follows:: + + nsh> mksmartfs /dev/smart0 + nsh> mount -t smartfs /dev/smart0 /mnt + +Details of Operation +-------------------- + +Pages, Blocks, Sectors and things that FLASH +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +There are a number of companies that manufacture FLASH parts, and typically +they use varying terminology in their data sheets when referring +to device geometry. +All devices generally have different region sizes for programming, erasing +and reading and usually use terms like page, erase block, sector +and/or sub-sector. + +To avoid confusion, NuttX uses the following terminology: + +* **Page or Block:** The smallest area that a device can program + with a single command. This is typically 256 bytes. +* **Erase Block:** The smallest area that a device can erase. + If multiple erase block sizes are supported (e.g. 4K, 32K and 64K), then + the smallest of those sizes, though for larger parts (8M Byte - 32M Byte), + sometimes it is better to use the larger erase block size to keep RAM usage + to a minimum. +* **Sector:** Same as Erase Block. +* **Logical Sector:** A size used by a specific filesystem for managing + a region independently from the actual device specified blocks / sectors. + SmartFS (for example) can use any power of 2 value from 256 through 32768. + +Given that each manufacturer and each part has varying geometry sizes, +SmartFS uses a Logical Sector whose size is determined when the device +is formatted (defaulting to ``CONFIG_MTD_SMART_SECTOR_SIZE``). +The SmartFS code then performs all operations using logical sectors and maps +physical accesses to the device based on it's reported geometry. +This allows filesystem performance tuning (total sectors, minimum allocation +size, overhead waste, etc.) independently from the device's erase block size, +etc. Each logical sector contains a 10-byte header (5 for MTD layer, +5 for FS layer) for format management. + +An example 128K Byte Flash with 32K and 4K Erase Block sizes and 256 byte page +read/write sizes: + ++---------------------+-----------------------------------------------------------------------------------------------------+ +| Bulk Erase | Entire Device | ++=====================+========================================+===============+===============+============================+ +| Sector Erase | 32K | 32K | 32K | 32K | ++---------------------+-----------------------------+-----+----+----+-----+----+----+-----+----+----+-----+-----------------+ +| Sector Erase | 4K | ... | 4K | 4K | ... | 4K | 4K | ... | 4K | 4K | ... | 4K | ++---------------------+-----+-----+-----+-----+-----+-----+----+----+-----+----+----+-----+----+----+-----+-----+-----+-----+ +| Page read/write | 256 | 256 | ... | 256 | 256 | ... | 256 | ... | 256 | ++---------------------+-----+-----+-----+-----+-----+-----------------------------------------------------+-----+-----+-----+ +| Logical Sector (FS) | 512 | ... | 512 | ... | ++---------------------+-----------+-----+-----------+-----------------------------------------------------------------------+ + + +In the example above, a SMART MTD logical sector size of 512 bytes was chosen. +On the 128K Byte FLASH represented in the table above, +this means there would be: + +* 128K / 512 = 256 Logical Sectors Total. +* 4K / 512 = 8 Logical Sectors per Erase Block. +* 512 / 256 = 2 MTD Read/Write Blocks per SMART Logical Sector. +* 10 / 512 = 1.95% Overhead for logical sector headers. +* A maximum of about 250 files / directories supported + (considering format overhead). +* Allocations to files in 512 byte chunks + (a zero-length file consumes 512 bytes). + +Choosing a SMART Logical sector size of 256 bytes instead would give +the following results: + +* 128K / 256 = 512 Logical Sectors Total. +* 4K / 256 = 16 Logical Sectors per Erase Block. +* 256 / 256 = 1 MTD Read/Write Block per SMART Logical Sector. +* 10 / 256 = 3.9% Overhead for logical sector headers. +* A maximum of about 506 files / directories supported + (considering format overhead). +* Allocations to files in 256 byte chunks + (a zero-length file consumes 256 bytes). + +Checking Your MTD Geometry +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Most of the NuttX MTD drivers (in the ``drivers/mtd directory``) have been +tested to work with SmartFS / SMART MTD layer and to report block / erase +block sizes correctly. +However there may be one or more drivers that have not been validated. +When configuring SmartFS for the first time using a new / unknown MTD driver, +validate the reported geometries are sane based on the description above. + +Additionally, care must be taken when selecting the SMART MTD logical sector +size. The selected size is represented using 3 bits in the logical sector +status byte and stored on each sector. +This means valid values for the logical sector are: + +* 256, 512. +* 1024, 2048. +* 4096, 8192. +* 16384, 32768. + +As shown in the table / example above, the logical sector size selection +controls the maximum number of files and minimum file size on the volume. +There are some limitations to the value selected imposed by the SMART MTD +code. The logical sector size must be selected such that: + +* The total number of sectors on the device / partition fits in a 16-bit word. + This means 65536 or less total sectors (65536 is supported, though + the topmost 2 sectors will never be used). +* The logical sector size cannot be smaller than the MTD device's + block/page size. +* The logical sector size cannot be larger than the MTD devices's + reported erase block size. +* The total RAM used by the SMART MTD layer is dependent on the logical + sector size (when not using Minimize RAM Config option). + The selected logical sector size must not create a RAM requirement greater + than the available RAM. +* The number of logical sectors per MTD device erase block cannot be greater + than 256, and 4 - 128 is the best choice. +* If wear-leveling is enabled, then the following condition must be met: + + ``Total MTD Erase Blocks / 2 < (Logical Sector Size - 36) * 3``. + +Selecting a logical sector size that creates 256 logical sectors per erase +block will create some wasted space on the device. +The SMART MTD layer uses a 1-byte variable for the "free sector count" +and the "released sector count". +This means it can only track up to 255 logical sectors per erase block. +When the sectPerEraseBlk == 256, the last logical sector in each erase +block will never be used, thus causing wasted space on the device. + +RAM Usage Calculation +^^^^^^^^^^^^^^^^^^^^^ + +Efficient management of the filesystem requires building and maintaining RAM +resident status information of the SMART MTD logical sector structure +on the physical device. +During the ``smart_initialize()`` function, the code performs a device scan +(``smart_scan()`` routine) to perform this action. +Tasks performed by the smart_scan are: + +* Detect if the format sector exists (logical sector zero with "SMRT" tag). +* Determine the logical sector size that was used to format the device. +* Locate the root directory logical sectors (logical sector 3-11). +* Count the number of free sectors on the device and per erase block. +* Count the number of released sectors on the device and per erase block. + +If ``CONFIG_MTD_SMART_MINIMIZE_RAM`` is **not** set: + +* Build a map of logical sector to physical sector numbers. + +If ``CONFIG_MTD_SMART_WEAR_LEVEL`` is set: + +* Allocate wear-level RAM and read leveling info from the format sector. + +The amount of RAM consumed is dependent on the config settings +(``MINIMIZE_RAM``, ``WEAR_LEVEL``, etc.). +Rough calculation details are presented for both settings +of ``CONFIG_MTD_SMART_MINIMIZE_RAM``: + +When ``CONFIG_MTD_SMART_MINIMIZE_RAM`` is **not** set + ++---------------------------+------------------------+-------------------+--------------------+ +| Item | RAM Requirement | 1MB / 256 Logical | 8MB / 1024 Logical | ++===========================+========================+===================+====================+ +| dev struct | 368-380 (approx) | 376 | 376 | ++---------------------------+------------------------+-------------------+--------------------+ +| Logical sector map | total_sectors * 2 | 8192 | 16384 | ++---------------------------+------------------------+-------------------+--------------------+ +| Erase block free count | total erase blocks | 16 | 128 | ++---------------------------+------------------------+-------------------+--------------------+ +| Erase block release count | total erase blocks | 16 | 128 | ++---------------------------+------------------------+-------------------+--------------------+ +| Wear status | total erase blocks / 2 | 8 | 64 | ++---------------------------+------------------------+-------------------+--------------------+ +| MTD sector R/W buffer | logical sector size | 256 | 1024 | ++---------------------------+------------------------+-------------------+--------------------+ +| FS sector R/W buffer | logical sector size | 256 | 1024 | ++---------------------------+------------------------+-------------------+--------------------+ +| Total | **9,176** | **19,128** | ++----------------------------------------------------+-------------------+--------------------+ + +On larger volumes, the RAM requirement increases significantly because +of the logical sector to physical sector mapping table. +To help keep RAM requirement under control, setting +the ``CONFIG_MTD_SMART_MINIMIZE_RAM`` option eliminates this sector-to-sector +map and replaces it with a sector-to-sector cache. +The cache size is user defined via the +``CONFIG_MTD_SMART_SECTOR_CACHE_SIZE`` option. +Using this RAM reduction mode trades off RAM usage for performance. + +The cache always contains the format and root-directory logical to physical +mapping entries, and then stores additional mappings of recently used +logical sectors. When a logical sector is requested that is not contained +in the cache, then the MTD device is scanned front-to-back until +it's physical location on the device is located. +Additionally, if the number of logical sectors per erase block is 16 or less, +then the "free count" and "release count" can be packed into a single byte +per erase block. + +When ``CONFIG_MTD_SMART_MINIMIZE_RAM=y`` and +``CONFIG_MTD_SMART_SECTOR_CACHE_SIZE=64``: + ++---------------------------+-------------------------+-------------------+--------------------+ +| Item | RAM Requirement | 1MB / 256 Logical | 8MB / 1024 Logical | ++===========================+=========================+===================+====================+ +| dev struct | 368-400 (approx) | 392 | 392 | ++---------------------------+-------------------------+-------------------+--------------------+ +| Logical sector | cache cache entries * 6 | 384 | 384 | ++---------------------------+-------------------------+-------------------+--------------------+ +| Free sector bitmap | total sectors / 8 | 512 | 1024 | ++---------------------------+-------------------------+-------------------+--------------------+ +| Erase block free count | total erase blocks | 16 | 128 | ++---------------------------+-------------------------+-------------------+--------------------+ +| Erase block release count | total erase blocks | 16 | 128 | ++---------------------------+-------------------------+-------------------+--------------------+ +| Wear status | total erase blocks / 2 | 8 | 64 | ++---------------------------+-------------------------+-------------------+--------------------+ +| MTD sector R/W buffer | logical sector size | 256 | 1024 | ++---------------------------+-------------------------+-------------------+--------------------+ +| FS sector R/W buffer | logical sector size | 256 | 1024 | ++---------------------------+-------------------------+-------------------+--------------------+ +| Total | **1,832** | **4,168** | ++-----------------------------------------------------+-------------------+--------------------+ + +Partitions and Mount Points +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Creating partitions on the MTD flash / media provides the benefits +of physically isolating one filesystem from another, as well as a mechanism +for creating multiple mount points within the VFS. +There are some pros and cons to consider when deciding to use partitions +with SmartFS: + +Partitions - PROS: + +* Physically separate filesystems ... possible errors in one + doesn't affect the other. +* Smaller partitions on a large device can use smaller logical sector size + (good if files are small). +* Partitions could be used for redundancy. +* Multiple mount-points within the VFS can be performed if needed. + +Partitions - CONS: + +* Partition sizes must be understood ahead of time. +* Directory structure / files cannot span across partitions. +* Additional overhead is needed for sector management (one + erase block + 4 pages/blocks reserved per partition + vs. one+4 for entire device). +* Wear leveling (if enabled) will be limited to operate only within + each partition and cannot span multiple partitions globally. + +When using partitions with SmartFS, a partition name should be specified +during invocation of the ``smart_initialize()`` routine. +Though not a rule, a suggested standard is to provide partition names +like ``p1``, ``p2``, etc. Then the SMART MTD device entries +in the ``/dev`` directory will take the form:: + + /dev/smart0p1 + /dev/smart0p2 + etc. + +Prior to the addition of partitions in NuttX, the SmartFS implementation +had already implemented a feature which allows multiple VFS mount-points +to a single SMART MTD device. +This gives the appearance of multiple SmartFS filesystems, though in reality +it simply is a single filesystem with multiple logical root-directories. +Each of the root-directories on the filesystem will be logically isolated +from the others, and after the 'mksmartfs' each will have it's own +``/dev/smart*`` entry. + +Multi-Root Directory PROS: + +* Supports multiple mount points within the NuttX VFS. +* Each mount-point "appears" to be it's own partition ... each directory + structure is isolated from the others. +* Any mount-point / directory structure can occupy as little or as much + of the filesystem as needed. +* Mount-points can be enabled within partitions. +* Sector management overhead (reserved erase blocks, etc.) are shared + by all mount-points. +* All mount points share RAM structures. +* Wear leveling is performed evenly over all mount-points. + +Multi-Root Directory CONS: + +* The logical directories are not physically isolated. + Files, directories and individual sectors are all intermixed + on the physical device. +* Any error in the filesystem can effect all mount-points / root directories. + +This feature must be enabled specifically using the +``CONFIG_SMARTFS_MULTI_ROOT_DIRS=y`` option. +Setting this option will cause the SMART MTD ``/dev`` entries to be appended +with a directory number, such as ``d1``, ``d2``, etc. +Prior to creating a SmartFS filesystem on the raw device, +a single entry will be identified, such as:: + + /dev/smart0d1 + /dev/smart1p1d1 (device with partitions and multi-root directories) + +Then after executing the ``mksmartfs`` command, additional entries +will appear (each of which can be mounted to a different VFS location):: + + nsh> ls /dev + ... + /dev/ram0 + /dev/smart0d1 + /dev/zero + nsh> mksmartfs /dev/smart0d1 3 + nsh> ls /dev + ... + /dev/ram0 + /dev/smart0d1 + /dev/smart0d2 + /dev/smart0d3 + /dev/zero + nsh> mount -t smartfs /dev/smart0d1 /data + nsh> mount -t smartfs /dev/smart0d2 /apps + nsh> mount -t smartfs /dev/smart0d3 /recover + +The ProcFS Interface +^^^^^^^^^^^^^^^^^^^^ + +When the PROCFS interface is enabled, each mounted SmartFS device +will appear under:: + + /proc/fs/smartfs/smart# + +The pseudo files reported for each entry will depend on the configured +options. Entries that can currently appear are: + ++-------------+-------------------------+---------------------------+ +| Entry | ``CONFIG_MTD_SMART_*`` | Meaning | ++=============+=========================+===========================+ +| status | | Report volume status | +| | | including geometry. | ++-------------+-------------------------+---------------------------+ +| debuglevel | | Write ASCII '0' - '2' | +| | | to set debug print level. | ++-------------+-------------------------+---------------------------+ +| erasemap | WEAR_LEVEL=y | Report map (A-N) of erase | +| | | block erasures. | ++-------------+-------------------------+---------------------------+ +| mem | ALLOC_DEBUG=y | Print report of all SMART | +| | | MTD memory allocs. | ++-------------+-------------------------+---------------------------+ + +Example ``procfs`` usage:: + + nsh> mount -t smartfs /dev/smart0 /mnt + nsh> mount -t procfs /proc + nsh> cat /proc/fs/smartfs/smart0/status + + Format version: 1 + Name Len: 16 + Total Sectors: 4096 + Sector Size: 256 + Format Sector: 0 + Dir Sector: 256 + Free Sectors: 4078 + Released Sectors: 0 + Unused Sectors: 0 + Block Erases: 0 + Sectors Per Block: 256 + Sector Utilization:100% + Uneven Wear Count: 0 + + nsh> cat /proc/fs/smartfs/smart0/erasemap + BBACAACA + AABAAAAA + + nsh> + +When Things Don't Work +^^^^^^^^^^^^^^^^^^^^^^ + +The SmartFS code and MTD layer have been pretty well tested and used +in production products. If things aren't working for you, there are +a couple of places to start debugging first. + +Check FLASH MTD Driver +~~~~~~~~~~~~~~~~~~~~~~ + +Ensure the geometry of the FLASH MTD driver is following the NuttX standard +for block / erase block geometry sizes. Most of them do, but via simple +inspection of the code (as of version 7.16, July 12, 1016), the drivers +that are likely to have improper Geometry reporting (and thus incompatible +with SmartFS) are:: + + s25fl1.c + sst39vf.c + +When configuring to use SmartFS, check the reported geometry of the MTD +driver you are using. If the ``geo.blocksize`` is reported to be the same +as the ``geo.sectorsize``, then there is likely an issue with the MTD driver +implementation. +But it is likely to be a reporting problem of the ``geo.blocksize`` that +is incorrect AND possibly the starting address calculations in the +``_bwrite`` / ``_bread`` routines may be incorrect. +These are BLOCK read / write operations and calculations need to be +performed using the block size (typically 256), not the sector size +(4K, 32K, etc.). + +Check CONFIG Options +~~~~~~~~~~~~~~~~~~~~ + +Double check all of the CONFIG options for both the MTD driver +and the SMART MTD layer: + +* ``CONFIG_MTD_SMART_SECTOR_SIZE``: Ensure it's not smaller than the block + size or larger than erase block size. +* ``CONFIG_MTD_BYTE_WRITE``: If enabled, ensure the FLASH actually supports + this mode (single byte programming). +* ``CONFIG_MTD_XXX_SECTOR512``: Ensure this is not set. This should only + be use with FAT volumes. +* ``CONFIG_MTD_XXX_MANUFACTURER``: Double check this with the data + sheet / MTD driver code. +* ``CONFIG_MTD_XXX_MEMORY_TYPE``: Double check this with the data sheet. + +Check Writability to the Part +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Ensure the FLASH can be written successfully. +If the WP pin is pull active (i.e. the part is write protected), +then the SMART MTD layer will not be able to write any data. +Additionally validate there are no individual protected sectors on the device. + + +SmartFS Internals +================= + +.. warning:: This section is a copy-paste from old wiki documentation. + It needs review and probably an update. + +General Structure +----------------- + +As described in Using SmartFS, the code is divided into a SMART MTD layer +and a filesystem layer. The MTD layer divides the flash or partition into +equal sized logical sectors, allocates, deallocates and moves them around +as needed to fulfill requests from the filesystem layer. + +The filesystem layer uses the logical sectors to store directory and file +information, and to create chains of logical sector numbers to build larger +files (and directories). + +The diagram below depicts a portion of a SmartFS device showing the erase +blocks, logical sectors and sector assignments. In the diagram, the top row +of numbers is the absolute sector number within the device. +The bottom row with numbers represents the logical sector number assigned +to each absolute sector. +Using 5 bytes from the header in each sector, the MTD layer manages +the assignments of the logical sector numbers. + +The filesystem layer receives the logical sector numbers reported from MTD +layer and assigns them to specific files and/or directories. +As a file grows in size, additional logical sector numbers are requested +and "chained" together. In the diagram, three files are depicted +(files ``a``, ``b`` and ``c``) with varying lengths. +The files shown are as follows: + +* File ``a``: Has data in 3 sectors, ``a0-a2``, logical sectors ``12-14``. +* File ``b``: Has data in 4 sectors, ``b0-b3``, logical sectors ``15-18``. +* File ``c``: Has data in a single sector, ``c0``, logical sector ``19``. + ++----------------------------------------------------------------------------------------------------+ +| Device or partition | ++===================+===================+===================+===================+====================+ +| EB | EB | EB | EB | ... | ++----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+-----+ +| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | ... | ++----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+-----+ +| LS | LS | LS | LS | LS | LS | LS | LS | LS | LS | LS | LS | LS | LS | LS | LS | LS | LS | LS | ... | ++----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+-----+ +| FS | b0 | -- | -- | RS | b1 | -- | -- | a0 | b2 | -- | -- | a1 | b3 | -- | -- | a2 | c0 | -- | ... | ++----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+-----+ +| 0 | 15 | | | 3 | 16 | | | 12 | 17 | | | 13 | 18 | | | 14 | 19 | | | ++----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+-----+ + +* EB = Erase Block. +* LS = Logical Sector. +* FS = Format Sector. +* RS = Root-directory Sector. +* -- = Free Sector. + +The filesystem layer uses last 5-bytes of each sector's header to save +the logical sector number of the next logical sector in the chain. +When a file is first created, it's name and beginning logical sector number +are recorded in the Root-directory Sector (or a sub-directory +sector as needed). + +The filesystem layer uses the recorded logical sector numbers +in the directory sectors and header sector-chain information to perform +logical sector allocate, read/write and sector release requests to carry out +all typical file system operations. +The SMART MTD layer then performs all the logical to physical mapping, +wear-leveling, sector relocation and block erase operations, etc. +When a sector needs to be physically relocated, it will retain +it's logical sector number, preventing the need to update file sector-chain +information, etc. +The MTD layer will simply update the logical to physical map assignments. + +When things change +^^^^^^^^^^^^^^^^^^ + +Writing data to the filesystem and then reading it back is great. +But at some point you might actually want to change or delete something. +When this happens, the existing data on the flash has to be modified. +As you probably already know, data on a flash can't simply be re-written, +it must be erased in large chunks (erase blocks) that are typically 4K, +32K or 64K in size. When erasing this large a chunk of the flash, +it is highly likely that there will be data in that erase block +which should NOT be erased. + +When data in a logical sector on the flash needs to be modified or deleted, +SmartFS simply marks that logical sector as "released" without actually +touching the data. This is done using the characteristic of NOR flash +that allows multiple writes to a given address. +This feature allows any bit of any byte to be changed from a ``1`` state +to a ``0`` state, regardless of whether that byte had previously been written +with other bits set to a ``0`` state. +As long as there is no attempt to change any bits from a ``0`` to a ``1``, +each address can be programmed multiple times. + +As shown in the diagrams below, the SMART MTD layer uses the 5th byte +of the logical sector header as a status byte. +The most significant bit of this status byte (``0x80``) indicates +if the sector has been allocated (contains valid data) while the next bit +(``0x40``) indicates if the sector has been "released" (contains data +that is no longer valid). When a sector is first allocated, the +``RELEASE`` bit is held at a ``1`` state, meaning the data +has not been released. +If the data in that sector needs to be changed, deleted or relocated, +the SMART MTD code will simply re-program the status byte, +changing the ``RELEASE`` bit to a ``0`` state. +At that point, all data in that logical sector becomes invalid. + +Logical Sector MTD Header +^^^^^^^^^^^^^^^^^^^^^^^^^ + ++------------------------------------------------------------------------+------------+ +| MTD Header (5 bytes) | FS Header | ++=======================+=======+=====+==================================+============+ +| Logical Sector Number | Seq # | CRC | Status | 5 Bytes | ++----------+------------+-------+-----+----+----+----+---------+---------+------------+ +| LSB | MSB | | | CB | RB | CE | SS(2-0) | FV(1-0) | | ++----------+------------+-------+-----+----+----+----+---------+---------+------------+ + +* CB: Commit Bit. +* RB: Release Bit. +* CE: CRC Enable Bit. +* SS: Sector Size Bits. +* FV: Format Version Bits. diff --git a/Documentation/implementation/cancellation_points.rst b/Documentation/implementation/cancellation_points.rst new file mode 100644 index 00000000000..95ab8da30e7 --- /dev/null +++ b/Documentation/implementation/cancellation_points.rst @@ -0,0 +1,441 @@ +.. _cancellation-points: + +=================== +Cancellation Points +=================== + +Cancellation Types +================== + +In POSIX, there are two pthread cancellation types: +``PTHREAD_CANCEL_DEFERRED`` and ``PTHREAD_CANCEL_ASYNCHRONOUS``. + +``PTHREAD_CANCEL_DEFERRED`` is the default and the normal kind of pthread +cancellation that should be used. +You cannot have the ``PTHREAD_CANCEL_DEFERRED`` cancellation type without +cancellation points and you will not have cancellation points unless you +enable them with ``CONFIG_CANCELLATION_POINTS=y``. + +``PTHREAD_CANCEL_ASYNCHRONOUS`` is brutal; it simply kills the pthread +immediately without regard for what it is doing. +``PTHREAD_CANCEL_DEFERRED`` works differently; nothing happens immediately. +Instead, the cancellation will be deferred until the pthread is at +(or a better preposition would be within) a cancellation point. + +Cancellation points add special instrumentation to a specific set +of interface functions. Those interface functions are listed here: +http://OpenGroup.org. + + +Enter and Leave Cancelation Point +================================= + +In NuttX, each of these functions ``CONFIG_CANCELLATION_POINTS=y`` will enable +a special call to ``enter_cancellation_point()`` at the beginning +of the function and a matching call to ``leave_cancellation_point()`` before +the interface function returns. + +These may be nested: One cancellation point function may call another which +may call another, etc. The ``select()`` function, for example, calls +``poll()`` which calls ``sem_wait()`` so the nesting will be three deep. + +These two cancellation point hooks behave in a complementary way. +``enter_cancellation_point()`` increments the nesting level and +``leave_cancellation_point()`` decrements the nesting level. +No action is taken unless the nesting level is zero then, in that case, +if there is a pending cancellation then the thread exists normally +by simply calling ``pthread_exit()``. + + +pthread_cancel() +================ + +When ``pthread_cancel()`` is called, it will check if the deferred +cancellation mode is enabled for the target pthread. If so it will: + +1. Mark the cancellation as pending. +2. If the thread is waiting on a semaphore or message queue (and is in + a cancelable state), it will wake it just in the same was as if + the thread received a single (except with ``ECANCELED`` + instead of ``EINTR``). + +If, on the other hand, the asynchronous mode is enabled (and the thread is +in a cancelable state), ``pthread_cancel()`` will, instead, +kill the thread immediately. + +.. note:: For pthreads, the cancelable state is controlled by the interface + ``pthread_setcancelstate().`` This allows the pthread to disable or + re-enable cancelability. If the thread is not cancelable when + ``pthread_cancel()`` is called, it will never do more than just mark + the cancellation as pending. When the cancelable state is + re-enabled, then any pending cancellation will take effect. + + +select() +======== + +Let's walk through the case of ``select()`` to see how it works: + +* When ``select()`` is called, it allocates some memory that it needs to + handle the conversion to ``poll()``. +* It then calls ``poll()`` which sets up the poll with all of the relevant + drivers and, eventually, calls ``sem_wait()``. + +So this pthread will spend almost all of its time waiting in ``sem_wait()`` +and that is the most likely state of the pthread when ``pthread_cancel()`` +is called with deferred cancellation eanbled. + +* When ``pthread_cancel()`` is called, it will set the pending cancellation + state and wake up ``sem_wait()`` with the ``ECANCELED`` error. +* ``sem_wait()`` will see the pending cancellation, but will do nothing + because the nesting level is decremented only to two. It will return to + ``poll()`` with the ``ECANCELED`` error. Note that the semaphore is left + in a healthy state. +* ``poll()`` will tear-down down the poll from all of the drivers and call + ``leave_cancellation_point()`` before it exits. Again, it will see the + pending cancellation but will do nothing because the nesting level + decrements only to 1. Note that since ``poll()`` does the complete + tear-down and all of the drivers are left in a healthy state. +* Finally, ``select()`` cleans up all of its memory allocations and calls + ``leave_cancellation_point()``. This time, the nesting level decrements + to zero and ``leave_cancellation_point()`` will call ``pthread_exit()`` + with everything in a proper state. + +If asynchronous pthread cancellation were selected, then the behavior would +be very different. The first two steps would be the same but then: + +* When ``pthread_cancel()`` is called, the thread would be immediately killed. + The semaphore would be left in the locked state; the memory allocated by + ``select()`` would be stranded; drivers would be left with stale pointers + to stranded memory. Not a good state to leave the system! +* ``select()`` will not return to the caller in either case. + + +Application Interfaces +====================== + +The following pthread application interfaces are available to manage cancellation: + +* ``pthread_cancel()`` cancels a thread. +* ``pthread_setcancelstate()`` can enable or disable a cancellation. +* ``pthread_setcanceltype()`` can be used to switched between asynchronous + and deferred thread cancellation. +* ``pthread_testcancel()`` can be used to force a cancellation point. + + +Task Deletion +============= + +NuttX also supports some non-standard interfaces for task deletion +(i.e., cancellation) of tasks as well. The cancellation point logic +is identical and the task application interfaces are very similar: + +* ``task_delete()`` deletes (cancels) a task. +* ``task_setcancelstate()`` can enable or disable task cancellation. +* ``task_setcanceltype()`` can be used to switched between asynchronous + and deferred task cancellation. +* ``task_testcancel()`` can be used to force a cancellation point. + + +Design Issues +============= + +.. note:: The issue addressed in this paragraph is covered by + `Issue 619 `_ + has been corrected in the code base through a series of changes + culiminating in + `PR 733 `_. + This discussion still has value, however, as a warning + for things that one needs to take into consideration + when designing logic within the OS. + +There is one logical problem in many parts of the system. The following +sequence of code appears in many places in the code base. +It will work not work with thread/task cancellation: + +.. code-block:: c + + void some_lock(FAR sem_t *sem) + { + while ((ret = nxsem_wait(sem)) < 0) + { + DEBUGASSERT(ret = -EINTR || ret = -ECANCELED); + } + } + +That logic will makes it impossible to cancel the thread. +Let me explain why: + +* The fact that ``-ECANCELED`` is occurring means that some other task + tried to cancel this one. +* The logic sequence is probably something like: + + * OS Function ``A`` is called by the application. Function ``A`` is + a cancellation point. It calls ``enter_cancellation_point()`` on entry + and ``leave_cancellation_point()`` on exit. If the application task is + canceled, then the call to ``leave_cancellation_point()`` is where + the task actually ends (by simply calling exit()). + * OS Function ``A`` calls some internal Function ``B`` which has the above + killer logic in it. It calls ``some_lock()`` and the task is suspended + on waiting to get a count on the semaphore. + +When the task is canceled, ``nxsem_wait()`` wakes up with the ``-ECANCELED`` +error and returns that error to Function ``B``. But because Function ``B`` +just loops and calls ``nxsem_wait()`` again, the task does not terminate; +it just goes back to waiting. + +The end result is that Function ``B`` does not return to Function ``A`` so +``leave_cancellation_point()`` is never called and the task is not canceled. +The fix requires a little re-design. The locking function would +have to be like: + +.. code-block:: c + + int some_lock(FAR sem_t *sem) + { + while ((ret = nxsem_wait(sem)) < 0) + { + DEBUGASSERT(ret = -EINTR || ret = -ECANCELED); + if (ret != -EINTR) + { + break; + } + } + + return ret; + } + +It then returns a success/failure indication. Calling logic would also have +to be modified to detect the failure case and handle it appropriately. +In the case that the failure case, the calling logic (Function ``B`` in this +example) needs to clean up and return the failure upstream (to Function ``A``) +which must also clean up. Then, eventually, ``leave_critical_section()`` will +be called and the function will exit correctly. + + +Issue 619 Discussion +==================== + +.. note:: The + `Issue 619 `_ + has been corrected in the code base through a series of changes + culiminating in + `PR 733 `_. + +nxsem_wait_uninterruptible() +---------------------------- + +In NuttX, thread or tasks may be canceled using ``pthread_cancle()`` or +``task_delete()``. These can also be canceled via certain signals if default +signal actions are enabled. + +When threads or tasks are canceled asynchronously, resources may be stranded. +For example, if memory is allocated, it will not be deallocated when the +task/thread is canceled in the FLAT build. These kind on leaks can be handled +by the application with functions registered with ``pthread_cleanup()`` or +``on_exit()``, but others cannot. + +The generic solution for NuttX is through the use of cancellation points. +Cancellation points are a mechanism that assures that all resources used +internal by OS interface calls are cleaned-up automatically at designated +cancellation points. An interface is normally a cancellation point if +it has any possibility of blocking. + +In NuttX, when threads are blocked waiting of a semaphore when they are +canceled, ``nxsem_wait()`` will return ``-ECANCELED`` and the correct behavior +to let the error ripple back to the cancellation point where the thread/task +will be canceled after all resources have been cleaned up. + +Many internal OS functions use ``nxsem_wait_uninterruptible()`` for waiting. +``nxsem_wait_uninterruptible()`` is defined in ``include/nuttx/semaphore.h``. +``nxsem_wait_uninterruptible()`` wraps a call to ``nxsem_wait()`` in +a loop like: + +.. code-block:: c + + static inline int nxsem_wait_uninterruptible(FAR sem_t *sem) + { + int ret; + + do + { + /* Take the semaphore (perhaps waiting) */ + + ret = nxsem_wait(sem); + } + while (ret == -EINTR || ret == -ECANCELED); + + return ret; + } + +The correct behavior is to continue waiting on ``-EINTR`` only. +If the task/thread is awakened by a signal just continue waiting. +The behavior for ``-ECANCELED`` is not correct. It should not loop. +Looping like this will prevent the cancellation from working. +Instead of returning through the cancellation point, the code will be stuck +in the above loop (in defferred cancellation mode) and cannot be cancelled. + +The correct behavior is to return when ``-ECANCELED`` is encountered, +not to continue looping: + +* GOOD: ``while (ret == -EINTR);`` +* BAD: ``while (ret == -EINTR || ret == -ECANCELED);`` + +The only complexity to this suggested change is that the callers of +``nxsem_wait_uninterruptible()`` must also check the return value for any +errors and make sure that that error is returned to the caller. +The error must propogate all the way back up to the cancellation point. + +These are other related inline functions in ``include/nuttx/semaphore.h`` +that behave in this same way. This issue applies to them as well. + +.. important:: If cancellation points are enabled, the default cancellation + mode should be deferred mode. + +.. note:: There are three interfaces affected by this issue: + ``nxsem_wait_uninterruptible()`` as discussed above, but also + ``nxsem_timedwait_uninterruptible()`` and + ``nxsem_tickwait_uninteruptible()`` which are closely related. + +Mutual Exclusion Semaphores vs. Signaling Semaphores +---------------------------------------------------- + +Semaphore have two usages in NuttX: + +* Mutual Exclusion Semaphores: Protect the shared data and force mutually + exclusive use of semaphores. +* Signling Semaphores: Used to wait for events to be signaled asynchronously. + +Should we return ``-ECANCEL/-EINTR`` for both? Or only for ithe second? +I think both should work the same: For both the semaphore wait should be +terminated and correct error should be returned. + +Mutual Exclusion Semaphores +--------------------------- + +If the API requires returning ``EINTR`` if a signal is received, we should +return the error. This is required in many POSIX interfaces and don't see +how you could avoid that. However, the mutual exclusion semaphores are +the just there for the correctness in the design. There is seldom any real +competitor for the exclusion semaphore so, in general, they do not block and, +hence, would never generate any error. + +And even if taking the mutual exclusion semaphore does cause the thread +to block, it should block for only a very brief moment of time and the +likelihood of a signal received or of the task being canceled is very small. + +But if there is even a small possibility of that event happening, +it WILL happen in an embedded system (Murphy's law). + +But if during the that brief moment while the thread is blocked, +a signal is received, I think that it is correct to return ``-EINTR``. +This is not so critical only because such an event it is rare. +But the specification requires it so we should do it. + +When to Return ECANCELED +^^^^^^^^^^^^^^^^^^^^^^^^ + +The ``ECANCELED`` error is **ABSOLUTELY** critical to return under any +circumstance. That cancellation notification is received only once and +it if is ignored, then the thread will not cancel it will continue to run. +It really must return immediately with ``ECANCELED`` for the cancellation +to work quickly and reliably, where ever ``ECANCELED`` is detected. +Ignoring it is a hard bug in any case. + +Signaling Semaphores +^^^^^^^^^^^^^^^^^^^^ + +The signaling semaphores are a little different story. They may block +for a very long time, for example, waiting for the receipt of data that +may never come. For example, a read from a serial port will hang indefinitely +if nothing is received on the serial port. So if a signal is received or +if the thread is canceled, then it is essential to terminate the semaphore +wait and return the error condition. +We should have no difference of opinion on that. + +There is really no difference in the actions that must be taken if a signal +is received of if the task is canceled. Both mutual exclusion semaphores +and signaling semaphores should wake up and return the error condition. +The only real differences is that mutual exclusion semaphores either +do not block or, if they do, the do not block as long. + +Mutual Exclusion Semaphores and Serialization +--------------------------------------------- + +There are situations were the task may block for a very long time, +even indefinitely, on a mutual exclusion semaphore. Consider the following +scenario: A driver that reads data could have a sequence like this: + +1. Get exclusive access to the driver by taking a semaphore. +2. Wait on a signaling semaphore for data (might be a long time). +3. Release the mutual exclusion semaphore. + +Now suppose there are two applications, Task ``A`` and Task ``B``, +that open the driver and try to read from it. The first, Task ``A``, will get +the mutual exclusion semaphore at ``(1)`` and then will hang waiting +for data at (2). It may have to wait for a very long time to receive data. +When re-entered by the second, Task ``B``, it will hang at ``(1)`` +also for very long time. Task ``B`` is effectively queued and cannot event +begin its wait on the signaling semphore until Task ``A`` receives its data +and releases the mutual exclusion semaphore. This is normal behavior +and exactly, how this kind of driver is supposed to work: +This use of mutual exclusion impelements serialization of I/O. + +But now suppose that Task ``B`` receives a signal interrupt or a cancellation +event while waiting on the mutual exclusion semaphore. It MUST terminate the +wait and return ``-EINTR`` or ``-ECANCEL`` immediately. Task ``A`` may never +receive data and if that action is not taken, the signal interrup +or cancellation event is lost and the functionality has failed. + +When to Return EINTR +^^^^^^^^^^^^^^^^^^^^ + +Returning ``EINTR`` really only applies to ``read()`` and ``write()`` +(and ``open()``) functions as covered by Issue #669 . In other places, +``EINTR`` can be ignored. ``open()`` and ``close()`` should also +return ``EINTR``. But I don't think that is very meaningful for ``close()`` +since most people ignore the return value from ``close()``. +But, on the other hand, ``ECANCELED`` can never be ignored under any +condition. + +Detecting, Remembering, and Propagating Signal Interrupts and Cancellation Events +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Currently, the only way for an OS interface function to know if a signal +was received is to be awakened with an ``EINTR`` error. That is why +it is critical to always return the ``EINTR`` error and conform with +the POSIX requirements. + +However, if a signal is received while the OS interface is NOT waiting, +then there is no way to know if the signal was received. I think that +is a limitation in the current design. There probably should be some latching +indication, perhaps in the TCB, that a signal interrupt has been received. + +There is already a ``TCB_FLAG_CANCEL_PENDING`` in the TCB that will tell us +that the if the task has been canceled. That flag is tested in the +``leave_cancellation_point()`` function so I don't think that there is any +corresponding issue for thread cancellation. We just need to make sure that +all waits are aborted if ``ECANCELED`` is received and let the error +indication ripple all they back to the ``leave_cancellation_point()`` +function. Then the function will exit cleanly, safely, and quickly. + +Interestingly, the ``ECANCELED`` error will never be seen by the application. +It just triggers the return uwind sequence where all resources are recovered +and finally until ``leave_cancellation_point()`` is called. Then the thread +will exit before it returns to the applicaton. + +You can see all of this working in the ``board/sim/sim/sim/configs/ostest`` +configuration if you also enable:: + + CONFIG_CANCELLATION_POINTS=y + CONFIG_PTHREAD_CLEANUP=y + +There is a ``cancel.c`` test within the OS test, but the more interesting test +is the ``pthread_cleanup.c`` test. You can see how all this works when the code +unwinds with the ``ECANCELED`` error and calls ``leave_cancellation_point()``. +Just single step through ``pthread_cond_wait()`` to see this. +``pthread_cond_wait()`` is a ``cancellation_point()`` and that is where +the thread exit will occur. + +`PR#749 `_ adds those settings +to that ``sim:otest`` defconfig. diff --git a/Documentation/implementation/chip_h.rst b/Documentation/implementation/chip_h.rst new file mode 100644 index 00000000000..a2b71584126 --- /dev/null +++ b/Documentation/implementation/chip_h.rst @@ -0,0 +1,125 @@ +.. _chip-h: + +====== +chip.h +====== + +The purpose of the two chip.h files in each arm chip +==================================================== + +If you wonder about the purpose of the two ``chip.h`` files in each arm chip. + +.. code:: sh + + $ find arch/arm -name chip.h | grep stm32 + arch/arm/include/stm32/chip.h + arch/arm/src/stm32/chip.h + +The reason behind ``arch/arm/src/stm32/chip.h`` file was a bad idea +that happened a long time ago. + +Right now, I believe that its only required when ``CONFIG_ARMV7M_CMNVECTOR`` +is selected in the configuration. In that case, ``arch/arm/src/stm32/chip.h`` +is included by ``arch/arm/src/armv7-m/up_vectors.c`` in order provide +the number of interrupt vectors. In stm32, ``arch/arm/src/stm32/chip.h`` +provides the number of vectors indirectly by including the correct, +chip-specific vectors.h file. +This function is a little more obvious in ``arch/arm/srch/lpc43xx/chip.h``. + +This ``arch/arm/src/xyz/chip.h`` is also a good way to export awkward +internal header files in a cleaner way. But that use is optional +and not required outside of the chip- and board-related directories. + +For the ``arch/include/stm32/chip.h`` file, only set of definitions +is required there, the NVIC priorities: + +* ``NVIC_SYSH_PRIORITY_MIN`` provides the lowest interrupt priority + supported by the system. This is the highest numeric value; + but the lowest interrupt priority. +* ``NVIC_SYSH_PRIORITY_DEFAULT`` provides the default priority of all + interrupts. Since truly nested interrupts are not supported, this is + the priority of all interrupts except for the SVCall interrupt. +* ``NVIC_SYSH_PRIORITY_MAX`` provides the maximum interrupt priority. + For all ARMv7-M's, this will be the value zero. +* ``NVIC_SYSH_PRIORITY_STEP`` is the value need to increment from one + priority level to next, lower priority level. + +If ``CONFIG_ARMV7M_USEBASEPRI`` is selected, then interrupts will be disabled +by setting the ``BASEPRI`` register to ``NVIC_SYSH_DISABLE_PRIORITY`` so that +most interrupts will not have execution priority. +SVCall must have execution priority in all cases. + +In the normal cases, interrupts are not nest-able and all interrupts run +at an execution priority between ``NVIC_SYSH_PRIORITY_MIN`` and +``NVIC_SYSH_PRIORITY_MAX`` (with ``NVIC_SYSH_PRIORITY_MAX`` reserved +for SVCall). + +If, in addition, ``CONFIG_ARCH_HIPRI_INTERRUPT`` is defined, then special high +priority interrupts are supported. These are not "nested" in the normal sense +of the word. These high priority interrupts can interrupt normal interrupt +processing but execute outside of OS (although they can "get back +into the game" via a PendSV interrupt). + +In the normal course of things, interrupts must occasionally be disabled +using the ``up_irq_save()`` inline function to prevent contention in use +of resources that may be shared between interrupt level and non-interrupt +level logic. Now the question arises, if we are using the ``BASEPRI`` +to disable interrupts and have high priority interrupts enabled +(``CONFIG_ARCH_HIPRI_INTERRUPT=y``), do we disable all interrupts except +SVCall (we cannot disable SVCall interrupts)? +Or do we only disable the "normal" interrupts? + +If we are using the ``BASEPRI`` register to disable interrupts, then +the answer is that we must disable ONLY the normal interrupts. That is +because we cannot disable SVCALL interrupts and we cannot permit SVCAll +interrupts running at a higher priority than the high priority interrupts. +Otherwise, they will introduce jitter in the high priority interrupt +response time. + +Hence, if you need to disable the high priority interrupt, you will have to +disable the interrupt either at the peripheral that generates the interrupt +or at the interrupt controller (e.g., ``NVIC``). Disabling global interrupts +via the ``BASEPRI`` register cannot effect high priority interrupts. + +* ``NVIC_SYSH_MAXNORMAL_PRIORITY`` is the maximum priority of "normal" + interrupts. Interrupt are disabled at the level + ``NVIC_SYSH_MAXNORMAL_PRIORITY``, disabling all "normal" interrupt + but leaving the high priority and SVCALL interrupts enabled. +* ``NVIC_SYSH_HIGH_PRIORITY`` is the priority of the high priority interrupt. +* ``NVIC_SYSH_DISABLE_PRIORITY`` is the level at which interrupts will + be disabled. This will be equal to ``NVIC_SYSH_MAXNORMAL_PRIORITY``. +* ``NVIC_SYSH_SVCALL_PRIORITY`` is the priority of the SVCall interrupt. + This must be higher priority than ``NVIC_SYSH_DISABLE_PRIORITY`` so that + it is never disabled, but lower in priority than + ``NVIC_SYSH_DISABLE_PRIORITY`` so that SVCall handling cannot interfere + with high priority interrupt handling. + +These definitions only have to be here because NVIC implementations may differ +in the number of priority levels. + +Now, by convention, the ``arch/xyz/include/chip/abc/chip.h`` file is also used +to provide all of the definitions that discriminate the features of the +different members of the chip family. +It is not technically necessary that those chip feature definitions exist +in this header file, but it is a good convention to follow so that +it is easier for people who have to work across the differ architectures. + +There is another really big difference between the two chip header files: +They also different in scope: + +1. Code in the ``arch/arm/src/xyz`` and in ``boards/arm/xyz/abc/src`` + directories can both include the ``arch/arm/src/xyz/chip.h`` header file, + but nothing else can (only because nothing else is provided the header + file path in the build system). So that ``chip.h`` file is intended to be + used only in low-level board/chip logic. +2. The ``chip.h`` header file at ``arch/arm/include/xyz/chip.h``, + on the other hand, will be linked at include/arch/chip and, hence, + can be included anywhere in the system, even from applications. + It can be included like:: + + #include + +So the ``arch/arm/include/xyz/chip.h`` header file is a place where you would +want to put chip-related stuff that can be made visible to application code. +Chip capabilities is a good example of the kind of knowledge that +applications might care about. diff --git a/Documentation/implementation/context_switches.rst b/Documentation/implementation/context_switches.rst new file mode 100644 index 00000000000..986023eaafc --- /dev/null +++ b/Documentation/implementation/context_switches.rst @@ -0,0 +1,87 @@ +.. _context-switches: + +================ +Context Switches +================ + +Two Types of Context Switches +============================= + +There are really two different kinds of context switches. +We refer to them as synchronous and asynchronous context switches (but there +might be better names): + +* **Synchronous context switch** occurs when the system is interrupted and, + because of on actions within the interrupt handler, a context switch is + generated. In this case, the state of the interrupted task is saved when + the interrupt handler is entered but a different task state is restored + when the interrupt handler returns. + +* **Synchronous context switch** in our terminology occurs when a task + explicitly suspends itself by calling some OS interface that causes + the task to block, such as ``usleep()``. + + +Synchronous Context Switches +============================ + +There are two ways to implement a synchronous context switch: +``up_savecontext()`` and ``up_fullcontextrestore()``. + +You can implement the moral equivalent of ``setjmp()`` and ``longjmp()`` +on steroids. Some architectures have a function called ``up_savecontext()`` +that is the moral equivalent of ``setjmp()``, it saves the current state +of the task (and like ``setjmp()`` returns ``0`` or ``1`` to indicate +if the context is being restored. +Another function ``up_fullcontextrestore()`` is like ``longjmp()``, +it restores the context saved by either the interrupt handler during +a previous asynchronous context switch or by the ``up_savecontext()``. + +The naming differences ``save`` vs ``fullrestore`` is because when +``up_savecontext()`` is called, it does not need to save all of registers. +Only a subset needs to be saved because the processor ABI provides that some +registers are volatile or caller-saved when ``up_savecontext()`` is called. + +The are a couple of downsides to the this approach. +First, the ``up_savecontext()`` and ``up_fullcontextrestore()`` functions +are tricky to write. Second, they have limited usage. +They can be used only in the FLAT build mode where all tasks are running with +the same privileges. If you were to try to do ``up_fullcontextrestore()`` +to get from an unprivileged task to a privileged task, you would get an +access violation exception of some sort. + +System Calls +============ + +In order to the limitations of ``up_savecontext()`` and +``up_fullcontextrestore()``, you have to do something a little differently. +One way is to use a system call (a **software interrupt**, a **trap** in x86 +or an **SVCALL** in ARM land). This generates a software interrupt and uses +the mechanization of the asynchronous context switch: The software interrupt +saves the context of the old task on entry (replacing the functionality of +``up_savecontext()``) and restores the new task context on return (replacing +the functionality of ``up_fullcontextrestore()``). +The tiny software interrupt handler just sets up the context switch. + +The ARMv7-M does synchronous contest switches in this way. You can see how +this is done in the ARMv-7M the SVCALL software interrupt handler. + +The advantages of this approach are: + +1. It is trivially easy to implement. If interrupt level asynchronous context + switches work, then so will these synchronous context switches. +2. You an switch between privileged and unprivileged tasks. + That happens for free when the interrupt returns. + +The downside is only that it causes significantly slower synchronous context +switching times. It adds the overhead of interrupt processing and interrupt +IRQ dispatching to each context switch. +I think this is still the correct way to go despite its worse performance. + +Several modifications would be required to convert from the first to the +second type of synchronous context switches: + +1. Creation of the software handlers for the context switch. +2. Converting all of the back-to-back calls to ``up_savecontext()`` and + ``up_restorefullcontext()`` to a single call to a function that executes + the software interrupt, usually something like ``up_switchcontext()``. diff --git a/Documentation/implementation/device_nodes.rst b/Documentation/implementation/device_nodes.rst new file mode 100644 index 00000000000..f506d882c0f --- /dev/null +++ b/Documentation/implementation/device_nodes.rst @@ -0,0 +1,94 @@ +.. _device-nodes: + +============ +Device Nodes +============ + +Linux Device Nodes +================== + +I used to have good Linux expertise a decade or so ago. +But my current Linux knowledge is dated and rusty. +I don't know anything about udev, SystemD, devtmpfs, sysfs, or any of that. +So this is my simplified understanding. + +Device files work quite a bit differently in Linux and NuttX. +A device node in Unix/Linux only really contains the only type of the device +and its major and minor device numbers, i.e., it just holds data. +So creating the device node does not install or create the driver; +it simply writes a tiny file containing some special data. + +Nothing happens until you try to open the device. +If something in the operating system has not initialized and registered +a driver for that type and major/minor numbers, +then you fail to open the device. + +So the device nodes and the device drivers are decoupled in Linux/Unix +and there is a rendezvous that must occur later for the device node +to actually refer to the device. + + "(..) Linux maps the device special file passed in system calls + (say to mount a file system on a block device) to the device's + device driver using the major device number and a number of system + tables, ...The major number is actually the offset into the + kernel's device driver table, which tells the kernel what + kind of device it is (whether it is a hard disk or a serial + terminal) (..)" + + -- Source: + www.linux-tutorial.info/modules.php?name=MContent&pageid=94. + +Normally, when you create a Linux file system, you also create all of the +standard device nodes. But most of these do not map to real devices. +If you try to access most of the devices under ``/dev`` in Linux, they will +fail because the underlying driver that maps to that major/minor number +has not been initialized. + + +NuttX Device Nodes +================== + +NuttX does not use major/minor device numbers and there are no device +"system tables" to associate major/minor numbers to a driver implementation. +NuttX simplifies this be removing the "man in the middle": When you register +the driver, you also create the device node. + +.. important:: The device node IS the driver registry. + This is a tremendous simplification and one of the things that + makes NuttX usable in the constrained MCU environment. + +In NuttX, device nodes are not really files at all. +They are special entries in the NuttX root pseudo-filesystem. +See :ref:`NuttX Pseudo File System ` for more details. + +Usage Differences +================= + +.. important:: Only devices drivers can create device nodes and the existence + of the device node means that the device has been initialized, + registered, and is ready for use (with the exception of some + removable devices that may not actually be ready). + + You cannot create device nodes from applications! + +You could argue that this simplification is a deviation from my Unix/Linux +roadmap and would have to agree that you are right. +But it is also the kind of enabling simplification that makes a tiny +Unix-like operating system feasible on these lower end MCUs. + +In Linux standard device drivers are initialized and registered as with NuttX. +A (privileged) application can create a device node, but cannot initialize +or register a device driver directly (as far as I know). +I believe that if you want to instantiate an uninitialized, unregistered +device driver you would have to install a kernel module containing +the driver (which would probably also create the device nodes corresponding +to the driver). + +boardctl() +========== + +NuttX does support a sneak interface to support interactions with board-level +OS logic. That sneak interface is ``boardctl()`` (see :ref:`board-ioctl` and +:ref:`nuttx-initialization-sequence` for more details). +That interface could potentially be used to force initialization of device +drivers by application code. That discussion is to be provided. diff --git a/Documentation/implementation/file_descriptors.rst b/Documentation/implementation/file_descriptors.rst new file mode 100644 index 00000000000..dfad984a347 --- /dev/null +++ b/Documentation/implementation/file_descriptors.rst @@ -0,0 +1,231 @@ +.. _file-descriptors: + +================ +File Descriptors +================ + +Drivers within Drivers +====================== + +Have you ever wanted to write a driver the uses another driver? +For example, suppose you want to develop a character driver for a module +that provides a serial interface to the host. Wouldn't you like to be able to +open and use the serial driver within your module driver? + +Or suppose the you have a device with analog inputs such as joystick +and you would like to use the ADC character driver to sample the joystick? +Would you not want to contain the ADC character driver within the joystick +driver? + + +Drivers File Descriptors +======================== + +One obstacle to containing a driver instance within another driver +is the nature of the file descriptor. When you open a file or a driver from +any thread in a task, you receive a file descriptor that you then may use +in the task to access the device. + +In the NuttX implementation, that file descriptor is really an index into +a pre-allocated array of type struct file. +It is really the underlying struct file instance referred to by the +file descriptor that use used to access the device. +The file descriptor/index is simply a portable way to reference the struct +file instance. + +.. note:: In NuttX, threads are organized into task groups, see + :ref:`Tasks vs. Threads ` for more details. + +Each task group has its own array of struct file. All of the threads within +the same task group share the same array allocation. Therefore, the same file +descriptor index is valid for all threads within the task group. + +But, on the other hand, threads running within a different task group +use a different array of struct file and file descriptors opened +in this other task group have no relationship. +In short, file descriptors cannot be used by different tasks. + +A driver must not be associated with any specific task group's resources. +That would make the driver dependent upon that task group. Any task should be +able top open and use the driver. Therefore, the driver cannot use +file descriptors to access the contained driver instance. + +So, if you cannot use file descriptors within a device drivers, how can +one driver contain another driver instance? The answer is by detaching +the file descriptor from the open file instance. + + +Detaching Open Files +==================== + +There is an internal OS interface called ``file_detach()`` that is prototyped +and described in the header file ``include/nuttx/fs/fs.h``: + +.. code-block:: c + + /**************************************************************************** + * Name: file_detach + * + * Description: + * This function is used to device drivers to create a task-independent + * handle to an entity in the file system. file_detach() duplicates the + * 'struct file' that underlies the file descriptor, then closes the file + * descriptor. + * + * This function will fail if fd is not a valid file descriptor. In + * particular, it will fail if fd is a socket descriptor. + * + * Input Parameters: + * fd - The file descriptor to be detached. This descriptor will be + * closed and invalid if the file was successfully detached. + * filep - A pointer to a user provided memory location in which to + * received the duplicated, detached file structure. + * + * Returned Value: + * Zero (OK) is returned on success; A negated errno value is returned on + * any failure to indicate the nature of the failure. + * + ****************************************************************************/ + + #if CONFIG_NFILE_DESCRIPTORS > 0 + int file_detach(int fd, FAR struct file *filep); + #endif + +In order to use ``file_detach()`` you would do the following: + +* Allocate an instance of struct file either dynamically or statically. + This instance will represent the contained driver. +* Open the file or driver to get a file descriptor +* Call ``file_detach()`` in order to clone the underlying struct file to your + allocated instance. Detaching the file descriptor has the side-effect + of closing the original instance. +* You can then use the driver methods of the contained driver directly from + your driver. These driver methods are described in ``include/nuttx/fs/fs.h``. + +There is also a ``file_close_detached()`` internal OS function that you can use +to recover resources if you no longer need the contained driver instance: + +.. code-block:: c + + /**************************************************************************** + * Name: file_close_detached + * + * Description: + * Close a file that was previously detached with file_detach(). + * + * Input Parameters: + * filep - A pointer to a user provided memory location containing the + * open file data returned by file_detach(). + * + * Returned Value: + * Zero (OK) is returned on success; A negated errno value is returned on + * any failure to indicate the nature of the failure. + * + ****************************************************************************/ + + int file_close_detached(FAR struct file *filep); + +This technique is not limited to character drivers but can also be used +with files in a file system or even block drivers. + + +Detached File Helpers +===================== + +Once the file structure has been detached from its file descriptor, +you can no longer use the standard VFS functions ``read()``, ``write()``, +``ioctl()``, etc. Fortunately, there are a parallel set of interfaces +that can be used with detached files. These are decribed in detail +in ``include/nuttx/fs/fs.h`` and only listed here below: + +.. code-block:: c + + ssize_t file_read(FAR struct file *filep, FAR void *buf, size_t nbytes); + ssize_t file_write(FAR struct file *filep, FAR const void *buf, size_t nbytes); + ssize_t file_pread(FAR struct file *filep, FAR void *buf, size_t nbytes, + off_t offset); + ssize_t file_pwrite(FAR struct file *filep, FAR const void *buf, + size_t nbytes, off_t offset); + off_t file_seek(FAR struct file *filep, off_t offset, int whence); + int file_ioctl(FAR struct file *filep, int req, unsigned long arg); + int file_fsync(FAR struct file *filep); + int file_dup2(FAR struct file *filep1, FAR struct file *filep2); + int file_vfcntl(FAR struct file *filep, int cmd, va_list ap); + + +The SYLOG Device: A Case Study +============================== + +This technique is used for the SYSLOG device. Originally, NuttX used +file descriptor ``1`` for SYSLOG output by default. For most task groups, +file descriptor ``1`` (``stdout``) mapped to ``/dev/console`` and this +solution worked well in most cases. + +But using file descriptor ``1`` caused some very strange results when +``stdout`` was redirected such as when a USB or a Telnet console was used. +In those cases, SYSLOG output would go to different places depending upon +how stdout was re-directed within a given task group. + +This was fixed using ``file_detach()``. In the default case, the SYSLOG +initialization logic now opens ``/dev/console`` then calls ``file_detach()`` +to disassociate the open file instance from any task group. +Now, SYSLOG output consistently goes to ``/dev/console`` regardless of how +file descriptor ``1`` may be re-directed when SYSLOG output is generated. + + +Other Examples +============== + +There are some other examples in analog joystick lower half drivers +that use the ADC character driver to read joystick positions:: + + boards/arm/stm32/nucleo-f4x1re/src/stm32_ajoystick.c: + ret = file_detach(fd, &g_adcfile); + + boards/arm/stm32l4/nucleo-l476rg/src/stm32_ajoystick.c: + ret = file_detach(fd, &g_adcfile); + + boards/arm/sama5/sama5d3-xplained/src/sam_ajoystick.c: + ret = file_detach(fd, &g_adcfile); + + +Socket Descriptors +================== + +There is a similar story for socket descriptors but this is probably +not the place for that whole story. Instead, we will just summarize +that story here. First, let's compare file and socket descriptors here. +Socket descriptors are similar to file descriptors in that: + +* They are a numeric index into a task-private array of structures. + For the case of socket descriptors, this is an array of type struct socket. +* Like the file descriptor, the socket descriptor is simply a portable way + to reference the underlying socket structure. +* And also like the file descriptor, the socket descriptor is meaningful + only in the context of task group where the socket descriptor was created. + +As in the case for file descriptors, there are also a set of socket interfaces +that accept a reference to struct socket as an input parameter. +For each standard socket interface that accepts a socket descriptor, +there is a corresponding non-standard, OS-internal interface that accepts +a reference to struct socket as as an input parameters. These low-level socket +interfaces all have the prefix ``psock_`` and are all prototyped in the file +``include/nuttx/net/net.h``. They are not discussed further here. + +These low-level socket interfaces are also independent of task groups +and may be used without regard to the thread that created the socket. +The primary usage difference is in how the struct socket instances are created: + +* In the case of struct file, there is no special ``file_open()`` interface + to create the detached file reference. Instead, a two step procedure is used: + First the file is opened the standard ``open()`` function to obtain the + file descriptor, then the struct file is detached from the file descriptor + using ``file_detach()``. +* The struct socket instance, on the other hand, is created in the detached + state directly using ``psock_socket()``. + +You can find a good example of a character driver that contains +a task-independent socket interface in the Telnet character driver at +``driver/net/telnet.c``. That driver is used to encapsulate a Telnet +session and to provide Telnet ``stdin``, ``stdout``, and ``stderr`` +for remote NSH sessions. diff --git a/Documentation/implementation/hardfaults.rst b/Documentation/implementation/hardfaults.rst new file mode 100644 index 00000000000..c72c8bd80ef --- /dev/null +++ b/Documentation/implementation/hardfaults.rst @@ -0,0 +1,144 @@ +.. _hardfaults: + +========== +Hardfaults +========== + +ARMv7 +===== + +Cortex-M3 and Cortex-M4 +----------------------- + +The most popular CPUs in current MCU designs are the Cortex-M3 (ARMv7-M) and +the Cortex-M4 (ARMv7E-M). Handling of these two architectures is almost +identical in NuttX (unless hardware floating point is enabled). + +SVCALL +------ + +NuttX uses the SVCALL software interrupt in order to perform certain steps +in the context switching for the Cortex-M3 and Cortext-M4. +This sequence of logic appears in several places: + +* Create a short critical section by disabling exceptions, +* Perform some set-up, +* Initiate the software exception / SVCALL, and +* When the software exception processing returns, re-enable exceptions. + +.. note:: There is a technical difference between interrupts and exceptions. + In this section the term exception will be used. It is probably + more accurate since interrupts are really exceptions that result + from device interrupt lines. Furthermore, when we refer to + exceptions in this section, we are referring specifically to + ARMv7-M configurable exceptions. + +Disabling Interrupts via the PRIMASK register +--------------------------------------------- + +The ARMv7-M architecture supports a register called the ``PRIMASK`` register. +The ``PRIMASK`` register contains a single valid bit. If that bit is set to +one, then exceptions are disabled. If that bit is zero, exceptions are enabled. +More correctly, when this bit is set to one, it prevents the activation of all +exceptions with configurable priority. + +The original NuttX implementation used this ``PRIMASK`` register to enable +and disable exceptions. Things now get interesting in the sequence of logic +listed above because the ``PRIMASK`` bit also disables the SVCALL exception! +So, instead of taking the ``SVCALL`` exception vector, the Cortex-M3/4 +generates a hardfault exception (see ARM.com's discussion of +`Activation Levels `_). + +These hardfaults are not really a problem. The design of the NuttX hardfault +handler expects these exceptions and does the right thing. +However, the occurrence of hardfaults may come as a surprise to many people +and especially to some debuggers. + +Hardfaults and Debuggers +------------------------ + +These hardfaults only become a technical issue when dealing with a debugger. +What does the debugger do when the hardfault occurs? + +We need to back off and think about some system philosophy here. +The deep philosophical question here is: Who is in charge of system integrity? +The debugger or the RTOS? + +If you are running a primitive NoOS program (like the famous blinky test +program), then you are running a barebones system and you need +all of the help you can get. So having the debugger make decisions about what +is the proper behavior of the blinky program and what is not is a good +thing for you. + +But if you are using an advanced RTOS, then the RTOS will want to take +responsibility of the health of your system and now there is the possibility +of inconsistencies between the decisions that the RTOS makes and the decisions +that your debugger makes this hardfault handling is a perfect example here. + +In NuttX, the hardfaults are controlled by the RTOS, but some debuggers will +break when the hardfault occurs and make debugging impossible. + +Some people might take issue with this. Breaking on hardfault can make +debugging easier, since the break happens in the throwing context and you have +a hope of obtaining a backtrace and debugging the throwing side callstack. +However, I would suggest that putting a break point on ``up_assert()`` would +accomplish the same thing without being so intrusive. + +How did the debugger know that the hardfault occurred? It knew because of +settings in the "ARM's Debug Exception and Monitor Control Register" or +``DEMCR``. Proper settings of the ``DEMCR`` register will allow debugger +to get break exceptions when a hardfault occrs. +So one workaround is to just reconfigure the ``DEMCR`` register so that break +exceptions are no longer generated when hardfaults occur. + +Here is an example of such logic for the LPC43xx MCU. +Decoupling the hardfault from the break exception in this way does not work +with all debuggers, however. Presumably because some debuggers re-enable break +exceptions on hardfaults. + +Disabling Interrupts via the BASEPRI register +--------------------------------------------- + +The ARMv7-M architecture supports another way to disable exceptions using +a register called the ``BASEPRI`` register. ARMv7-M exceptions are prioritized. +Each exception can be assigned an 8-bit priority. If the ``BASEPRI`` register +is set to a non-zero priority value, then it will filter exceptions in this sense: + +* Exceptions with priority lower than or equal to the ``BASEPRI`` register + will be disabled. +* Exceptions with priority higher than the ``BASEPRI`` register will still + be enabled. + +Normally this interrupt prioritization is used to support nested interrupt +handling, but it can also be used for disabling of all exceptions +if configured properly. + +.. note:: In the ARMv7-M, higher values correspond to lower priority. + This can be really confusing! + +NuttX supports a configuration option called ``CONFIG_ARMV7M_USEBASEPRI``. +If this option is selected, then the exception prioritization and control +logic will be configured to use the ``BASEPRI`` register instead of the +``PRIMASK`` register to disable exceptions. +This configuration includes the following changes in the behavior: + +* Normal interrupts and exceptions are restricted to the range + ``{ lowest priority ... (highest priority - 1) }``. +* The priority of the ``SVCALL`` exception is set to highest priority. +* When exceptions are enabled, the ``BASEPRI`` register is set to zero, + enabling exceptions of all priorities. +* When exceptions are disabled, the ``BASEPRI`` register is set to + ``(highest priority - 1)``, disabling all exceptions except for the + ``SVCALL`` exception. + +In this way, the ``SVCALL`` exception remains enabled when exceptions +are disabled and no hardfault occurs. + +.. note:: The above is inaccurate on several counts. It was simplified + to make the discussion sane. Not only do higher values correspond + to lower priorities, but the increment between consecutive 8-bit + priority values is probably not one. The supported maximum and + minimum priority values (as well as the step in each priority value) + may be different for each MCU. To handle this, these values are + exported in NuttX for each ARMv7-M architecture by header files at + ``nuttx/arch/arm/include//chip.h``. diff --git a/Documentation/implementation/index.rst b/Documentation/implementation/index.rst index b22b452a9ab..b05cffad841 100644 --- a/Documentation/implementation/index.rst +++ b/Documentation/implementation/index.rst @@ -6,12 +6,36 @@ Implementation Details :maxdepth: 2 :caption: Contents: - make_build_system.rst - drivers_design.rst - device_drivers.rst - processes_vs_tasks.rst - critical_sections.rst - interrupt_controls.rst - preemption_latency.rst bottomhalf_interrupt.rst + cancellation_points.rst + chip_h.rst + context_switches.rst + critical_sections.rst + device_drivers.rst + device_nodes.rst + drivers_design.rst + file_descriptors.rst + hardfaults.rst + interrupt_controls.rst + ioctl.rst + kernel_modules_vs_shared_libraries.rst + kernel_threads_vs_pthreads.rst + make_build_system.rst + memory_configurations.rst + naming_arch_mcu_board_interfaces.rst + naming_os_internals.rst + nuttx_initialization_sequence.rst + nuttx_tasking.rst + oneshot_timers_and_cpu_load.rst + power_management.rst + preemption_latency.rst + processes_vs_tasks.rst + short_time_delays.rst + signal_handlers.rst simulation.rst + smp.rst + syslog.rst + tasks_vs_threads.rst + tickless_os.rst + tls.rst + usb.rst diff --git a/Documentation/implementation/ioctl.rst b/Documentation/implementation/ioctl.rst new file mode 100644 index 00000000000..b33427c2646 --- /dev/null +++ b/Documentation/implementation/ioctl.rst @@ -0,0 +1,137 @@ +.. _ioctl: + +===== +ioctl +===== + +Not a Typewriter +================ + + "In computing, **Not a typewriter** or ``ENOTTY`` is an error code defined + in the ``errno.h`` found on many Unix systems. + This code is now used to indicate that + an invalid ``ioctl`` (input/output control) number was specified + in an ``ioctl`` system call." + + --Source: https://en.wikipedia.org/wiki/Not_a_typewriter. + + +NuttX +===== + +In ``ioctl()`` implementations in NuttX, ``-ENOTTY`` is always returned +if the ``ioctl()`` command is not recognized. You will often see driver +``ioctl()`` implement ions with a general structure similar to the following: + +.. code:: c + + int driver_ioctl(FAR struct file *filep, int cmd, unsigned long arg) + { + int ret; + + switch (cmd) + { + ... + + default: + ret = -ENOTTY; + break; + } + } + + return ret; + } + +Note that ``-ENOTTY`` is returned internally in NuttX. +This will subsequently be used to set the errno value to ``ENOTTY`` +and to ``return -1`` to indicate the error condition. + +ERRORS +------ + +These are the return values from a Linux ``ioctl()`` call: + +* ``EBADF`` fd is not a valid file descriptor. +* ``EFAULT`` argp references an inaccessible memory area. +* ``EINVAL`` request or argp is not valid. +* ``ENOTTY`` fd is not associated with a character special device. +* ``ENOTTY`` The specified request does not apply to the kind of object + that the file descriptor fd references. + +Reference: https://www.man7.org/linux/man-pages/man2/ioctl.2.html. + + +Linux Explanation +================= + +On Jun 27 Linus Torvalds wrote: + + "The correct error code for "I don't understand this ioctl" is ENOTTY. + The naming may be odd, but you should think of that error value as a + "unrecognized ioctl number, you're feeding me random numbers that I + don't understand and I assume for historical reasons that you tried to + do some tty operation on me". + ... + The EINVAL thing goes way back, and is a disaster. It predates Linux + itself, as far as I can tell. You'll find lots of man-pages that have + this line in it: + + EINVAL Request or argp is not valid. + + and it shows up in POSIX etc. And sadly, it generally shows up + _before_ the line that says + + ENOTTY The specified request does not apply to the kind of object + that the descriptor d references. + + so a lot of people get to the EINVAL, and never even notice the ENOTTY. + + (..) + + At least glibc (and hopefully other C libraries) use a _string_ that + makes much more sense: strerror(ENOTTY) is "Inappropriate ioctl for + device"." + + --Source: https://lore.kernel.org/patchwork/patch/258361. + + +How is this useful? + +Knowing that no error occurred but the ``ioctl()`` command was not recognized +is a useful piece of information. +Suppose, for example, I have nfds open character drivers in an array ``fd[]``. +Then I could do something like this: + +.. code:: c + + int do_command(FAR int *fd, int nfds, int cmd) + { + int ret; + int i; + + /* Try all file descriptors */ + + for (i = 0; i < nfds; i++) + { + ret = ioctl(fd[i], cmd, 0ul); /* No argument in this example */ + if (ret < 0) + { + int errcode = errno; + + /* Try the next file descriptor if this one return ENOTTY */ + + if (errcode != ENOTTY) + { + /* Other errors, including EINVAL, are fatal */ + + return -errcode + } + } + else if (ret >= 0) + { + return OK; /* Success! */ + } + } + + return -ENOENT; + } diff --git a/Documentation/implementation/kernel_modules_vs_shared_libraries.rst b/Documentation/implementation/kernel_modules_vs_shared_libraries.rst new file mode 100644 index 00000000000..73f5cd60158 --- /dev/null +++ b/Documentation/implementation/kernel_modules_vs_shared_libraries.rst @@ -0,0 +1,176 @@ +.. _kernel-modules: + +================================== +Kernel Modules vs Shared Libraries +================================== + +Kernel Modules +============== + +NuttX has had support for Kernel Modules for some time. +A kernel module allows you to extend the functionality of the OS +at runtime by installing ELF modules into the kernel. +A kernel module might be used, to example, to load device drivers +into RAM at runtime. + +Here are some general properties of kernel modules: + +* The first kernel module is loaded into the kernel address space + by ``insmod()`` using only a symbol table exported by the OS. +* Kernel modules can also (optionally) export a symbol table. + Such symbol tables are remembered and will be used by ``insmod()`` + to resolved undefined symbols in subsequently loaded modules. +* This requires dependency checking logic: A module that imports symbols + from another module must be added after the modules that it depends upon. + And a module exports a symbol may not be removed while there are such + inter-module dependencies in place. + A module that imports symbols from another module must be removed before + the module that exports the symbol can be removed. +* There is a (non-standard) OS interface ``modsym()`` that will allow kernel + logic to look up symbols within a kernel module. +* Handles are used at the kernel module interfaces: ``insmod()`` returns + a handle to the module data structure. That handle can subsequently be used + to retrieve symbols with ``modsym()``. +* ``rmmod()`` uses the handle to remove the module. +* ``modhandle()`` is also available for backward compatibility: + Given the assigned module name, you can use this to retrieve + the module handle at any time. +* There is a test case at ``nuttx-apps/examples/module``. + +In the FLAT build, ELF kernel modules are simply loaded into RAM and linked +with the base firmware. But things get a little more complex with PROTECTED +and KERNEL builds. + +In those case, there are separate address spaces for the kernel +and for applications. +Kernel modules are only loaded in the kernel address space and, hence, +are not available to applications. + + +Shared Libraries +================ + +A shared library is another software module with these properties: +The ``.text`` address space is accessible to all applications. +The ``.data`` and ``.bss`` address space is in the same address space +as the application that uses the shared library. +So, they are shared in the sense that the ``.text`` is shared. + + +FLAT Build +========== + +In the FLAT build environment, there is only one address space. +So what is the difference between a kernel module and a shared library +in this case? Certainly a kernel module meets all of the requirements +of a shared library in that environment. +In this case kernel modules really only differ from shared +libraries in their usage semantics: + +For the FLAT build, I have added the standard ``include/dllfcn.h`` +and have implemented the FLAT shared library support as a thin wrapper +around the kernel module support: + +* ``dlopen()`` maps to ``insmod()``. +* ``dlclose()`` maps to ``rmmod()``. +* ``dlsym()`` maps to ``modsym()``. +* ``dlerror()`` is only a stub at the present time. + +There is a shared library test case at ``nuttx-apps/examples/sotest``. + + +PROTECTED Build +=============== + +The PROTECTED build is equivalent to the FLAT build except that there are +two address spaces: The kernel address space and the user address space. +But all applications still share the same user address spaces. + +As a result ``.text`` along with ``.data`` and ``.bss`` are naturally shared. +This requires using two copies of the the module logic: One residing in kernel +address space and using the kernel symbol table and one residing in user space +using the user space symbol table. +The first provides only kernel module support; the second only PROTECTED mode +shared library support. + +This is accomplished by breaking the kernel module logic in two components +with OS kernel module interfaces in sched/module but with a sharable module +library at libc/modlib. + +The shared library functions no longer call the kernel module logic but rather +implement their one top-level management logic using the lower-level routines +in the module library. + + +Better FLAT and PROTECTED Mode Shared Libraries +=============================================== + +A better implementation of shared libraries in the FLAT and PROTECTED builds +would, however, have a separate copy of the ``.bss`` and ``.data`` region +for each NuttX task group. + +A task group is the moral equivalent of a Unix process. +That is how a shared library would have to work in uClinux, for example. +But that would be a substantial effort! For example, since each +``.bss``/``.data`` would lie at a different physical addres, +the ``.text`` section logic would need support +Position-Independent-Data (PID). +Embedded PID support, however, is pretty much broken on all current GCC +implementations. See NxFlat Compatibility Problem. + +Perhaps the xFLAT work that I did for uClinux shared libraries could be +ported to Nuttx: `xFLAT Web Page `_. + +For more information about task groups see :ref:`nuttx-tasking` +or :ref:`tasks-vs-threads`. + +The current implementation also assumes that all firmware resides in base +FLASH and hence is fully linked prior to loading any modules or shared +libraries. There is no support for loading programs into RAM and binding +them to symbols exported by modules or shared libraries. + +This extension would, however, not be so difficult for the FLAT build; +it would be a simple matter of integrating the exported module symbol +tables into the symbol lookup. +That is already done in ``sched/module`` files, but not yet in the +very similar ``binfmt/libelf`` files. + +In the PROTECTED build, this would require some special start-up logic +in the user address space as the initial steps of the newly started task. +Some kind of dynamic loader, such as ``ld.so``, would have to integrate +with ``crt0`` logic to automatically bind user space tasks to shared libraries +as they are loaded into and memory before the programs ``main()`` function +is called. + + +KERNEL Build +============ + +The KERNEL build, however, is a completely different creature. +In that build, the kernel and each process has its own adress space. + +This means that a shared library in the kernel build has to be considerably +more complex: In order to be shared, the ``.text`` portion of the module +(1) must lie in a single shared memory region accessible from all processes +and (2) built for Position-Independent-Code (PIC) operation since +it must execute from an arbitrarily different virtual address in each process +address space. + +The ``.data``/``.bss`` portion of the module must be allocated +in the user address space of each process, but either: + +1. ``.data``/``.bss`` section must lie at a consistent virtual address + so that it can be referenced from the one copy of the ``.text`` in + the shared memory region, OR +2. The ``.text`` section logic must support Position-Independent-Data (PID). + The latter approach provides for a simpler build, but embedded PID support + is pretty much broken on all current GCC implementations. + See :ref:`nxflat` Compatibility Problem. + +Some kind of dynamic loader, such as ``ld.so``, would have to integrate with +``crt0`` logic to automatically bind processes to shared libraries as they +are loaded into memory before the programs ``main()`` logic is called. + +.. note:: There is not yet any shared library support in the KERNEL build mode. + This would be quite a large effort and not on the plan of record + at the present time. diff --git a/Documentation/implementation/kernel_threads_vs_pthreads.rst b/Documentation/implementation/kernel_threads_vs_pthreads.rst new file mode 100644 index 00000000000..432acf3334b --- /dev/null +++ b/Documentation/implementation/kernel_threads_vs_pthreads.rst @@ -0,0 +1,191 @@ +.. _kernel-threads-vs-pthreads: + +=========================== +Kernel Threads vs. Pthreads +=========================== + +Why Can't Kernel Threads Have pthreads? +======================================= + +Kernel threads are special "tasks" that reside within the OS. +They are similar to application tasks, so why can't they have pthreads +like application tasks? + + +The FLAT build +============== + +In the FLAT build, kernel threads are, in fact, identical to application +threads except that: + +1. They follow some slightly different syntax, AND +2. Can only use OS internal interfaces, never application interfaces. + +Since they are otherwise identical, there is really no technical reason +why pthreads could not be supported. + +The real reasons in this case is: + +1. It is inappropriate, AND +2. It is incompatible with the PROTECTED and KERNEL builds where pthreads + cannot be supported. + +Why inappropriate? Because.. + +pthread Interfaces Are User Interfaces +-------------------------------------- + +In all Unix systems, pthread interface support is not provided +by the operating system but rather by the user-facing C library. +The role of the OS is only to provide some low level hooks needed by +C library implementation of pthread interfaces. +But all of the implementation is in the C library and only for use +by applications. + +That is not the current situation in NuttX. +Currently pthread support is deeply entangled in the OS. +This is problem that must be fixed someday and must not exploited as some +permanent feature of the OS. It is not. +It will go away some day when NuttX becomes a mature Unix-family system. + +As user facing interfaces, pthread interfaces include some behaviors that +are not appropriate within the OS. +Inappropriate behaviors include modification of the ``errno`` value and +cancellation points. Those are user-only features. +While pthread interfaces do not, in general, modify the errno setting +they do create cancellation points which is not desirable within the OS. + +PROTECTED and KERNEL Builds +--------------------------- + +The PROTECTED and KERNEL builds differ from the FLAT build in that they +segregate the memory into two regions: +A privileged kernel address space and unprivileged, user address space(s). + +The primary difference is that the PROTECTED build uses a physical address +space and different regions of the physical address space have different +properties. This is usually accomplished using a Memory Protection Unit (MPU). +There is one protected kernel address space and one unprotected +user address space. + +The KERNEL build, on the other hand, uses a Memory Management Unit (MMU) +to create a virtual address space in which there is still a separate +protected kernel address space, but many user address spaces for user +programs. These are usually called processes. +This is the familiar build model that you find with high-end Unix-like +systems such as Linux. + +See Memory Configurations for additional information. + +Address Spaces, Memory Allocators, and User Mode +------------------------------------------------ + +What would happen if you tried to create a pthread in the PROTECTED or KERNEL +build? The effect of these address space differences become very pronounced. +pthreads are, by their very definition, user-space threads. +That means that the OS will attempt to create a user-space environment +for the pthread: The thread's stack and other resources. + +And when the pthread is started, it will be started in user mode, +not kernel mode. + +It would require a significant change to the OS to alter that behavior +and that is not under consideration (because the change is also inappropriate). + +Entry points +------------ + +Because the application space and the kernel space are separately built +and separately linked in the PROTECTED and KERNEL builds. +No application addresses are known by the kernel and no kernel addresses +are known by the application (applications interface via system call traps, +not C function calls). So what address would the kernel thread provide +for the pthread entry point? A known kernel space address? + +Of course, the system would crash with a memory fault immediately +if a protected address were executed in user mode. + +Mutexes +------- + +Okay, so there are no kernel pthreads. But could other pthread resources +be usable in the kernel. The short answer is no. + +Consider mutexes, for example. By definition, a mutex can only be used within +threads of the current process (or task groups as they are often called +in NuttX). They have no meaning outside of the task group. + +Since the kernel thread "task group" can have no pthreads, +how could these mutexes be used under the proper standard definition +of what a mutex must be? + +This is true of all pthread interfaces: None of the pthread interfaces were +intended from inter-process communications; only for inter-pthread +communications within the same process (task group). + + +Roadmap +======= + +As alluded to before about the appropriateness of pthread in the OS. +There is a roadmap for these kinds of features. +That roadmap is to continue to conform strictly to the standard OS definitions +of _ and to continue to evolve as a fully compliant, +very standard, tiny Unix-like operating system. + +1. Part of this is getting all user interfaces out of the OS. +2. Another part is migrating all pthread support out of operating system + and into the user-facing C library. +3. An additional objective on the roadmap is to streamline the kernel threads + and to disconnect them from task groups. + Kernel threads should not be part of any task group. + +In regard to this last point, the following is taken from Apache NuttX +Issue 1108: "Remove streams from Kernel Threads". + +No Streams in Kernel Threads +---------------------------- + +Kernel threads are not permitted to use the C library buffer I/O, "stream", +interfaces. Those are interfaces like ``fopen()``, ``fread()``, ``fwrite()``, +``fclose()``, etc. +Those are strictly for use by user applications. +This is because these functions modify errno variables and create +cancellation points and perhaps other things that are undesirable +within the OS. + +The thread create logic is largely the same for both kernel threads +and for application threads: They both allocate a large buffer +from the user memory pool for the stream ``FILE`` array. +Since kernel threads this is a waste of memory since the kernel threads +should not be using streams. + +Remove Kernel Thread Stream Allocation +-------------------------------------- + +The proposal is to: + +1. Verify that there is no use of streams within the OS, +2. Remove the stream allocation for kernel threads, +3. Assure that there are proper checks in place so that there are + no uses of the ``NULL`` stream array pointer. + +For the most part, the OS is clean, there are essentially no use +of streams within the OS. +There are, however, a few violations of this that will need to be fixed; +``fopen()`` is called in some of the ``lc3850`` code. + +Remove the File Descriptor Array Too +------------------------------------ + +A second phase would be to remove the file descriptor array from kernel +threads. File descriptors are, again, only for use by applications; +within the OS, file access is done using the struct file (aka, ``FILE``), +structure directly. +However, I suspect that there are many hidden uses of file descriptors +in the system. +For one, ``file_open()`` which opens the detached file, is not fully +implemented; it cheats and uses file descriptors. + +So let's consider removal of the file descriptor allocation +as a second step after the stream allocations have been removed. diff --git a/Documentation/implementation/memory_configurations.rst b/Documentation/implementation/memory_configurations.rst new file mode 100644 index 00000000000..6ad536df5d6 --- /dev/null +++ b/Documentation/implementation/memory_configurations.rst @@ -0,0 +1,1000 @@ +.. _memory-configurations: + +===================== +Memory Configurations +===================== + +.. _flat-build: + +Flat, Embedded Build +==================== + +The normal build of NuttX for the typical embedded environment uses +a single blob of code in a flat address space. +For most lower end CPUs (such as the ARM Cortex-M family), +this means executing directly out of the physical address space. + +Even if the CPU has an MMU (such as with the ARM Cortex-A family), +the typical NuttX build still uses a flat address space with the MMU +providing only an identity mapping. + +In this case, there is still benefit from using the MMU because +the MMU provides fine control over caching and memory behavior +over the address space. + + +.. _on-demand-paging: + +On-Demand Paging +================ + +NuttX also supports on-demand paging via ``CONFIG_PAGING``. +On-demand paging is a method of virtual memory management and requires +the the CPU architecutre support a MMU. + +In a system that uses on-demand paging, the OS responds to a page fault +by copying data from some storage media into physical memory and setting up +the MMU to provided the necessary virtual address mapping. +The CPU can then continue from the page fault with the necessary memory +in place for the virtual address. + +Execution Image +--------------- + +The execution image is still built as one blob and appears as one blob on the +storage media. But the execution image is paged into arbitrary physical +addresses with non-contiguous virtual addresses. +The physical and virtual address spaces are then "checker boards" +of memory in use. + +Advantages +---------- + +The main advantage of on-demand paging is that you can execute a single +program that is much larger than the physical address space or a collection +of programs that together are much larger than the physical address space. + +Current Implementation +---------------------- + +On-demand paging is currently implemented only for the NXP LPC31xx family. +The LPC31xx has a 192KiB internal SRAM and with on-demand paging the LPC31xx +can execute a huge program residing in SerialFLASH by bringing in new pages +as needed from the SerialFLASH when ``_page`` ``fault_s`` occur. + + +.. _protected-build: + +Protected Build +=============== + +Protected Build Mode +-------------------- + +NuttX also supports a protected build mode for certain CPU architectures +if ``CONFIG_BUILD_PROTECTED`` is selected. + +**In this mode, NuttX is built as two blobs, one privileged and +one unprivileged.** The privileged blob contains the RTOS, and the other, +unprivileged blob holds all of the applications. + +The build supports system calls via a call gate so that the unprivileged, +application code can access the privileged RTOS services. + +Memory Protection +----------------- + +Within each blob, the address space is flat. No MMU is required to support +the protected build since no address mapping is performed. + +In fact, this feature is currently available only for the ARM Cortex-M family. +In this case the Cortex-M's MPU provides the security in the address spaces +of the two blobs. + +This feature could also be implemented with a CPU that supports an MMU, +but there has thus far been no reason to implement such a configuration. + +Dynamic Memory Allocation +------------------------- + +The purpose of protected build then is focused primarily on securing the OS +and CPU resources from potential rogue applications. + +The MPU simply protects the hardware, code regions, and data regions +of the RTOS. But dynamic memory allocations become more complex. +Protection is also required for (certain) memory allocations made by the RTOS. +The RTOS must also be capable of allocating memory that is accessible +by user applications (such as the user thread stacks). + +Dual Heaps +---------- + +In systems with MMUs, the privilege of each page of memory can be controlled +and there are established architectures for memory management of processes +(see below). However, with only an MPU with a limited number of pages +(the Cortex-M has 8 pages only!) we are forced to resolve this problem +by dividing available heap memory into two heaps: +a privileged heap and an unprivileged heap, using different allocations +mechanisms for each (kmalloc and malloc, respectively). + + +.. _addrenv: + +Address Environments +==================== + +If the option ``CONFIG_ARCH_ADDRENV`` is selected, then NuttX will support +address environments in the following way: the base code is still one blob +and identical in every way to the "Flat Embedded Build" discussed above. +But all applications are loaded into RAM from executable files, +separately compiled and separately linked programs, that reside +in a file system. + +Instead of starting the user application at a fixed, in-memory address +(such as ``nsh_main()``), the system will start the program contained +in an executable file, given its path. + +That initial user program can then start additional applications +from executable files in a file system. + +Per Program +----------- + +As each program is started, a new address environment is created +for the new task. This address environment is then unique for each task. +A task does not have the capability to access just anything in the address +environment. A task may only access addresses within its own address space +and within the address space of the base code. + +MMU (Memory Management Unit) +---------------------------- + +The CPU must support an MMU in order to provide address environments. + +This feature was originally implemented to support the ZiLOG Z180 which +is an 8-bit CPU (basically a Z80) that also supports a simple MMU. +Specifically for the P112 platform. Unfortunately, due to complex tool +issues and fading interest, that port was fully implemented but never tested. + +As of this writing, the implementation of address environment support +for the Cortex-A family is complete and verified. +An example configuration is available at +``nuttx/boards/arm/sama5/sama5d4-ek/configs/elf``. + + +.. _kernel-build: + +Kernel Build +============ + +The Kernel Build +---------------- + +And finally, there is the kernel build that is enabled with +``CONFIG_BUILD_KERNEL=y``. +The NuttX kernel build mode is similar to building with address environments: + +* Each application process executes from its own, private address environment. + +But, in addition, there are some features similar to the protected build mode: + +* NuttX is built as a monolithic kernel, similar to the way that NuttX + is built in the Protected Build Mode. +* All of the code that executes within the kernel executes in privileged, + kernel mode. Again, this is analogous to the Protected Build Mode. +* All user applications are executed with their own private + address environments in unprivileged, user-mode. + +MMU Required +------------ + +In order to support this kernel build mode, the **processor must provide +a Memory Management Unit (MMU)**. The MMU is used to provide both the address +environment for the user application as well as to enforce +the user-/kernel-mode privileges. + +This kernel build feature has been fully implemented and verified +on the Cortex-A family of processes. +A functioning example can be found at +``nuttx/boards/arm/sama5/sama5d4-ek/configs/knsh``. + +Process Environment +------------------- + +Such user applications that execute their own private, unprivileged address +environments are usually referred to as processes. + +The ``CONFIG_BUILD_KERNEL=y`` build is the first step toward support +for processes in NuttX. + + +The Roadmap Toward Processes +============================ + +Processes +--------- + +Let's call these programs with their own address spaces processes. +The term process may bring along some additional baggage and imply more than +is intended, at least in the short term. +Conceptually, processes are a natural extension of what is referred +as a task group in NuttX parlance. + + +Binding to the Base Code +------------------------ + +So how does the process communicate with the base code? +There are two ways now. Both are fully implemented, either would work: + +Symbol Tables +------------- + +In this case, the object file is only partially linked. It contains references +to undefined external symbols. In this case, the base code must provide +a symbol table, that is, a table that maps a symbol to its address +in the base code. The NuttX dynamic loader (``binfmt/``) will automatically +link the the program to base code using this symbol table when the program +is loaded from the file system into RAM. + +Symbol Table Helpers +^^^^^^^^^^^^^^^^^^^^ + +NuttX provides several helpers to make dealing with symbol tables exported +from the base code less painful. + +All base code symbols are maintained in comma-separated-value (CSV) files. +There are three: ``nuttx/libc/libc.csv``, ``nuttx/libc/math.csv``, and +``nuttx/syscall/syscall.csv``. +These CSV files contain descriptions of all symbols that could be exported +by the base code. +Then there is the program at ``nuttx/tools/mksymtab.c`` that can be used +to generate symbol tables from these CSV files. + +Call Gates +^^^^^^^^^^ + +A second way for the program loaded into memory to communicate with the +base code is via a call gate, i.e., via system calls. + +A call gate is normally used to change the privilege level of the thread +when calling into system code. +However, the same system call mechanism can be used to simply call into +the OS without having any a priori knowledge of the address of the RTOS +service in the base code. +Thus, the symbol tables could be eliminated completely at least for the case +of OS system calls (the C library is another story). + +Status +^^^^^^ + +Both the symbol table logic and the system call logic are already fully +implemented and verified and fully integrated with address environments +up to this point. + +A sample configuration that does all of these things is here: +``nuttx/boards/arm/sama5/sama5d4-ek/configs/elf`` and additional information +is available in ``nuttx/boards/arm/sama5/sama5d4-ek/README.txt``. + + +Protection and Privileges +------------------------- + +Protection +^^^^^^^^^^ + +One of the greatest benefits of processes, however, is the security that they +can provide. As described above, on program cannot access memory resources +of any other program because those resources lie in a different address +environment. + +But none of the address space outside of the program private address +environment is protected: +The base code and its private memory are not protected; +the hardware is not protected; +and anything allocated from the heap is not protected. + +So a misbehaving, rogue program can still crash the system or corrupt +the stack or memory allocation made by other programs. + +MMU Protection +^^^^^^^^^^^^^^ + +Of course, the MMU can also be used to protect the resources +outside of programs address environment. It would be a simple matter +to protect the hardware and the base code memory resources from programs. + +The base code would run in privileged mode with full access; +the user applications would run in unprivileged mode and only have the ability +to access memory resources within their own address environment. +This use of the MMU, however, raises some additional issues: + +No Symbol Tables +^^^^^^^^^^^^^^^^ + +Symbol Tables could not be used in such a protected environment to call +into the base code. Only system calls could be supported. These call gates +could switch to privileged mode temporarily in order execute the RTOS service, +then return to unprivileged mode when returning to the program. + + +The Line +-------- + +.. attention:: Let's draw a line right here. + +Everything above this point is history and just a summary of the way +things are. Every thing below this point is a roadmap that I will be +following in the further development of these features. +Some bits and pieces and been implemented and indicated. + +So continuing from the other side of this line... + + +File Descriptors +---------------- + +In the FLAT and PROTECTED builds, the file descriptors are maintained +in a table internal to the OS. + +In PROTECTED mode, this requires a system call to access the file descriptors. +In the KERNEL build mode, however, this system call overhead is wasteful. +Ideally, the file descriptor table would be moved out of the OS and into the +process address space where it can be accessed directly. + + +Overlapped Address Spaces +------------------------- + +Currently, the user-process virtual address space and the kernel-mode +virtual address space do not overlapped. +This is an odd arrangement and forces the user address space into awkward +regions. This was done so that a single address environment can support +both user- and kernel- mode operation. + +More correctly, the user address space should include the entire virtual +address space (other that regions that may have specific hardware +functionality such as vector tables). +And the user address space should overlap the kernel address space. + +Supporting such overlapping address spaces would require to changes +to the currently MMU handling: + +* First, on entry into a kernel mode system call, MMU mapping of the user + address space must be disabled so that the kernel address space + is accessible, and +* When pointer parameters are passed to the OS, these will be references + to user space data. The user-address space must be re-established prior + to a accessing the user-space data passed with the system call. +* Care must be taken in general when interacting with any user-space + resource, memory, callbacks, etc., to assure that the correct address + environment is in place. Many places now assume that that is true. + + +Libraries +--------- + +But not all of the symbols that might be exported from the base code map +to system calls. Many of the NuttX facilities operate simply in user mode: +Think of ``strlen()``, ``printf()``, ``rand()``, etc. + +With this strict enforcement of address spaces, the only way that these +addition functions can be called is if they are brought into the same +address space as the program. + +NuttX Libraries +^^^^^^^^^^^^^^^ + +User programs are separately compiled and separately linked. +NuttX currently builds all of the user callable functions into static +libraries. So, as part of their build process, user programs can simply +link with these static libraries to include to bring the callable code into +the address space of each program. + +Libraries created by NuttX include: ``libsyscall.a`` that holds the system +call "proxies", ``libc.a`` that holds the NuttX C library, +and ``libnx.a`` that hold the graphics interface library. (FULLY implemented). + +Shared Libraries +^^^^^^^^^^^^^^^^ + +The downside to using static libraries like this is that a function +in the static libraries would be duplicated in the address environment +of every process that uses that function. Some very commonly used functions, +such as ``printf()``, can be quite large and the penalty for duplicating +``printf()`` in every process address environment could be a significant +problem. + +The solution is to used shared libraries, that is, libraries +of functions that: + +1. Have only one ``.text`` space in RAM, but +2. Separate ``.data`` and ``.bass`` space, and are +3. Separately linked into with the program in each address environmnet. + +(not implemented). + +Dynamic Loader +^^^^^^^^^^^^^^ + +Such shared library support would be a significant yet natural extensions +to the existing NuttX dynamic loader.(not implemented). + + +Partially Linked vs Fully Linked Objects +---------------------------------------- + +Partially Linked Objects +^^^^^^^^^^^^^^^^^^^^^^^^ + +The NuttX ELF loader currently accepts partially linked objects, that is, +programs that do not have the final addresses of functions and data resolved. + +Instead, these ELF program files include relocation and symbol information +inside them. The NuttX ELF loader then accesses this relocation information +and resolves all of the function and data addresses when the program +is loaded into memory. (FULLY implemented). + +Two things to note about this approach: + +1. First, note that retaining the relocation information in the ELF program + files makes the partially linked object files much bigger than necessary + because they have to carry all of this relocation and symbol information + along with all of the code and data. +2. A second thing to note is that every in-memory representation is unique; + each is a one-off, resolved version of the partially linked object file. + Each might have been relocated differently. + A consequence of this is that if there are multiple copies of the same + program running, all of the code must be duplicated in memory. + It is not possible to share anything even though the programs may be identical. + +Fully Linked Objects +^^^^^^^^^^^^^^^^^^^^ + +These partially linked objects are required in systems that have no MMU. +In that case, each ELF executable will be loaded into a unique physical memory +location and, hence, truly will be unique and truly not shareable. + +But there is a difference if the ELF programs are loaded in the virtualized, +kernel build; in that case, all ELF executables are loaded into the same +virtual memory space! The executables can be fully linked at build time +because the final addresses are known and program files can be stripped +of all unnecessary relocation and symbolic information. + +In the fully linked build, all function and data addresses are fully resolved +to their final virtual addresses when the ELF executable is built. + +(Possibly functional, but not tested). + +Shared ``.text`` Regions +^^^^^^^^^^^^^^^^^^^^^^^^ + +So in the case of fully linked objects, there is no obstacle to sharing +the code. All that is required is a minor modification to the way that the ELF +``.text`` region is allocated. If the ELF ``.text`` region is mapped +into memory from the ELF file using the ``mmap()`` interface instead of +being allocated from the page memory pool, then the ``.text`` region +is naturally share-able. + +NuttX does include a partial implementation of the ``mmap()`` file mapping +interface but that implementation is tailored for use only in the flat, +embedded build and cannot be used with the kernel build, +at least in its current form. + +That full implementation of ``mmap()`` plus the minor changes +to the NuttX ELF loader are all that are required to support fully +share-able ``.text`` sections – as well as the memory savings +from not carrying aroung the relocation and symbol information + +(Not implemented). + + +Memory Management +----------------- + +When memory is allocated by the privileged base code or by an unprivileged +application. The resulting memory allocation must be accessible in and only +in the address space where the memory was allocated. + +The strategy of using two heaps as was described above for the simple +"Protected Build" cannot work in this case. +Rather, each address space must have its own heap! + +The function ``malloc()`` must exist in each address environment and memory +allocated via ``malloc()`` must be available only in that address space. + +(FULLY implemented, not tested). + +Page Allocator +^^^^^^^^^^^^^^ + +In such an environment, memory is used controlled by a simple +"page allocator". A page allocator is really a very simple memory allocator +that allocates physical memory in pages that can then be mapped into the +appropriate address environment using the MMU. + +The interface between each instance of ``malloc()`` and the base page +allocator are via the ``brk()`` and ``sbrk()`` system calls. + +For historic reasons, these system calls deal with something called the break +value, hence their names, but let's just think of this as how much memory +is available in the process local heap.(FULLY implemented): + +.. code-block:: c + + #include + int brk(void *addr); + void *sbrk(intptr_t incr); + +* The ``brk()`` function sets the break value to addr and changes the + allocated space accordingly (not implemented). +* The ``sbrk()`` function adds incr bytes to the break value and changes + the allocated space accordingly. If ``incr`` is negative, the amount ofi + allocated space is decreased by incr bytes. + The current value of the program break is returned by ``sbrk(0)``. + (FULLY implemented). +* See https://www.OpenGroup.org for further information about these system + calls. + +Shared Memory +^^^^^^^^^^^^^ + +Once you have all of the user application logic encapsulated as processes +within their own private address environment, then you following strict rules +about how these different user processes communicate with each other. +Of course, all of the standard Inter-Process Communication (IPC) methods work +fine: Semaphores, signals, message queues, etc. +But what about data? How do processes share large, in-memory data sets? + +From the title of this section you can see that the answer is +via Shared Memory, that is via chunks of memory which are mapped into each +process' virtual address space. +Here are the set of interfaces implemented for this purpose in NuttX: + +* ``shmget()``. Get the shared memory identifier, the shmid, of a shared + memory region. The ``shmid`` is like the file descriptor that you get when + you open a file and ``shmget()`` is much like ``open()``. And like opening + a file, there are flags that can control where you want to open the shared + memory object for read/write or read-only purpose or if you want to create + the shared memory region if it does not exist. + (FULLY implemented but without protection modes and also untested). +* ``shmctl()``. Once you have the ``shmid``, you can use that value with other + interfaces to manage the shared memory interface. ``shmctl()`` will, + for example, let you get and modify the characteristics of the shared memory + region. ``shmctl()`` will also let you remove a shared memory region when + it is no longer needed. + (FULLY implemented but without privilege modes and also untested). +* ``shmat()``. Given the shmid, ``shmat()`` can be used to attach the shared + memory region, i.e., to map it into the user process address space. + (FULLY implemented but untested) +* ``shmdt()``. Complementing ``shmat()``, ``shmdt()`` can be used to detachi + a shared memory region from the user process address space. + (FULLY implemented but untested) + +Closely related to to these interfaces are the ``mmap()`` and ``munmap()`` +interfaces. While these interfaces are implemented in NuttX and available +in the flat build, they have not yet been extended to provide full shared +memory support as described above for the shm interfaces. (not implemented). + + +Dual Stacks +^^^^^^^^^^^ + +Having a program loading from a file system is only interesting if that +program can also run other programs that reside in a file system. +But that raises another level of complexity: We cannot instantiate the new +program environment without also destroying the current program environment. +So all data must be preserved by copying the caller's data into the common, +neutral kernel address environment before the switch. + +But what about the callers stack? That stack lies in the calling process' +address environment. If in the system call, the kernel logic runs on the +caller's user stack, then there will almost surely be some disaster down +the road when switching process contexts. +How do you avoid losing the caller's stack contents that the C logic needs +to run while also instantiating the new program's address environment? + +The usual solution is to have two stacks per thread: +The large, possibly dynamically sized, user stack and a much smaller kernel +stack. When the caller issues the system call, the system call logic switches +stacks: It replaces the stack pointer with the reference to the user stack +with a new reference to the thread's kernel stack. Then the system +all executes with a process neutral kernel stack avoiding the stack transition +problems. The system call logic then restores the program's stack pointer +before returning from the system call (this is, of course, complicate +by the possibility of nested system calls and system calls that can generate +context switches or can dispatch signals). + +Doesn't it cost a lot of memory to have two stacks? +Yes and no, depending on what kind of memory usage you are used to. +Remember that with the MMU, the user memory space is quantized into units +of pages which are typically 4KiB and can grow upward from there in multiples +of 4KiB as needed. The kernel stack has limited depth. +It does not need to be dynamically sized and can probably be very small +(perhaps as little as 1KiB). So, yes, the dual stacks do use more memory +but the impact is not as significant as might first think. + +(Dual stack support is FULLY implemented in NuttX). + + +Further Down the Road +--------------------- + +Other Topics +^^^^^^^^^^^^ + +If all of the above were implemented, then NuttX could probably rightfully +claim to be a small Unix work-alike. From there, several additional topics +could be addresses but this is too far down the roadmap for me +the contemplate in any real detail: + +* On-Demand Paging and Swap. Why not keep programs and data in a file system + and swap the state into physical memory as needed?(not implemented). +* ``mmap()``. True shared memory and true file mapping could be supported. + I am repeating myself (not implemented). +* ``fork()``. The ``fork()`` interface could be supported. NuttX currently + supports the "crippled" version, ``vfork()`` but with these process address + environments, the real ``fork()`` interface could be supported. + (not implemented). +* Dynamic Stack Allocation. Completely eliminate the need for constant tuning + of static stack sizes.(not implemented). +* Shared Libraries. Am I repeating myself again?(not implemented). +* Program build tools. Then how do we make building programs for NuttX easy. + It should be as easy as arm-nuttx-eabi-gcc myprogram.c.(not implemented). + + +Some Conseuences +---------------- + +task_create() +^^^^^^^^^^^^^ + +In the traditional, flat NuttX build, the interface ``task_create()`` and +``task_spawn()`` are the standard interfaces for creating new tasks. +Both take the entry point address of the new task as an argument. +These task creation interface, however, cannot be used within code +executing in an address environment. + +Why not? It is kind of a long story. + +Remember that when a new task is created, a new task group is also created. +That task group provides all of the resources shared between the parent task +and all of its child threads. +One of these resources that is shared is the address environment. + +The behavior of ``task_create()`` is to create a new task with no address +environment, or more correctly with the common kernel address environment. + +When the parent process was created, it was created in the its own private +address. The address of the new task entry point passed to ``task_create()`` +must lie in this same private address environment. + +So you can see that we have a perverse configuration here: +The ``.text`` address of the new task lies in the private address space +of the parent task; but the task itself uses the common OS address space. +There are two problems here: + +* (1) because of privilege issues, the new task may not have access either + to the parent task's address environment or to the common kernel address + environment. It may just crash immediately, depending upon how these things + are configured. +* Or assuming that there are no privilege issues, (2) the new task certainly + will depend on the address space of the parent task being in place in order + to run. As soon as the parent's address space disappears + (due, perhaps, because the parent task's address space was de-activated + or perhaps the parent task exits destroying its address environment). + The child task would crash immediately thereafter. + +So how do you create new tasks/processes in such a context. +There is only one way possible; by using an interface that takes a file name +as an argument (rather than absolute address). +New processes started with ``vfork()`` and ``exec()`` or with +``posix_spawn()`` should not have any of these issues. + + +ARM Memory Management +--------------------- + +ARM Page Table Organization +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Cortex-A Page Tables. Let's focus on the ARMv7-A (aka, Cortex-A) MMU for now. +That MMU uses a two-level page table: + +* Level 1 Page Table + + * The size of virtually addressable region is 4GiB. + * The size of first level page table is 4096 entries (16KiB). + * Each each entry in the page table provides the mapping for 1MiB + of virtual memory. One entry may be either: + * A section mapping which maps 1MiB of contiguous virtual addresses to 1MiB + of physical memory (only occasionally used), or + * It may refer to a beginning of a second level page table + (much more commonly used). + +* Level 2 Page Table + + * Differing pages sizes are possible, but use of 4KiB pages used + (this is the smallest page size for the Cortex-A). + * With a 4KiB page table, 256 page table entries (PTEs) are required + to span the 1Mib region. + +Advantages / Disavantages +^^^^^^^^^^^^^^^^^^^^^^^^^ + +The most important benefits of the ARMv7-A page table are: + +* The TLB reloads are done automatically by the hardware without software + intervention. This is a huge performance advantage to other architecture + where each mapping must be instantiated via logic in a page fault exception + handler. +* The multiple levels and configurable page sizes add flexibility. + + +The ARMV7-A page table is, in fact, well-suited for higher end platforms that +do not suffer from memory and performance constraints. +But for the most constrained platforms, the following is a big issue: + +* Each page table required 16KiB of memory PLUS 1KiB of memory for each + level 2 page table (assuming 4KiB page size). + That would result in a maximum size of over 4MiB! +* The state of the page table is part of the process' context and must be + saved and restored on every context switch! + +Let's look first at how Linux deals with the ARM page tables; +then let's propose a scaled down approach for NuttX. + +ARM/Linux Page Table Notes +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Linux Summary +~~~~~~~~~~~~~ + +This is a summary of how the Linux kernel uses the page table to control +a process' memory mapping with a Cortex-A CPU. +This is not based upon my authoritative knowledge, but is rather based +on Google for explanations. + +Virtual Address Space Partitioning +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +* The 4GiB virtual address is partitioned with 3GiB of user space and 1GiB + of kernel space: Virtual address ``0x0000:0000-0xbfff:ffff`` is user space + while ``0xc000:0000-0xffff:ffff`` is kernel space. +* Level 1 page table entries ``0-3071`` map user space virtual addressees + and entries ``3072-4095`` map the kernel space addresses. + +Process Page Tables +~~~~~~~~~~~~~~~~~~~ + +* The ARM co-processor register ``TTBR0`` holds the address for the current + page directory (the page table that the MMU is using for translations). +* Each user process has its own page table located in the kernel + address space. +* For each process context switch, the kernel changes the ``TTBR0`` to the + new user process page table. +* Only ``TTBR0`` is used. ``TTBR1`` only holds the address of the initial + swapper page (which contains all the kernel mappings) and isn't really used + for virtual address translations. +* For each new user process, the kernel creates a new page table, copies all + the kernel mappings from the swapper page (page frames from 3-4GiB) to the + new page table and clears the user pages (page frames from 0-3GiB). + It then sets ``TTB0`` to the base address of this page directory and flushes + cache to install the new address space. + The swapper page is also always kept up to date with changes to the mappings. +* The swapper page is usually located at addresses ``0xc0004000-0xc0008000``. + +ARM/NuttX Page Table Proposal +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Size Reduction Tradeoff +~~~~~~~~~~~~~~~~~~~~~~~ + +We can reduce the memory usage by page tables in NuttX by: + +* Keeping only a single level 1 page table that is shared by all task groups. +* Level 2 page tables are, of course, still need to be duplicated for each + process. +* To avoid copying the 3GiB mapping used by Linux, we can simply reduce + the size of the virtual address space so that instead of copying + 3,072 entries on each process switch, we copy perhaps 4. + That would limit the virtual address range for each process from 3GiB + to only 4MiB. But that is probably reasonable but would also be configurable. +* Reducing the supported virtual address from 3GiB to, say, 4MiB would also + reduce the amount of memory that has to be allocated for each process. + Continuing with this 4MiB suggestion, it would following that no more than + 4KiB would need to set aside for level 2 page table support for each process. +* Further, let's not make any assumptions about what virtual address range + corresponds to user space and which corresponds to kernel space. + It is simply an agreement that must be made between the platform + implementation and the program's linker script. + +Per-Process/Per-Thread Regions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Three regions must set aside for each process to hold: + +* A level 2 mapping for the ``.text`` region, +* A level 2 mapping for the static data region (``.bss`` and ``.data``), and +* A level 2 mapping for the process' heap. + +Instantiation/Extensibility +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The first two mappings would be created when the program is loaded +into memory; The first heap mapping would be created when ``sbrk()`` +is first called. All three could be extended at runtime if shared libraries +are supported (augmenting ``.text``, ``.bss`` and ``.data``) or when the heap +is extended by subsequent ``sbrk()`` calls. + +Configuration +^^^^^^^^^^^^^ + +The following configuration options are proposed: + +* ``CONFIG_ARCH_TEXT_VBASE`` - The virtual address of the beginning + the ``.text`` region. +* ``CONFIG_ARCH_DATA_VBASE`` - The virtual address of the beginning + of the ``.bss``/``.data`` region. +* ``CONFIG_ARCH_HEAP_VBASE`` - The virtual address of the beginning + of the heap region. +* ``CONFIG_ARCH_TEXT_NPAGES`` - The maximum number of pages that can be + allocated for the ``.text`` region. This, along with knowledge of the page + size, determines the size of the ``.text`` virtual address space. + Default is 1. +* ``CONFIG_ARCH_DATA_NPAGES`` - The maximum number of pages that can be + allocated for the ``.bss``/``.data`` region. This, along with knowledge + of the page size, determines the size of the ``.bss``/``.data`` virtual + address space. Default is 1. +* ``CONFIG_ARCH_HEAP_NPAGES`` - The maximum number of pages that can be + allocated for the heap region. This, along with knowledge of the page size, + determines the size of the heap virtual address space. Default is 1. + +Implementation +^^^^^^^^^^^^^^ + +The task group resources are retained in a single structure, +``task_group_s`` that is defined in the header file +``nuttx/include/nuttx/sched.h``. +The type ``group_addrenv_t`` must be defined by platform specific logic +in ``nuttx/arch/*/include/arch.h``. +This is a first cut proposal at that type might be: + +.. code-block:: c + + struct group_addrenv_s + { + FAR uintptr_t *text[ARCH_TEXT_NSECTS]; + FAR uintptr_t *data[ARCH_DATA_NSECTS]; + FAR uintptr_t *heap[ARCH_HEAP_NSECTS]; + }; + typedef struct group_addrenv_s group_addrenv_t + +Where each 1MiB section refers to a level 2 page table that maps +256 4KiB pages: + +.. code-block:: c + + #define __PG2SECT_SHIFT (20 - MM_PGSHIFT) + #define __PG2SECT_MASK ((1 << __PG2SECT_SHIFT) - 1) + #define ARCH_PG2SECT(p) (((p) + __PG2SECT_MASK) >> __PG2SECT_SHIFT) + #define ARCH_SECT2PG(s) ((s) << __PG2SECT_SHIFT) + #define ARCH_TEXT_NSECTS ARCH_PG2SECT(CONFIG_ARCH_TEXT_NPAGES) + #define ARCH_DATA_NSECTS ARCH_PG2SECT(CONFIG_ARCH_DATA_NPAGES) + #define ARCH_HEAP_NSECTS ARCH_PG2SECT(CONFIG_ARCH_HEAP_NPAGES) + +These tables would hold the physical address of the level 2 page tables. +All would be initially ``NULL`` and would not be backed up with physical +memory until mappings in the level 2 page table are required. + +Per-Thread Regions +^^^^^^^^^^^^^^^^^^ + +One region must set aside for each thread to hold: + +* The thread's stack + +This stack would be initially of size zero and would be backed-up with +physical pages during page fault exception handling to support dynamically +sized stacks for each thread. + +The following configuration options are proposed: + +* ``CONFIG_ARCH_STACK_VBASE`` - The virtual address of the beginning + of the stack region +* ``CONFIG_ARCH_STACK_NPAGES`` - The maximum number of pages that can be + allocated for the stack region. This, along with knowledge of the page size, + determines the size of the stack virtual address space. Default is 1. + +The thread resources are retained in a single structure, ``tcb_s`` that is +defined in the header ``file nuttx/include/nuttx/sched.h``. +The type ``xcptcontext`` must be defined by platform specific logic +in ``nuttx/arch/*/include/irq.h``. +This structure might be extended to include: + +.. code-block:: c + + FAR uintptr_t *stack[ARCH_STACK_NSECTS]; + +Where again: + +.. code-block:: c + + #define ARCH_STACK_NSECTS ARCH_PG2SECT(CONFIG_ARCH_STACK_NPAGES) + +Context Switches +^^^^^^^^^^^^^^^^ + +Then what happens on each context switch? + +* Since there is only a single page table, ``TTBR0`` never changes. +* Instead, the particular Level 1 page entries are replace based upon the + physical page allocations in ``group_addrenv_t`` and ``xcptcontext``. +* Assuming again a 4MiB per-process virtual address space, at most only + four elements of the level 1 page table would have to change: + ``.text``, ``.bss``/``.data``, heap, and stack. +* For context switches within the same task group, only the stack level 1 + table entry would need to change. +* The MMU TLBs and processor caches would still have to be flushed + and invalidated for the (smaller) user virtual address range. + + +Terminology +=========== + +* **Address Environment:** This is really a generic phrase to refers to the + memory addressable by the software. However, in the context of this + document, we will be referring to something more specific. + We will be referring a memory architecture that supports multiple. + per-task address environments: Each task can execute within its own + address environment. Tasks with private address environments + may sometimes be called processes. +* **Blob:** A block of code and/or data within a restricted range + of contiguous addresses. +* **Call Gate:** A call gate is a mechanism for calling into privileged code + from unprivileged code, raising the privilege level of the thread + temporarily for the call. A typical method of implementing a call gate + is through software trap or software interrupt instructions: + The interrupt will place the thread into a privileged mode of operation + where it can then execute the call. + These call gates are used to implement system calls (``SYSCALLS``), + i.e., calls from user application code into OS system services. +* **Flat Address Space:** An address space is flat if either + (1) there is no mapping of physical addresses to virtual addresses, or + (2) there is a 1-to-1 mapping of the physical address space + to a virtual address space. +* **Identity Mapping:** When a CPU has a MMU that is used to map the physical + address space to a virtual address space and the virtual addresses map + to the same virtual address. In this case, the MMU is is not being used + for memory mapping, but rather only for its ability to color memory regions. +* **MMU:** Memory Management Unit. Can be configured to map physical addresses + in one address region to virtual addresses in a different address region. + Can also color an address region by controlling the privileges required + to access the memory, the cache properties of the memory, + and other memory attributes. +* **MPU:** Memory Protection Unit. Can be be configured to protect memory. +* **Page:** The size of a block of memory that can be mapped using an MMU. + The MMU can handle pages of different sizes. Other terminology may be used + for very large pages; ARM calls these sections. And sections may be divided + down to to smaller pages of differing sizes. +* **Page Fault:** When the MMU is unable to map a virtual address to a physical + address, then a page fault occurs. The page fault means that there is no TLB + in the MMU that can provided the necessary mapping. + In response to a page fault, the MMU may consult a page table in an attempt + to resolve the page fault or it may generate a page fault interrupt so that + software can resolve the page fault. + An unresolvable page fault is fatal and usually results in a crash. +* **Page Table:** A data structure in memory that is accessed by the MMU to + reload TLBs. The able has a precise format determined by the MMU hardware. + It is configured by software to support the desired mapping and accessed + via hardware DMA when a page fault occurs. +* **Physical Address:** The actually address that appears on the CPU bus and + provided to the memory parts when the memory is accessed. +* **PTE:** Page Table Entry. +* **TLB:** Translation Look-Aside Buffer. An element of the MMU that maps + one page of memory. This may be loaded by explicit logic as part of page + fault handling (as is typically done with MIPS) or may be loaded + automatically by the MMU from a page table (as with the ARM). +* **Virtual Address:** The memory addresses used by the software running + in the CPU. This might be mapped to a different virtual address by an MMU. + diff --git a/Documentation/implementation/naming_arch_mcu_board_interfaces.rst b/Documentation/implementation/naming_arch_mcu_board_interfaces.rst new file mode 100644 index 00000000000..541146717d8 --- /dev/null +++ b/Documentation/implementation/naming_arch_mcu_board_interfaces.rst @@ -0,0 +1,76 @@ +================================================= +Naming of Architecture, MCU, and Board Interfaces +================================================= + +What's the meaning of the prefix ``up_`` in NuttX? +Which functions should be prefixed with ``up_``? + +``up_`` is supposed to stand for **microprocessor**; +the `u` is like the Greek letter micron: µ. So it would be **µP** which +is a common shortening of the word microprocessor. +I don't like that name very much. I wish I would have used ``arch_`` instead. +But now I think I am stuck with ``up_``. + +Common Microcontroller Interfaces +================================= + +Any interface that is common to all microcontroller and is used throughout +the system should be prefixed with ``up_`` and prototyped in +``include/nuttx/arch.h``. The definitions in that header file provide the +common interface between NuttX and the architecture-specific implementation +in ``arch/``. + +Microcontroller-Specific Interfaces +=================================== + +An interface which is unique to a certain microcontroller should be prefixed +with the name of the microcontroller, for example ``stm32_``, +and be prototyped in some header file in the ``arch/`` directories. +These are interfaces used only by logic within the ``arch/`` and ``boards/`` +directories. + +Architecture-Specific Interfaces +================================ + +An interface which is unique to a CPU architecture, but shared among +microcontrollers that implement that CPU architecture should be prefixed +with the name of architecture. For example, the architecture-specific +interfaces used by STM32 would begin with ``arm_``. +These are interfaces used only by logic within the ``arch/`` and ``boards/`` +directories. + +There is also a ``arch//include//chip.h`` header file +that can be used to communicate other microprocessor-specific information +between the board logic and even application logic. + +Application logic may, for example, need to know specific capabilities +of the chip. Prototypes in that ``chip.h`` header file should follow +the microprocessor-specific naming convention. + +Common Board Interfaces +======================= + +Any interface that is common to all boards should be prefixed with ``board_`` +and should also be prototyped in ``include/nuttx/arch.h``. + +These ``board_`` definitions provide the interface between the board-level +logic and the architecture-specific logic. + +There is also a ``boards////include/board.h`` header file +that can be used to communicate other board-specific information between +the architecture logic and even application logic. + +Any definitions which are common between a single architecture and several +boards should go in this ``board.h`` header file; +``include/nuttx/arch.h`` is reserved for board-related definitions common +to all architectures. + +Specific Interfaces +=================== + +Any interface which is unique to a board should be prefixed with the board +name, for example ``stm32f4discovery_``. Sometimes the board name is too long +so ``stm32_`` would be okay too. These should be prototyped in +``boards////src/.h`` and should not be used outside +of that directory since board-specific definitions have no meaning outside +of the board directory. diff --git a/Documentation/implementation/naming_os_internals.rst b/Documentation/implementation/naming_os_internals.rst new file mode 100644 index 00000000000..39a1622c49e --- /dev/null +++ b/Documentation/implementation/naming_os_internals.rst @@ -0,0 +1,181 @@ +=============================== +Naming of OS Internal Functions +=============================== + +.. note:: Most of the naming examples used in this section no longer exist. + They have been converted to the naming conventions described + in this section. While they are good examples, they no longer + reflect the current state of the code base. + + +Legacy smooshed-together names +============================== + +There has been some controvery lately about the consistency of naming of OS +internal functions. We should come to a consensus on the desired naming +of conventions for internal OS functions before we start arbitrarily +changing names. + +NuttX function names are all lower case. +That is required by the naming standard. +Historically, has used a subsystem name, an underscore, then all of the +renaming words smooshed together. For example: + +.. code-block:: c + + void sched_mergeprioritized(FAR dq_queue_t *list1, FAR dq_queue_t *list2, + uint8_t task_state); + +I would represent that form as:: + + _ + +Many of these names are impossible to read unless you stop and parse +the letters to find the word boundaries. + + +System-Object-Verb Naming +========================= + +Most recent naming has broken out one more of the smooshed-together-words +and separated them with undersore characters like: + +.. code-block:: c + + unsigned int sched_timer_cancel(void); + void sched_timer_resume(void); + void sched_timer_reassess(void); + +Which I would describe as:: + + __ + + +System-Verb-Object Naming +========================= + +Another form, which we have been using lately, is to switch the +```` and the ```` like:: + + __ + +For example: + +.. code-block:: c + + int sched_get_stackinfo(pid_t pid, FAR struct stackinfo_s *stackinfo); + +By the ``__`` naming this would have been: + +.. code-block:: c + + int sched_stackinfo_get(pid_t pid, FAR struct stackinfo_s *stackinfo); + +And the timer interfaces would become the following under the +``__`` rule: + +.. code-block:: c + + unsigned int sched_cancel_timerl(void); + void sched_resume_timer(void); + void sched_reassess_timer(void); + +And the "smooshed" name ``sched_mergeprioritized()`` would become: + +.. code-block:: c + + void sched_merge_prioritized(FAR dq_queue_t *list1, FAR dq_queue_t *list2, + uint8_t task_state); + +We won't even consider the ``_``. +It has a history but is pretty much odious. +We should not consider, for example, the smooshed-together form: + +.. code-block:: c + + int sched_getstackinfo(pid_t pid, FAR struct stackinfo_s *stackinfo); + +What an ugly mess! :-) + + +Unit Suffixes +============= + +Often, you will have the functions that return the same value, +but in different units. +For example, these two functions return the system time: + +.. code-block:: c + + clock_t clock_systimer(void); + int clock_systimespec(FAR struct timespec *ts); + +The first returns the time in system clock ticks. +The second returns the time as a struct timespec. + +I have seen a naming practice The keeps the naming of the functions the same, +but appends the units, preceded by an underscore character, at the end +of the function name. For example, these could be: + +.. code-block:: c + + clock_t clock_systime_ticks(void); + int clock_systime_timespec(FAR struct timespec *ts); + +Or should they include a ````: + +.. code-block:: c + + clock_t clock_get_systime_ticks(void); + int clock_get_systime_timespec(FAR struct timespec *ts); + +That form would then be:: + + __[_] + +Where the square bracket represent and optional unit suffix. + + +Implicit get +============ + +In many cases, I think that the ``_get_`` is implicit and need not appear +in the function name. But if there are multiple operations +on the ````, then the ``_get_`` would be necessary. +Hence, I think the naming: + +.. code-block:: c + + clock_t clock_systime_ticks(void); + int clock_systime_timespec(FAR struct timespec *ts); + +is perfectly accepable without the ``_get_``. +And since ``_get_`` is the only operation on ``stackinfo``, +the following would be a perfectly acceptable renaming: + +.. code-block:: c + + int sched_stackinfo(pid_t pid, FAR struct stackinfo_s *stackinfo); + + +Other Naming +============ + +There is other naming that uses long names with many underscore characters +separating each word in the long name. We are not a fan of long names. + + +Conclusion +========== + +Our preference would be the ``__`` form. +We have been using this form in creating all new internal interfaces. +Does anyone else have a thought? or a preference? + +We should come to an understanding and stop the arbitrary name changes. +We hope that we can put an end to the smooshed-together naming +(except wheree required) in any case. + +.. note:: This applies only to the internal naming of functions within the OS. + Other names are imposed on us by standards for application + interfaces to the OS which may follow other rules. diff --git a/Documentation/implementation/nuttx_initialization_sequence.rst b/Documentation/implementation/nuttx_initialization_sequence.rst new file mode 100644 index 00000000000..d3255a5e8ca --- /dev/null +++ b/Documentation/implementation/nuttx_initialization_sequence.rst @@ -0,0 +1,713 @@ +.. _nuttx-initialization-sequence: + +============================= +NuttX Initialization Sequence +============================= + +Overview +======== + +This initialization sequence is really quite simple because the system runs +in single-thread mode up until the point the that is starts the application. +That means the initialization sequence is just a simple, straight-line +of function calls. +It is until just before starting the application that system goes to +multi-threaded mode and things can get more complex. + +At the highest level, the NuttX initialization sequence can be +represented in three phases: + +* **Phase A** - The hardware-specific power-on reset initialization, + +* **Phase B** - NuttX RTOS initialization, and + +* **Phase C** - Application Initialization. + +Each of these will be discussed in more detail in the following sections. + + +Case example: STM32 F4 +====================== + +In this discussion, we'll use the STM32 F4 MCU and its popular evaluation +board STM32F4Discovery as example but the explanation can be applied +to any supported architecture. + +Here is the map of initialization function calls:: + + __start()-arch/arm/src/stm32/stm32_start.c + | + +--*Set stack limit + +--stm32_clockconfig() + +--stm32_fpuconfig() + +--stm32_lowsetup() + +--stm32_gpioinit() + +--showprogress('A') + +-- + +-- + +--stm32_boardinitialize()-boards/arm/stm32/stm32f4discovery/src/stm32_boot.c + | | + | +--stm32_spidev_initialize()-stm32_spi.c:ONLY CHIP SELECTS + | +--stm32_usbinitialize()- + | +--stm32_netinitialize()- + | +--board_autoled_initialize()- + | + nx_start()-sched/init/nx_start.c + | + +--*Initialize global data structures + +--*Initialize OS facilities + +--net_initialize()-net/net_initialize.c + | | + | +--net_lockinitialize() + | +--mld_initialize() + | +--can_initialize() + | +--netlink_initialize() + | +--tcp_initialize() + | +--udp_initialize() + | +--usrsock_initialize() + | + +--up_initialize()-arch/arm/src/common/up_initialize.c + | | + | +--arm_dmainitialize() + | +--Config basic /dev nodes + | +--arm_serialinit() + | +--Console Init + | +--Crypto Config + | +--arm_netinitialize() + | | | + | | +--stm32_spibus_initialize() + | | + | | + | +--arm_usbinitialize() + | +--L2 Cache Init + | + +--board_early_initialize() + +--g_nx_initstate = OSINIT_HARDWARE + +--shm_initialize() + +--lib_initialize() + +--binfmt_initialize() + +--Start SMP + +--syslog_initialize() + +--g_nx_initstate = OSINIT_OSREADY + +--DEBUGVERIFY(nx_bringup())-sched/init/nx_bringup.c + | | + | +--nx_pgworker() + | +--nx_workqueues() + | +--nx_create_initthread()-sched/init/nx_bringup.c + | | + | +----+different thread + | : : + | : nx_start_task()-sched/init/nx_bringup.c + | : : + | same+----+--nx_start_application()-sched/init/nx_bringup.c + | thread | + | +--board_late_initialize()-stm32_boot.c:BOARD_DEPENDANT + | | | + | | +--stm32_bringup() + | | | + | | +--stm32_i2ctool() + | | +--board_bmp180_initialize() + | | +--stm32_sdio_initialize() + | | +--stm32_usbhost_initialize() + | | +--stm32_pwm_setup() + | | +--stm32_can_setup() + | | +--etc + | | + | +-- + | + | + | + | + | + +--kmm_givesemaphore() + +--up_idle() + + +Phase A - Power-On Reset Initialization +======================================= + +The system begins execution when the processor is reset. +This usually at power-on, but all resets are basically the same whether +they occur because of power-on, pressing the reset button, or on a watchdog +timer expiration. + +.. note:: The code that executes when the processor is reset is unique + to the particular CPU architecture and is not a common part + of NuttX. + +The kind of things that must be done by the architecture-specific reset +handling includes: + +* Putting the processor in its operational state. This may include things + like setting CPU modes; initializing co-processors, etc. +* Setting up clocking so that the software and peripherals operate as expected. +* Setting up the C stack pointer (and other processor registers). +* Initializing memory. +* Starting NuttX. + + +Memory Initialization +--------------------- + +In C implementations, there are two general classes of variable storage. +First there are the initialized variables. +For example, consider the global variable x: + +.. code-block:: c + + int x = 5; + +The C code must be assured that after reset, the variable x has the value 5. +Initialized variable of this kind are retained in a special memory section +called data (or ``.data``). + +Other variables are not initialized. Like the global variable y: + +.. code-block:: c + + int y; + +But the C code will still expect y to have an initial value. +That initial value will be zero. +All uninitialized variables of this type need to have the value zero. +These uninitialized variables are retained in a section called bss +(or ``.bss``). + +When we say that the reset handling logic initializes memory, we mean two things: + +1. It provides the (initial) values of the initialized variables by copying + the values from FLASH into the ``.data`` section, and +2. It resets all of the uninitialized variables to zero. + It clears the ``.bss`` section. + +Lets walk through reset sequence. This reset logic can be found in two files:: + + nuttx/arch/arm/src/stm32_vectors.S + nuttx/arch/arm/src/stm32_start.c + + +nuttx/arch/arm/src/stm32_vectors.S +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This file provides all of the STM32 exception vectors and power-on reset +is simply another exception vector. +There are few important things to note about this file. + +``.section .vectors, ax``. This pseudo operation will place all of the +vectors into a special section call ``.vectors``. +On of the STM32 F4 linker scripts is located at +``nuttx/boards/arm/stm32/stm32f4discovery/scripts/ld.script``. +In that file, you can see that section ``.vectors`` is forced to lie +at the very beginning of FLASH memory. +The STM32 F4 can be configured to boot in different ways via strapping. +If it is strapped to boot from FLASH, then the STM32 FLASH memory will +be aliased to address ``0x0000 0000`` when the reset occurs. +That is the address of the power-up reset interrupt vector. + +The first two 32-bit entries in the vector table represent the power-up +exception vector (which we know will be positioned at address +``0x0000 0000`` when the reset occurs). Those two entries are: + +.. code-block:: c + + .word IDLE_STACK /* Vector 0: Reset stack pointer */ + .word __start /* Vector 1: Reset vector */ + +The Cortex-M family is unique in the way that is handles the reset vector. +Notice that there are two values: the stack pointer for the start-up thread +(the IDLE thread), and the entry point in the IDLE thread. + +When the reset occurs, the the stack pointer is automatically set +to the first value and then the processor jumps to reset entry point +``__start`` specified in the second entry. + +This means that the reset exception handling code can be implemented in C +rather than assembly language. + + +nuttx/arch/arm/src/stm32_start.c +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The reset vector ``__start`` lies in the file +``nuttx/arch/arm/src/stm32/stm32_start.c`` and does the real, +low-level architecture-specific initialization. This initialization includes: + +1. ``stm32_clockconfig()`` - Initialize the PLLs and peripheral clocking + needed by the board. + +2. ``stm32_fpuconfig()`` - If the STM32 F4's hardware floating point + is initialized, then configure the FPU and enable access to the FPU + co-processors. + +3. ``stm32_lowsetup()`` - Enable the low-level UART. This is done very early + in initialization so that we can get serial debug output to the console + as soon as possible. If you are doing a board bring-up this + is very important. + +4. ``stm32_gpioinit()`` - Perform any GPIO remapping that is needed + (this is a stub for the F4, but the F1 family requires this step). + +5. ``showprogress('A')`` - This simply outputs the character ``A`` on the + serial console (only if ``CONFIG_DEBUG`` is enabled). If debug is enabled, + you will always see the letters ABDE output on the console. That output + all comes from this file. + +6. Next the memory is initialized: + + a. The ``.bss`` section is set to zero (Letter ``B`` is then output if + ``CONFIG_DEBUG`` is enabled), then + b. The ``.data`` section is set to its initial values (The letter ``C`` + is output if debug is enabled), + +7. ``stm32_boardinitialize()`` - Board-specific logic is initialized by + calling this function. For the case of the STM32F4Discovery board, + this logic can be found at + ``nuttx/boards/arm/stm32/stm32f4discovery/src/stm32_boot.c`` and does + the following operations: + + a. ``stm32_spidev_initialize()`` - Initialize SPI chip selects + if SPI is enabled. + b. ``stm32_usbinitialize()`` - Initialize hardware USB devices if enabled. + c. ``stm32_netinitialize()`` - Initialize hardware network devices + if enabled. + d. ``board_autoled_initialize()`` - Configure on-board LEDs + if LED support has been selected. + +8. When ``stm32_boardinitialize()`` returns to ``__start()``, the low-level, +architecture-specific initialization is complete. + + +Phase B - NuttX RTOS Initialization +=================================== + +``nx_start()`` +-------------- + +This function resides in the file ``nuttx/sched/init/nx_start.c`` and +is the NuttX entry point. + +It is called by ``__start()`` and performs the next phase of RTOS-specific +initialization before bringing up the application. + +The operations performed by ``nx_start()`` are summarized below. +Note that many of these features can be disabled from the NuttX configuration +file and in that case those operations are not performed: + +1. Initializes some NuttX global data structures, + +2. Initializes the TCB for the IDLE (i.e, the thread that the initialization + is performed on), + +3. ``nxsem_initialize()`` - Initialize the POSIX semaphore facilities. + This needs to be done first because almost all other OS features depend + on POSIX counting semaphores. + +4. Memory organization - This includes heap configuration, memory manager, + paging, I/O buffers, etc. + +5. ``task_initialize()`` - Initialize task data structures. + +6. ``fs_initialize()`` - Initialize the file system (needed to support + device drivers). + +7. ``irq_initialize()`` - Initialize the interrupt handler subsystem. + This initializes only data structures; CPU interrupts are still disabled. + +8. ``wd_initialize()`` - Initialize the NuttX watchdog timer facility, + +9. ``clock_initialize()`` - Initialize the system clock, + +10. ``timer_initialize()`` - Initialize the POSIX timer facilities, + +11. ``nxsig_initialize()`` - Initialize the POSIX signal facilities, + +12. ``nxmq_initialize()`` - Initialize the POSIX message queue facilities, + +13. ``pthread_initialize()`` - Initialize the POSIX pthread facilities, + +14. ``net_initialize()`` - Initialize networking facilities, + +.. note:: Up to this point, all of the initialization steps have only been + software initializations. Nothing has interacted with the hardware. + Rather, all of these steps simply prepared the environment so that + things like interrupts and threads can function properly. + The next phases depend upon that setup. + +15. ``up_initialize()`` - The processor specific details of running + the operating system will be handled here. Such things as setting up + interrupt service routines and starting the clock are some of the things + that are different for each processor and hardware platform. + + All ARM-based MCUs share a common ``up_initialize()`` implementation + provided at ``nuttx/arch/arm/src/common/up_initialize.c``. The operations + perform by this common ARM initialization will, however, call into + facilities provided by the particular ARM chip. + For the STM32 F4, those facilities would be provided by logic in files + as ``nuttx/arch/arm/src/stm32``. The common ARM initialization sequence is: + + * ``up_color_intstack()`` - Colorize the interrupt stack. + + * ``arm_addregion()`` - The basic heap was set up during processing by + ``nx_start()``. However, if the board supports multiple, discontiguous + memory regions, any addition memory regions can be added to the heap + by this function. For the STM32 F4, ``up_addregion()`` is implemented + in ``nuttx/arch/arm/src/stm32/stm32_allocateheap.c``. + + * ``arm_pminitialize()`` - If ``CONFIG_PM`` is defined, the function must + initialize the power management subsystem. This MCU-specific function + must be called very early in the intialization sequence before any other + device drivers are initialized (since they may attempt to register with + the power management subsystem). There is no implementation + of ``up_pminitialize()`` for any STM32 platform. + + * ``arm_dmainitialize()`` - Initialize the DMA subsystem. + For the STM32 F4, this DMA initialization can be found in + ``nuttx/arch/arm/src/stm32/stm32_dma.c`` (which includes + ``nuttx/arch/arm/src/stm32f4xxx_dma.c``). + + * ``devnull_register()`` - Registers the standard ``/dev/null``. + + * ``devrandom_register()`` - Registers the standard ``/dev/random``. + + * ``devurandom_register()`` - Registers the standard ``/dev/urandom``. + + * ``devzero_register()`` - Registers the standard ``/dev/zero``. + + * ``loop_register()`` - Registers the standard ``/dev/loop``. + + * ``note_register()`` - Registers the standard ``/dev/note``. + + * ``arm_serialinit()`` - Initialize the **standard** serial driver + (found at ``nuttx/arch/arm/src/stm32/stm32_serial.c`` STM32 F4). + + * ``arm_netinitialize()`` - Initialize the network. For the STM32 F4, + this function is in ``nuttx/arch/arm/src/stm32/stm32_eth.c``. + + * ``arm_usbinitialize()`` - Initialize USB (host or device). + For the STM32 F4, this function is in + ``nuttx/arch/arm/src/stm32/stm32_otgfsdev.c``. + + * ``arm_l2ccinitialize()`` - Initialize the L2 cache if present + and selected. + + * ``up_ledon(LED_IRQSENABLED)`` - Finally, ``up_initialize()`` + illuminates board-specific LEDs to indicate the IRQs are now enabled. + + +16. ``board_early_initialize()`` - If ``CONFIG_BOARD_EARLY_INITIALIZE`` is + selected, then an additional initialization call will be performed in the + boot-up sequence to a function called ``board_early_initialize()``. + It will be called immediately after ``up_initialize()`` (and may be + thought of as a board-specific, extension of ``up_initialize()``) + and well ``before board_late_initialize()`` is called and the initial + application is started. + +17. ``g_nx_initstate = OSINIT_HARDWARE`` - This signals that basic + hardware setup is complete. + +18. ``shm_initialize()`` - Initialize shared memory support. + +19. ``lib_initialize()`` - Initialize the C libraries. This is done last + because the libraries may depend on the above. + +20. ``binfmt_initialize()`` - Initialize binary loader subsystem. + +21. Start SMP support in multi-core MCUs. This is not the case + in STM32F4Discovery but relevant because here are created ``stdin``, + ``stdout`` and ``stderr`` for each CPU's (even if there is only one) + IDLE task. All tasks subsequently created by the IDLE thread will inherit + these file descriptors. + +22. ``syslog_initialize()`` - Late initialization of the system logging + device. Some SYSLOG channel must be initialized late in the initialization + sequence because it may depend on having IDLE task file structures setup. + +23. ``nx_bringup()`` - Create the initial tasks. This will be described + in more detail below. + +``nx_bringup()`` +---------------- + +This function is called at the very end of the initialization sequence +in ``nx_start()``, just before entering the IDLE loop. It is located +in ``nuttx/sched/init/nx_bringup.c`` and it starts all of the required +threads and tasks needed to bring up the system. + +This function performed the following specific operations: + +* ``nx_pgworker()`` - Start the page fill worker kernel thread that will + resolve page faults. This should always be the first thread started because + it may have to resolve page faults in other threads. This is the task that + runs in order to satisfy page faults in processors that have an MMU and + in configurations where on-demand paging is enabled. + +* ``nx_workqueues()`` - Start the worker thread. The worker thread may be used + to execute any processing deferred to the worker thread via APIs provided + in ``include/nuttx/wqueue.h``. The worker thread's primary function is + as the “bottom half” for extended device driver processing but can be used + for a variety of purposes like misc garbage clean-up. + +* ``nx_create_initthread()`` - Once the operating system has been initialized, + this funcions either directly calls ``nx_start_application()`` or creates + a thread for running it + +* ``nx_start_application()`` - If set in the NuttX configuration, + this function calls ``board_late_initialize()``. + + * ``board_late_initialize()`` is a last-minute, board-specific + initialization. Note that there was earlier, board-specific initialization + calls (to ``stm32_board_initialize()`` and + to ``board_early_initialize()``). The difference here is these first, + low-level initialization calls were made before the OS was completely + launched. ``board_late_initialize()``, on the other hand, is called + at the end after the OS has been initialized but before any application + tasks have been started. + ``board_late_initialize()`` would be an ideal place to do board-specific + initialization steps that depend on having a fully initialized OS + such as memory allocations, initialization of complex device drivers, + mounting of file systems, etc. + + * After that, ``nx_start_application()`` launches the application either + by creating a task for it or executing a program from a filesystem after + mounting it. + + * In the case of creating a task for the application, its entry has + the name ``user_start()``. ``user_start()`` is provided by application + code and when it runs, it begins the application-specific phase of the + initialization sequence as described below. + +.. note:: The default ``user_start()`` entry point can be changed to use + one of the named applications used by NSH. This is a start-up option + that is not often used and will not be discussed further here. + +And finally enter the IDLE loop. After completing the initialization, the role +of the IDLE thread changes. It becomes the thread that executes only when +there is nothing else to do in the system (hence, the name IDLE thread). + +IDLE Thread Activities +---------------------- + +As mention, the IDLE thread is the thread that executes only when there is +nothing else to do in the system. It has the lowest priority in the system. +It always has the priority 0. It is the only thread that is permitted to have +the priority 0. And it can never be blocked (otherwise, what would run then?). + +As a result, the IDLE thread is always in the ``g_readytorun`` list and, +in fact, since that list is prioritized, can guaranteed to always be +the final entry at the tail of the ``g_readytorun`` list. + +The IDLE is an an infinite loop. But this does not make it a “CPU hog”. +Since it is the lowest priority, the it can be suspended whenever anything +else needs to run. + +The IDLE thread does two things in this infinite loop: + +1. If the worker was not started (see ``nx_bringup()`` below), then the IDLE + thread will perform memory clean-up. Memory clean is required to handle + deferred memory deallocation. Memory allocations must be deferred when + the memory is freed in a context where the software does not have access + to the heap and, hence, cannot truly free the memory (such as + in an interrupt handler). In this case, the memory is simply put into + a list of freed memory and, eventually, cleaned up by the IDLE thread. + +.. note:: The worker thread's primary function is as the “bottom half” + for extended device driver processing. If the worker thread + was started, then it will run at a higher priority than the + IDLE thread. In this case, the worker thread will take over + responsibility for cleaning up these deferred allocations. + +2. Then the loop calls ``up_idle()``. The operations performed + by ``up_idle()`` are architecture- and board-specific. In general, + this is the location where CPU-specific reduced power operations + may be performed. + +STM32 F4 IDLE thread +-------------------- + +The default STM32 F4 IDLE thread is located at +``nuttx/arch/arm/src/stm32_idle.c``. +This default version does very little: + +1. It includes a example, “skeleton” function that illustrates that kinds + of things that you can do if ``CONFIG_PM`` is enabled (this example code + is not fully implemented in the default IDLE logic). +2. The it executes the Cortex-M thumb2 instruction wfi which causes the CPU + to sleep until the next interrupt occurs. + + +Phase C - Application Initialization +==================================== + +At the conclusion of the OS initialization phase in ``nx_start()``, +the user application is started by creating a new task +at the entry point ``user_start()``. + +There must be exactly one entry point called ``user_start()`` +in every application built on top of NuttX. + +Any additional initialization performed in the ``user_start()`` function +is purely application dependent. + + +A Simple Hello World Application +-------------------------------- + +The simplest user application would be the “Hello, World!” example. +See ``apps/examples/hello``. Here is the whole example: + +.. code-block:: c + + int user_start(int argc, char *argv[]) + { + printf("Hello, World!!\n"); + return 0; + } + +In this case, no additional application initialization is needed. +It just “says hello” and exits. + +A Nutt Shell User Application/Command +------------------------------------- + +The NuttShell (NSH) is a simple shell application that may be used with NuttX. +It is described here. It supports a variety of commands and is (very) loosely +based on the bash shell and the common utilities used +in Unix shell programming. + +NSH is implemented as a library that can be found at ``nuttx-apps/nshlib``. +The NSH start-up sequence is very simple. As an example, the the code at +``apps/examples/nsh/nsh_main.c`` illustrates how to start NuttX. +It simple does the following: + +1. If you have C++ static initializers, it will call your implementation + of ``up_cxxinitialize()`` which will, in turn, call those + static initializers. +2. This function then calls ``nsh_initialize()`` which initializes + the NSH library. ``nsh_initialize()`` is described in more detail below. +3. If the Telnet console is enabled, it calls ``nsh_telnetstart()`` which + resides in the NSH library. ``nsh_telnetstart()`` will start the Telnet + daemon that will listen for Telnet connections and start remote + NSH sessions. +4. If a local console is enabled (probably on a serial port), then + ``nsh_consolemain()`` is called. ``nsh_consolemain()`` also resides + in the NSH library. ``nsh_consolemain()`` does not return so that + finished the entire NSH initialization sequence. + +``nsh_initialize()`` +^^^^^^^^^^^^^^^^^^^^ + +The NSH initialization function, ``nsh_initialize()``, be found in +``apps/nshlib/nsh_init.c``. It does only three things: + +``nsh_romfsetc()`` if so configured, it executes an NSH start-up script that +can be found at ``/etc/init.d/rcS`` in the target file system. +``/etc`` is the location where a read-only, ROMFS file system is mounted by +``nsh_romfsetc()``. The ROMFS image is, itself, just built into the firmware. +By default, this rcS startup script contains the following logic: + +.. code-block:: sh + + # Create a RAMDISK and mount it at XXXRDMOUNTPOUNTXXX + + mkrd -m XXXMKRDMINORXXX -s XXMKRDSECTORSIZEXXX XXMKRDBLOCKSXXX + mkfatfs /dev/ramXXXMKRDMINORXXX + mount -t vfat /dev/ramXXXMKRDMINORXXX XXXRDMOUNTPOUNTXXX + +Where the ``XXXX*XXXX`` strings get replaced in the template +when the ROMFS image is created: + +* ``XXXMKRDMINORXXX`` will become the RAM device minor number. Default: ``0``. +* ``XXMKRDSECTORSIZEXXX`` will become the RAM device sector size. +* ``XXMKRDBLOCKSXXX`` will become the number of sectors in the device. +* ``XXXRDMOUNTPOUNTXXX`` will become the configured mount point. + Default: ``/etc``. + +This script will, then, create a RAMDISK, format a FAT file system +on the RAM disk, and then mount the FAT filesystem at a configured mountpoint. +This rcS template file can be found at ``apps/nshlib/rcS.template``. +The resulting ROMFS file system can be found in +``apps/nshlib/nsh_romfsimg.h``. + +* ``boardctl()`` - Next any architecture-specific NSH initialization will be + performed (if any). The NSH initialization logic will all the non-standard + OS interface ``boardctl()`` like: ``(void)boardctl(BOARDIOC_INIT, 0);``. + The first argument, the command ``BOARDIOC_INIT``, indicates that the + ``boardctl()`` is being requested to perform application-oriented + initialization. In response to this command, ``boardctl()`` will call the + board-specific implementation of ``board_app_initialize()``. + That function is not generally available to application level code in + all configurations but can always be accesses via ``boardctl()``. + +* ``board_app_initialize()`` - For the STM32F4Discovery, this architecture + specific initialization can be found at + ``boards/arm/stm32/stm32f4discovery/src/stm32_appinit.c``. + This it does things like: + + 1. Initialize SPI devices. + 2. Initialize SDIO. + 3. Mount any SD cards that may be inserted. + + +There are two possibilities then for application startup initialization: + +1. In the application itself via ``boardctl(BOARDIOC_INIT, 0)``, OR +2. in the OS using ``board_late_initialize()``. + +Each works well in certain contexts but there are also things that I don't +like about either possibility. + +``boardctl()`` +-------------- + +You can see this application-controlled initialization in, for example, +the NSH application in `nuttx-apps/nshlib``. This is seen as the call to +``boardctl(BOARDIOC_INIT, 0)``. This requires that the ``boardctl()`` +interface be enabled with ``CONFIG_LIB_BOARDCTL``. + +When ``CONFIG_LIB_BOARDCTL=y``, board-specific OS internal logic must provide +the interface ``board_app_initilize()``. ``boardctl(BOARDIOC_INIT, 0)`` is the +application level interface that provides the proper interface to +``board_app_initilize()``. All board-specific, driver-level initialization may +then be performed in ``board_app_initilize()`` under the control +of the application. + +With ``CONFIG_LIB_BOARDCTL=y``, there are two levels of application-specific +initialization: +(a) an OS/kernel level application initialization that needs to be performed +before the application is started, and +(b) a user/application level initialization that can be performed after +the application has started. + +``board_late_nitialize()`` +-------------------------- + +``board_late_initialize()`` is the OS/kernel level application initialization +that is performed before the application has started. It usually works +quite well and is usually a good alternative for ``boardctl(BOARDIOC_INIT)``. +To avoid issues with trying to perform initialization on the IDLE thread, +``board_late_initialize()`` is called from an internal kernel thread. + +The IDLE thread has limitations: It cannot wait for events so +it can only be used for simple, straight-line initialization logic. +That may be insufficient in some cases. +For this reason, ``board_late_initialize()`` must run on a kernel thread. + +Consider the ``board_late_initialize()`` calling sequence: +``board_late_initialize()`` is called directly from ``do_app_start()`` in +``nuttx/sched/init/nx_bringup.c`` just before starting the application task. +``nx_create_initthread()`` is, in turn, called from the function +``nx_start_application()``. If ``CONFIG_BOARD_LATE_INITIALIZE`` is +not defined, then this is just a normal C function call. +But if ``CONFIG_BOARD_LATE_INITIALIZE`` is defined, then an intermediate, +trampoline kernel thread is started. That kernel thread executes +``do_app_start()`` and moves the initialization off of the IDLE thread. +This works great but is a little more complex than I would like. + +``apps/platform`` +----------------- + +There is also special place for user/application initialization. +That is the ``nuttx-apps/platform`` platform directory. This directory +should be a mirror of the ``nuttx/configs`` directory. +There should be a board directory in ``nuttx-apps/platform`` for every +board that is in ``nuttx/configs``. diff --git a/Documentation/implementation/nuttx_tasking.rst b/Documentation/implementation/nuttx_tasking.rst new file mode 100644 index 00000000000..90a87b9e1e1 --- /dev/null +++ b/Documentation/implementation/nuttx_tasking.rst @@ -0,0 +1,380 @@ +.. _nuttx-tasking: + +============= +NuttX Tasking +============= + +An RTOS as a library +==================== + +What is an RTOS? NuttX, as with all RTOSs, is a collection of various features +bundled as a library. It does not execute except when either: + +1. The application calls into the NuttX library code, OR +2. An interrupt occurs. + +There is no meaningful way to represent an architecture that is implemented +as a library of user managed functions with a diagram. +You can however, pick any subsystem of an RTOS and represent +that in some fashion. + + +Kernel Threads +============== + +There are some RTOS functions that are implemented by internal threads, +for instance :ref:`kernel-threads-vs-pthreads`, :ref:`tasks-vs-threads`, +:ref:`kernel-modules`. + +.. todo:: Provide more content here :-) + + +The Scheduler +============= + +Schedulers and Operating Systems +-------------------------------- + +An operating system is a complete environment for developing applications. +One important component of an operating system is the scheduler. +That logic that controls when tasks or threads execute. + +Actually, more than that; the scheduler really determines what a task +or a thread is! Most tiny operating systems are really not operating +“systems” in the sense of providing a complete operating environment. +Rather these tiny operating systems consist really only of a scheduler. +That is how important the scheduler is. + +Task Control Block (TCB) +------------------------ + +In NuttX a thread is any controllable sequence of instruction execution +that has its own stack. +Each task is represented by a data structure called a task control block +or TCB. That data structure is defined in the header file +``include/nuttx/sched.h``. + +Task Lists +---------- + +These TCBs are retained in lists. The state of a task is indicated both +by the ``task_state`` field of the TCB and by a series of task lists. +Although it is not always necessary, most of these lists are prioritized +so that common list handling logic can be used (only the ``g_readytorun``, +the ``g_pendingtasks``, and the ``g_waitingforsemaphore`` lists +need to be prioritized). + +All new tasks start in an initial, non-running state: + +.. code-block:: c + + volatile dq_queue_t g_inactivetasks; + +* This is the list of all tasks that have been initialized, but not yet + activated. NOTE: This is the only list that is not prioritized. + +* When the task is initialized, it is moved to a ready-to-run list. + There are two lists representing ready-to-run threads and several + lists representing blocked threads. Here are the ready-to-run threads: + +.. code-block:: c + + volatile dq_queue_t g_readytorun; + +* This is the list of all tasks that are ready to run. + The head of this list is the currently active task; + the tail of this list is always the idle task. + +.. code-block:: c + + volatile dq_queue_t g_pendingtasks; + +* This is the list of all tasks that are ready-to-run, but cannot be placed + in the ``g_readytorun`` list because: + + 1. They are higher priority than the currently active task at the head + of the ``g_readytorun`` list, AND + 2. the currently active task has disabled pre-emption. + + These tasks will stay in this holding list until pre-emption is again + enabled (or until the currently active task voluntarily relinquishes + the CPU). + +* Tasks in the ``g_readytorun`` list may become blocked. + In this cased, their TCB will be moved to one of the blocked lists. + When the block task is ready-to-run, its TCB will be moved back to either + the ``g_readytorun`` to ``the g_pendingtasks`` lists, depending up + if pre-emption is disabled and upon the priority of the tasks. + +Here are the blocked task lists: + +.. code-block:: c + + volatile dq_queue_t g_waitingforsemaphore; + +* This is the list of all tasks that are blocked waiting for a semaphore. + +.. code-block:: c + + volatile dq_queue_t g_waitingforsignal; + +* This is the list of all tasks that are blocked waiting for a signal + (only if signal support has not been disabled). + +.. code-block:: c + + volatile dq_queue_t g_waitingformqnotempty; + +* This is the list of all tasks that are blocked waiting for a message queue + to become non-empty (only if message queue support has not been disabled). + +.. code-block:: c + + volatile dq_queue_t g_waitingformqnotfull; + +* This is the list of all tasks that are blocked waiting for a message queue + to become non-full (only if message queue support has not been disabled). + +.. code-block:: c + + volatile dq_queue_t g_waitingforfill; + +* This is the list of all tasks that are blocking waiting for a page fill + (only if on-demand paging is selected). + +Reference: ``nuttx/sched/sched/sched.h``. + + +State Transition Diagram +======================== + +The state of a thread can then be easily represented with this simple state +transition diagram. + +.. todo:: Provide State Transition Diagram. + + +Scheduling Policies +=================== + +In order to be a real-time OS, an RTOS must support ``SCHED_FIFO``. +That is, strict priority scheduling. The thread with the highest priority +runs.. Period. The thread with the highest priority is always associated +with the TCB at the head of the ``g_readytorun`` list. + +NuttX supports one additional real-time scheduling policy: ``SCHED_RR``. +The RR stands for **round-robin** and this is sometimes called +**round-robin scheduling**. In this case, NuttX supports timeslicing. +If a task with ``SCHED_RR`` scheduling policy is running, then when each +timeslice elapses, it will give up the CPU to the next task that is +at the same priority. + +.. note:: + + 1. If there is only one task at this priority, ``SCHED_RR`` and + ``SCHED_FIFO`` are the same, AND + 2. ``SCHED_FIFO`` tasks are never pre-empted in this way. + + +Task IDs +======== + +Each task is represented not only by a TCB but also by a numeric task ID. +Given a task ID, the RTOS can find the TCB. +Given a TCB, the RTOS can find the task ID. +So they are functionally equivalent. +Only the task ID, however, is exposed at the RTOS/application interfaces. + + +NuttX Tasks +=========== + +Processes vs. Threads +--------------------- + +In larger system OS such as BSD, Linux, or Windows you will often hear +the name process used to refer to threads managed by the OS. + +A process is more than a thread as we have been discussing so far. +A process is a protected environment that hosts one or more threads. +By environment we mean the set of resources set aside by the OS but +in the case of the protected environment of the process we are specifically +referring its address space. + +.. note:: + + In order to implement the process' address space, the CPU must support + a memory management unit (MMU). + **The MMU is used to enforce the protected process environment.** + +However, NuttX was designed to support the more resource constrained, +lower-end, deeply embedded MCUs. Those MCUs seldom have an MMU and, +as a consequence, can never support processes as are supported by BSD, Linux, +or Windows. + +.. important:: NuttX does not support processes. + +NuttX will support an MMU but it will not use the MMU to support processes. +NuttX operates only in a flat address space. +NuttX will use the MMU to control the instruction and data caches and +to support protected memory regions. +This may change in future, but this is how things are right now. + + +NuttX Tasks and Task Resources +------------------------------ + +All RTOSs support the notion of a task. A task is the RTOS's moral equivalent +of a process. Like a process, a task is a thread with an environment +associated with it. + +This environment is like environment of the process but does not include +a private address space. +This environment is private and unique to a task. +Each task has its own environment. + +This task environment consists of a number of resources +(as represented in the TCB). Of interest in this discussion are the following. +Note that any of these task resources may be disabled in the NuttX +configuration to reduce the NuttX memory footprint: + +1. **Environment Variables**. This is the collection of variable assignments + of the form: ``VARIABLE=VALUE``. + +2. **File Descriptors**. A file descriptor is a task specific number + that represents an open resource (a file or a device driver, for example). + +3. **Sockets**. A socket descriptor is like a file descriptor, but + the open resource in this case is a network socket. + +4. **Streams**. Streams represent standard C buffered I/O. + Streams wrap file descriptors or sockets to provide a new set of interface + functions for dealing with the standard C I/O (like ``fprintf()``, + ``fwrite()``, etc.). + +In NuttX, a task is created using the interface ``task_create()``. + +NuttX Task Exit Sequence +------------------------ + +.. figure:: task_exit_sequence.png + :alt: Task Exit Sequence diagram. + + Task Exit Sequence diagram. + + +The Pseudo File System and Device Drivers +========================================= + +A full discussion of the NuttX file system belongs elsewhere, +see :ref:`nuttx-filesystem` for more details. +But in order to talk about task resources, we also need to have +a little knowledge of the NuttX file system. + +NuttX implements a Virtual Files System (VFS) that may be used to communicate +with a number of different entities via the standard ``open()``, ``close()``, +``read()``, ``write()``, etc, interfaces. +Like other VFSs, the NuttX VFS will support file system mount points, +files, directories, device drivers, etc. + +Also, as with other VFSs, the NuttX file system will support +pseudo-file systems, that is, file systems that appear as normal media +but are really presented under programmatic control. +In Linux, for example, you have the ``/proc`` and the ``/sys`` +psuedo-file systems. +There is no physical media underlying the pseudo-file system. + +The NuttX root file system is always a psuedo-file system. +This is just the opposite from Linux. With Linux the root file system +must always be some physical block device (if only an initrd ram disk). +Then once you have mounted the physical root file system, you can mount +other file systems – including Linux pseudo-filesystems like ``/proc`` +or ``/sys``. + +With NuttX, the root file system is always +a pseudo-file system that does not require any underlying block driver +or physical device. +Then you can mount real filesystem in the pseudo-filesystem. + +This arrangement makes life much easier for the tiny embedded world (but also +has a few limitations — like where you can mount file systems). + +**NuttX interacts with devices via device drivers** – that is via software +that controls hardware and conforms to certain NuttX conventions +(see ``include/nuttx/fs/fs.h``). Device drivers are represented +by device nodes in the pseudo-file system. +By convention, these device nodes are created in the ``/dev`` directory. + +Now that we have digressed a little to introduce the NuttX file system +and device nodes, we can return to our discussion of task resources. + + +``/dev/console`` and Standard Streams +------------------------------------- + +There are three special cases of I/O: ``stdin``, ``stdout``, and ``stderr``. +These are type ``FILE*`` and correspond to file descriptors ``0``, ``1``, +and ``2`` respectively. +When the very first thread is created (called the IDLE thread), +the special device node ``/dev/console`` is opened. ``/dev/console`` provides +the ``stdin``, ``stdout``, and ``stderr`` for the initial task. + + +Inheritance of the Task Environment and I/O Redirection +======================================================= + +When one task creates a new task, that new task inherits the task resources +of its parent. This includes all of the environment variables, +file descriptors, and sockets. + +.. note:: + + Task resources inheritance can be limited by special options + in the NuttX configuration. + +So, if nothing special is done, then every task will use ``/dev/console`` +for the standard I/O. However, a task may close file descriptor +``0`` through ``2`` and open a new device for standard I/O. +Then any children tasks that are created will inherit that new re-directed +standard I/O as well. + +This mechanism is used throughout NuttX. +For example in the THTTPD server to redirect socket I/O to standard I/O +for CGI tasks. In the Telnet server so that new tasks inherit the +Telnet session. + + +Tasks vs. Pthreads +================== + +Systems like Linux also support POSIX pthreads. +In the Linux environment, the process is created with one thread running +in it. But by using interfaces like ``pthread_create()``, you can create +multiple threads that run and share the same process resources. + +NuttX also supports POSIX pthreads and the NuttX pthreads also support +this behavior. That is, the NuttX POSIX pthreads also share the resources +of the parent task. + +However, since NuttX does not support process address environments, +the difference is not so striking. +When a task creates a pthread, the newly create pthread will share +the environment variables, file descriptors, sockets, and streams +of the parent task. + +.. note:: + + Task resources are reference counted and will persist + as long as a thread in the task group is still active. + +See :ref:`tasks-vs-threads` for more details. + + +Process IDs / Task IDs / Pthread IDs +==================================== + +The term process ID is standard (usually abbreviated as pid) and used to +identify a task in NuttX. So, more technically, this number is a task ID +as was described above. +Pthreads are also described by a ``pthread_t`` ID. +In NuttX, the ``pthread_t`` ID is also the same task ID. diff --git a/Documentation/implementation/oneshot_timers_and_cpu_load.rst b/Documentation/implementation/oneshot_timers_and_cpu_load.rst new file mode 100644 index 00000000000..512a9637d2a --- /dev/null +++ b/Documentation/implementation/oneshot_timers_and_cpu_load.rst @@ -0,0 +1,126 @@ +======================================= +Oneshot Timers and CPU Load Measurement +======================================= + +Issues with CPU Load Measurement +================================ + +There is been support for CPU load measurement for a long time. +Enabled with: + +.. code-block:: sh + + CONFIG_SCHED_CPULOAD=y + CONFIG_SCHED_CPULOAD_TIMECONSTANT=2 + +The main problem with the existing logic is that it ran synchronously +with the system timer. That turns out to be useless for measurement of the +performance of some tasks. + +Many tasks operate synchronously with the timer interrupt, i.e., timed event +is the stimulus for task execution. +Those tasks are always sampled at essentially the same point in the execution +profile leading to nonsense results. + +In order to get believable results in all cases you would need to: + +* Sample at times are completely random with respect to program behavior, AND +* Sample at a high rate or for a very long time. + + +External Clock +============== + +In order to work around these things, an option to use an external clock +was added: + +.. code-block:: sh + + CONFIG_SCHED_CPULOAD_EXTCLK=y + CONFIG_SCHED_CPULOAD_TICKSPERSEC=77 + +This should work well if ``CONFIG_SCHED_CPULOAD_TICKSPERSEC`` is prime +and less than ``CONFIG_USEC_PER_TICK``. +However, you were completely on your own one how to implement this +external clock. But no longer. + + +Using a Oneshot Timer to Drive CPU Load Measurement +=================================================== + +Recently, a generic, platform-independent one-shot lower half interface +was developed. That interface is described in +``include/nuttx/timers/oneshot.h``. There are implementations of the one shot +timer lower half available for STM32, STM32L4, SAM4CM, SAMA5D3/4, +and SAMV71/SAME70. + +And now there is also an OS component that can be initialized and used to +drive the CPU load sampling from a precision, high rate oneshot timer. +The implementation of that feature resides at +``sched/sched/sched_cpuload_oneshot.c`` and is enabled with: + +.. code-block:: sh + + CONFIG_ONESHOT=y + CONFIG_CPULOAD_ONESHOT=y + +Perhaps with additional configuration options that are likely required +to enable board-specific oneshot timer lower-half support. +The oneshot timer must still be configured by board specific logic which +must call: + +.. code-block:: c + + void sched_oneshot_extclk(FAR struct oneshot_lowerhalf_s *lower); + +To start the CPU load measurement. + +``sched_oneshot_extclk()`` is prototyped in ``include/nuttx/clock.h``. +There is some example setup code in the NuttX simulation code at +``boards/sim/sim/sim/src/sim_bringup.c``. + + +Entropy +======= + +Another recent addition to NuttX came from David Alessio. +David contributed support for ``/dev/urandom`` with a built-in XorShift128 +pseduo-random number generator (PRNG). + +We have detached the XorShift128 implementation from the ``/dev/urandom`` +implementation and moved it to ``/libc/misc`` so that is it available +for other purposes. + +In particular, there is an option to use the XorShift128 PRNG to add entropy +to the CPU load measurement of the oneshot timer. +That feature is enabled with: + +.. code-block:: sh + + CONFIG_CPULOAD_ONESHOT_ENTROPY=n + +for ``n=1..30`` and disabled with ``CONFIG_CPULOAD_ONESHOT_ENTROPY=0``. +This value represents the number of bits of entropy that will be added +to the oneshot interval delays. + +The oneshot timer will be set to the following interval each time +the oneshot timer is restarted is ``CONFIG_CPULOAD_ONESHOT_ENTROPY``: + +.. code-block:: sh + + CPULOAD_ONESHOT_NOMINAL \ + - (CPULOAD_ONESHOT_ENTROPY / 2) \ + + error \ + + nrand(CPULOAD_ONESHOT_ENTROPY) + +Where: + +* ``CPULOAD_ONESHOT_NOMINAL`` is ``CONFIG_SCHED_CPULOAD_TICKSPERSEC`` + in units of microseconds. +* ``CPULOAD_ONESHOT_ENTROPY`` is ``(1 << CONFIG_CPULOAD_ONESHOT_ENTROPY)``. +* ``error`` is an error value that is retained from interval to interval + so that although individual intervals are randomized, + the average will still be equal to ``CONFIG_SCHED_CPULOAD_TICKSPERSEC``. + +If ``CONFIG_CPULOAD_ONESHOT_ENTROPY=0``, then the interval delay will +always be equal to ``CPULOAD_ONESHOT_NOMINAL``. diff --git a/Documentation/implementation/power_management.rst b/Documentation/implementation/power_management.rst new file mode 100644 index 00000000000..3a113fa002d --- /dev/null +++ b/Documentation/implementation/power_management.rst @@ -0,0 +1,221 @@ +.. power-management: + +================ +Power Management +================ + +Power Management (PM) Subsystem +=============================== + +I would expect that the logic that performs clocking adjustments would utilize +the NuttX Power Management (PM) subsystem. That subsystem basically implements +a random walk. It collects and monitors system utilization information from +devices drivers. This utilization information comes from critical device +drivers via calls to:: + + void pm_activity(int domain, int priority); + +This function is called by a device driver to indicate that it is performing +meaningful activities (non-idle). This increments an activity counter and will +prevent entering reduced power states. + +For a system driver by a human interface, ``pm_activity()`` might be called +with touchscreen in puts and/or keypad inputs. +As long as the human is interacting with the device it remains non-idle. + +When there is no further activity, the activity counter will decrease +and when it crosses a threshold, it will initiate a change to a lower power +usage state. + +There are many actions that systems may take when the reduced power state +is entered. They may, for example, dim that backlight on a display or enable +MCU-specific reduced power consumptions modes. And for the purposes +of this discussion, they may also reduce the clocking to the CPU. + +Alternative way to conserve power is using :ref:`tickless` with some +(dis)advantages you need to consider to know if it fits your needs. + + +.. _dynamic-clocking: + +Dynamic Clocking +================ + +Most configurations are created with fixed clock configurations. +One strategy for power management is to use variable, dynamic clocking +where the CPU clock frequency is lowered during periods of inactivity. + +Clock Update Function +--------------------- + +Variable clocking is not difficult to achieve but most ports are created +with fixed clock configuration using settings defined in the ``board.h`` +header file. For example, Kinetis has:: + + void kinetis_clockconfig(void) + +In order to make clocking variable a few things are needed: + +* The initial board settings should still be used at power up, + but there needs to be a new clock initialization function, + perhaps ``kinetis_clock_update()`` that takes a structure containing + all of the clock settings. +* The existing ``kinetis_clockconfig()`` would need to be gutted, + most of the logic being moved to ``kinetis_clock_update()``. + ``kinetis_clockconfig()`` would just create the structure using the constant + settings from the ``board.h`` header file then call + ``kinetis_clock_update()`` to establish the initial clock configuration. + +.. note:: Additional complexities may be associated with changing the clocking + other than just controlling dividers, PLLs, and clock sources. For example, + FLASH wait states. When increasing the clocking, the FLASH wait states must + be increased BEFORE reconfiguring the faster clocking. + But when reducing the clocking, FLASH wait states cannot be reduced until + AFTER reconfiguring the slower clocking. + +Handling Driver Dependencies +---------------------------- + +The first part of the problem was modifying the clock configuration logic +to support variable configurations. +The second part of the problem is that now all of the devices that depend +on clocking such as the timer, the system timer, all MCU-specific timers, +UARTs, and perhaps other device need to be notified of the change +in clock frequency. + +* The system timer, the SysTick in Cortex-M MCUs, needs to be notified + of the of the clock frequency change so that it can re-calculate + the system timer tick interrupt so that there is no disruption in timing. +* The same may be true of other MCU-specific timers. They may also need to + recalculate certain clocking. An MCU-specific timer may, for example, + be providing system time in Tickless mode. +* If serial devices are used, such as for a serial console, then + serial drivers must be notified of the clock change so that they can + recalculate their BAUD settings without lost of serial communications. +* Other devices such as SPI, I2C, I2S, SDIO, USB, etc. may also need to + be notified of the change in the clock configuration to behave correctly + across the clocking change. + +Full functionality is not normally expected in low power consumption states. +So the most typical behavior for drivers in response to reduced power +consumption state changes is simply to shut themselves down (turn off +the peripheral, disable clocking to the peripheral, and other actions +as appropriate). + +The above discussion only applies to peripherals that you expect +to continue normal operation in the reduced power state. +You may, as an example, want to keep timing accuracy throughout +the low power state or you may want to preserve serial debug output +in the reduced power state. + + +Driver PM Callback Functions +---------------------------- + +The NuttX PM subsystem provides the glue that integrates both parts +of the clock management problem. + +The first part is managed by PM activity monitor and PM state changes. +The PM subsystem integrates the second part of the problem using PM driver +callback functions. The timing sensitive device drivers can be notified +of the change in clocking using the driver callbacks that are a port +of NuttX Power Management (PM) functionality. + +Each timing sensitive device driver needs to register for a PM callback and, +on each callback, it must recalculate its frequency settings using the current +clock configuration. + +To receive this callbacks, the driver logic needs to provide functions +with prototypes like: + +.. code-block:: c + + /* Power management callback function prototypes */ + + #ifdef CONFIG_PM + static void xyz_pm_notify(struct pm_callback_s *cb, int domain, + enum pm_state_e pmstate); + static int xyz_pm_prepare(struct pm_callback_s *cb, int domain, + enum pm_state_e pmstate); + #endif + +Which are used to notify a callback vtable like: + +.. code-block:: c + + /* Power management callback function vtable */ + + #ifdef CONFIG_PM + static struct pm_callback_s g_xyz_pmcb = + { + .notify = xyz_pm_notify, + .prepare = xyz_pm_prepare, + }; + #endif + +The ``prepare()`` callback method gives the driver some advance notification +of the pending PM state change. + +Some drivers may need to put the device in a safe state in the ``prepare()`` +callback. +A serial driver, for example, may want to disable RX and TX so that garbage +is not generated or received during the clock change. + +The ``notify()`` callback method signifies the actually state change +and it is in this method that the driver logic should recalculate its timing +parameters based on the new clock settings and also resume an operations +that were disabled by the ``prepare()`` callback. + +The driver registers its PM callback vtable by calling pm_register() like: + +.. code-block:: c + + /* Register to receive power management callbacks */ + + int ret = pm_register(&g_xyz_pmcb); + +Dynamic Clock Frequency Utility Functions +----------------------------------------- + +Most timers and drivers in the system us the constant frequency values defined +in the ``board.h`` header file. In order to have variable time, the drivers +must be able to determine the current clock frequencies dynamically, +not via a fixed, constant definitions. +This means that there must be clock functions that derive various clock +firequencies in the clock distribution knowing only the input frequency +(i.e., crystal frequency of internal clock selection). + +When the driver receives the PM ``notify()`` informing it that the clocking +may have changed, it must call these functions to get the new clock settings +instead of using the constant settings in board.h. + +There is something similar in the source tree now for the case where NuttX +is started by a bootloader. In that case, the clock configuration is +set up by the bootloader, usually u-boot, and each driver that needs to know +clock frequencies must query the clock configuration to determine +the frequency. + +For the SAMA5D4-EK, that is still handled by definitions in the ``board.h`` +header file. See, for example, ``boards/arm/sama5/sama5d4-ek/include/*.h``. +``board.h`` includes another file that defines the clock setup based +on the configuration. If the SAMA5D4-EK is running out of SDRAM, +then it must have been started by a bootloader and the definitions appear in +``boards/ar/sama5/sama5d4-ek/include/board_sdram.h``. +In that case, the macros are redefined so that they map to a function call +to determine the clock frequency: + +.. code-block:: c + + #define BOARD_MAINCK_FREQUENCY BOARD_MAINOSC_FREQUENCY + #define BOARD_PLLA_FREQUENCY (sam_pllack_frequency(BOARD_MAINOSC_FREQUENCY)) + #define BOARD_PLLADIV2_FREQUENCY (sam_plladiv2_frequency(BOARD_MAINOSC_FREQUENCY)) + #define BOARD_PCK_FREQUENCY (sam_pck_frequency(BOARD_MAINOSC_FREQUENCY)) + #define BOARD_MCK_FREQUENCY (sam_mck_frequency(BOARD_MAINOSC_FREQUENCY)) + +Something like this could also be done for other architectures to support +dynamic clock management. + +You would probably have to do something similar if change the clocking. +``SysTick`` and each device would need a callback from the PM subsystem and each +would have to re-calculate is BAUD based on the new clock settings using +such functions calls to get the dynamic clock frequency. diff --git a/Documentation/implementation/short_delay.png b/Documentation/implementation/short_delay.png new file mode 100644 index 00000000000..491fb73a868 Binary files /dev/null and b/Documentation/implementation/short_delay.png differ diff --git a/Documentation/implementation/short_time_delays.rst b/Documentation/implementation/short_time_delays.rst new file mode 100644 index 00000000000..50aacc1756b --- /dev/null +++ b/Documentation/implementation/short_time_delays.rst @@ -0,0 +1,300 @@ +.. _short-time-delays: + +================= +Short Time Delays +================= + +System Timer Interrupt +====================== + +This section addresses some counter-intuitive properties of using very short +time delays. + +This section assumes that timing is generated by a system timer +"tick" interrupt. In Tickless Mode there is no system timer interrupt. +Much of the discussion here would also apply in the Tickless mode, however, +the terminology used here assumes system timer interrupts. + + +Timer Resolution +================ + +When we talk about short delays, we are talking about delays that are +on the same order of magnitude as the system timer interrupt interval. + +That interval is controlled the configuration setting +``CONFIG_USEC_PER_TICK``. The default value of ``CONFIG_USEC_PER_TICK`` +is 10,000 microseconds. +That equivalent to a timer interrupt frequency of 100Hz: + +.. code-block:: c + + Ftimer ticks/sec = 1,000,000 usec/sec / CONFIG_USEC_PER_TICK; + +There are many different OS interfaces that implement timed delays +(or function that have timeout values such as ``sem_timedwait()``). +These all behave in basically the same way. + +.. _usleep: + +``usleep()`` +============ + +For simplicity of discussion let's focus on on ``usleep()``. +``usleep()`` is a standard but deprecated interface that simply delays +for a specified number of microseconds. + +.. note:: ``usleep()`` is deprecated in favor of ``clock_nanosleep()``. + + +Requirements/Assumptions of ``usleep()`` +---------------------------------------- + +The prototype for ``usleep()`` is: + +.. code-block:: c + + int usleep(useconds_t usec); + +Where ``usec`` is the number microseconds to delay. +``usleep()`` will return zero unless an error occurs in which case +it will set the errno variable and return ``-1``. + +Now the interesting questions: +Suppose that ``CONFIG_USEC_PER_TICK`` is set to 10,000 microseconds, +What will happen when you try any of the following? + +.. code-block:: c + + usleep(0); + usleep(1); + usleep(10000); + +In order to predict such behavior, we will have to enumerate some of the +assumption and requirements of ``usleep()``: + +1. The contract that ``usleep()`` makes is that it will suspend the calling + thread for at least usec microseconds. It will never, under any condition, + return with a delay smaller than the requestion usec delay. +2. ``usleep()`` may wait longer than the requested delay due to small system + processing overhead, task priorities, and quantization errors. + +Task Priorities +--------------- + +When the requested delay expires, the calling task is ready to run but still +may not actually be able to run for some time due to higher priority tasks +that block execution of the ready to run task. + +Quantization Errors +------------------- + +``usleep()`` is only capable of waiting for multiples of the system timer +interrupt interval (``CONFIG_USEC_PER_TICK``) If the requested ``usec`` delay +is not an even multiple of the system timer interrupt interval, +then it will be rounded up as necessary to satisfy the first requirement above. + +This means that the following are really equivalent delays: + +.. code-block:: c + + usleep(1); + usleep(10000); + +And finally, the behavior that confuses most people.. + +.. note:: + + ``usleep()`` has no knowledge of the the phase of the system timer + when it is started: The next timer interrupt may occur immediately + or may be delayed for almost a full cycle. + In order to meet the contract of the first requirement, the requested + time is also always incremented by one. + +This means, for example, that the the delay: + +.. code-block:: c + + usleep(10000); + +will not delay for one clock tick! That would be impossible! + +There is no event exactly one clock tick after ``usleep()`` is called. +The next timer tick will always occur at some time strictly less than +the system timer interval. +Rather, ``usleep()`` will delay for two clock ticks resulting some +actual delay between 10 and 20 microseconds, exclusive. + +See the following figure: + +.. figure:: short_delay.png + :alt: Short Time Delays in NuttX. + + Short Time Delays in NuttX. + +For the most part, when delays are large – hundreds or thousands +of microseconds – this error is not significant. +It becomes noticeable only if you are using ``usleep()`` +for delays very close to the the timer resolution when the error +can be relatively significant. + +Finally, the easy one: + +.. code-block:: c + + usleep(0); + +This will return immediately with no delay. +That satisfies all requirements and assumptions of the interface. + +Using ``usleep()`` to implement periodic delays +----------------------------------------------- + +The third assumption is necessary because ``usleep()`` has +no knowledge of the current system timer phase. +But it also makes ``usleep()`` a very bad choice for implementing +periodic behavior if the ``usleep()`` delay is close to the system +timer resolution. + +For example, consider a loop such as the following +(again assuming that ``CONFIG_USEC_PER_TICK`` is set +to 10,000 microseconds): + +.. code-block:: c + + for (; ; ) + { + usleep(10000); + /* Do some periodic stuff */ + } + +From the preceding discussion, we know that this will not work: +It will not (and cannot) create periodic processing at the rate +of the system timer interrupt rate but, rather, at half the system +time interrupt rate in this case. + +In this case, that the third assumption of ``usleep()`` is not valid. +``usleep()`` does not start timing some random, unknown phase with respect +to the system timer interrupt. +In this case, ``usleep()`` will be started at a nearly constant phase +with respect to the system timer (at some some short delay after the +system timer interrupt) and will consistently show +near-worst case timing errors. + +Of course, ``usleep()`` cannot know that and, hence, is a bad choice +for implementing such periodic behavior. +The user in this case is assuming that ``usleep()`` simply waits for the +next timer tick to occur. That is not the behavior or ``usleep()`` that +describes some non-existent interface that waits for the next timer interrupt +event, not for a fixed time delay. + +.. note:: + + ``usleep()`` behavior is to wait to assure that at least usec + microseconds has elapsed. And it does that job quite well. + +A final note: The above periodic delay loop can be made to work well, +on the other hand, if the delay provided to ``usleep()`` is significantly +larger that the system timer resolution. + +What can you do? +---------------- + +What can you do to improve the resolution for such high frequency processing +loop? + +First, you might consider increasing the system timer interrupt rate. +You would do this by reducing the value of ``CONFIG_USEC_PER_TICK``. +For example a value of ``1000`` for ``CONFIG_USEC_PER_TICK`` would create +a timer interrupt rate of 1KHz and a minimum loop delay of +perhaps 2 milliseconds. + +The trade-off here is that when you increase the timer interrupt rate, +the timer interrupt processing will then take a proportionately larger amount +of your CPU bandwidth. + +The recommended way to get very high timer resolution without increasing +the timer interrupt rate (in most use cases) is to use a Tickless Mode OS. +For example, in the Tickless configuration, you could have +``CONFIG_USEC_PER_TICK`` set to ``1`` for a 1MHz timer resolution +(with no interrupts). + +If your target periodic processing time is still 1Khz, +then this periodic processing could be met with good precision. + +Another option is to abandon the system timer altogether and use +a dedicated timer peripheral to perform your timed operations with +high precision. +But if you really want to use the system timer than another thing +you should consider would be to implement a Timer Hook. + + +.. timer_hook: + +Timer Hook +========== + +A Timer Hook is a user provided function that is called from the OS +on each timer interrupt. +If you enable ``CONFIG_SYSTEMTICK_HOOK=y`` in your configuration, +then the OS timer interrupt handler will call out to a user-provided function, +``board_timerhook()``, on each timer interrupt. +The full prototype of this function is provided in ``included/nuttx/board.h`` +as: + +.. code-block:: c + + void board_timerhook(void); + +.. note:: + + The timer hook is only available when system timer interrupts + are used; it is not available in Tickless mode. + +This timer hook could be used in a scenario where you would like to have +your task run at the each timer interrupt without the strange rounding +performed by the standard delay functions. + +You might do something like the following as an example: +In your servicing task, implement a loop. At the top of the loop, +you would wait on a semaphore. +In the body of the loop you perform the periodic operation +then return to the wait at the top of the loop. Like: + +.. code-block:: c + + int ret; + + while ((ret = sem_wait(&g_waitsem) >= 0) + { + /* Do periodic operations */ + } + +.. note:: + + If the return value from ``sem_wait()`` is negative then + some unusual event occurred. In normal cases the errno might either: + ``EINTR`` meaning simply that the wait was awakened with a signal. + You can continue to wait in this case. Or ``ECANCELED`` meaning + that the thread has been cancelled and you should abort the periodic + operations. + +In your ``boards////src/`` directory you would implement +``void board_timerhook(void)``. +The implementation of this function could consist of only a single line +of code: It could just post the semaphore on each timer interrupt, +waking of the loop in the servicing task. Like: + +.. code-block:: c + + void board_timerhook(void) + { + (void)sem_post(&g_waitsem); + } + +This is very efficient. There is no context switch overhead at all +getting from the the timing interrupt to to the servicing task. +None at all other that the normal interrupt return logic. +The servicing task should be the highest priority task in the system +to assure that it is the one that runs immediately +when the timer interrupt returns. diff --git a/Documentation/implementation/signal_handlers.rst b/Documentation/implementation/signal_handlers.rst new file mode 100644 index 00000000000..bd20a4fd570 --- /dev/null +++ b/Documentation/implementation/signal_handlers.rst @@ -0,0 +1,192 @@ +.. _signal-handlers: + +=============== +Signal Handlers +=============== + +Signals are used to exchange information between sending and receiving thread. + + +Sending Thread +============== + +Posting Signals +--------------- + +These are the actions that run on the thread that posts the signal. +Signals are initiated in these ways: + +* ``signal/sig_kill.c: nxsig_kill()``. The standard function ``kill()`` is a + simple wrapper around ``nxsig_kill()``. ``nxsig_kill()`` can be used + to send a signal to any task group. It simply sets up the call + to ``nxsig_dispatch()``. +* ``signal/sig_queue.c: nxsig_queue()``. The standard function ``sigqueue()`` + is a simple wrapper around ``nxsig_queue()``. ``nxsig_kill()`` can be used + to send a signal to any task group, passing more information than + is possible with ``kill()``. + It again simply sets up the call to ``nxsig_dispatch()``. +* ``signal/sig_notification.c: nxsig_notification()``. This logic can also + generate signals via a call to ``nxsig_dispatch()``. + But this is part of the internal, NuttX signal notification system. + It sends signals to tasks via the work queue. + +signal/sig_dispatch.c: ``nxsig_dispatch()`` +------------------------------------------- + +.. code-block:: c + + int nxsig_dispatch(pid_t pid, FAR siginfo_t *info); + +This is the front-end for ``nxsig_tcbdispatch()`` that should be typically +be used to dispatch a signal. If ``HAVE_GROUP_MEMBERS`` is defined, +then this function will follow the group signal delivery algorithms: + +This front-end does the following things before calling +``nxsig_tcbdispatch()``: + +1. With ``HAVE_GROUP_MEMBERS`` defined: + + 1. Get the TCB associated with the ``pid``. + 2. If the TCB was found, get the group from the TCB. + 3. If the PID has already exited, lookup the group that that was + started by this task. + 4. Use the group to pick the TCB to receive the signal. + 5. Call ``nxsig_tcbdispatch()`` with the TCB. + +2. With ``HAVE_GROUP_MEMBERS`` not defined: + + 1. Get the TCB associated with the ``pid``. + 2. Call ``nxsig_tcbdispatch()`` with the TCB + +group/group_signal.c: ``group_signal()`` +---------------------------------------- + +.. code-block:: c + + int group_signal(FAR struct task_group_s *group, FAR siginfo_t *siginfo); + +Send a signal to the appropriate member(s) of the group. +This is typically called from ``nxsig_dispatch()`` as described above +but may also be called from ``task/task_exithook.c`` to handle +Death-of-Child (``SIGCHLD``) signals. + +group/group_signal.c: ``group_signal_handler()`` +------------------------------------------------ + +.. code-block:: c + + static int group_signal_handler(pid_t pid, FAR void *arg) + +Callback from ``group_foreachchild()`` that handles one member of the group. + +signal/sig_dispatch.c: ``nxsig_tcbdispatch()`` +---------------------------------------------- + +.. code-block:: c + + int nxsig_tcbdispatch(FAR struct tcb_s *stcb, siginfo_t *info) + +All signals received the task (whatever the source) go through this function +to be processed. This function is responsible for: + +1. Determining if the signal is blocked. +2. Queuing and dispatching signal actions +3. Unblocking tasks that are waiting for signals +4. Queuing pending signals. + +This function will deliver the signal to the specific task associated +with the specified TCB. + +This function is also called when the OS needs to deliver a signal +to a specific task. +It is normally only called via ``group_signal_handler()`` so that is follows +the rules of signal deliver in multi-threaded tasks. +But it is also called from a few other places: + +1. ``pthread/pthread_condtimedwait.c: pthread_condtimedout().`` + Used to wake-up a specific task waiting on a condition. +2. ``task/task_exithook.c: task_sigchild()`` + Used to send the Death-of-Child signal, ``SIGCHLD``, to the parent task. + +For unmasked signals that have a signal handler attached, +``nxsig_tcbdispatch()`` will call the architecture-specific interface, +``up_schedule_sigaction()``. + +arch/xxx/src/xxx/up_schedsigaction.c: ``up_schedule_sigaction()`` +----------------------------------------------------------------- + +.. code-block:: c + + void up_schedule_sigaction(struct tcb_s *tcb, sig_deliver_t sigdeliver) + +Where sigdeliver is always the OS function ``nxsig_deliver()``. + +This function is called by the OS when one or more signal handling actions +have been queued for execution for a specific task. + +The architecture specific-code must configure things so that the sigdeliver +callback (i.e., ``nxsig_deliver()``) is executed on the thread specified +by tcb as soon as possible. This amounts to saving the signal handler state +information in the TCB so that when the receiving task next executes, +it will be the signal handler that runs, not the normal, uninterrupted thread. + +One of the fixups performed by ``up_schedule_sigaction()`` is to force +the address of the signal handler to point to the trampoline function +``up_sigdeliver()``. + +There are other special cases when the signal is generated from interrupt +handler or when the a task signals itself for some reason. +Those variations are not addressed here. + + +Receiving Thread +================ + +signal/sig_deliver.c: ``nxsig_deliver()`` +----------------------------------------- + +.. code-block:: c + + void nxsig_deliver(FAR struct tcb_s *stcb) + +This function is called on the thread of execution of the signal receiving task +when that task next runs again. It processes all queued signals then returns. + +The mechanism by which a signal is deliver depends on the build configuration; +in PROTECTED and KERNEL build modes, it must go through a trampoline, +``up_signal_dispatch()`` to handle user-space signal actions. +``up_signal_dispatch()`` will drop from kernel mode to user mode, +then call the signal handler. + +Otherwise, the signal handler is called directly from ``nxsig_deliver()``. +When the signal handler returns, the action is over and there is only +clean up to be done. + +arch/xxx/src/xxx/up_sigdeliver.c: ``up_sigdeliver()`` +----------------------------------------------------- + +.. code-block:: c + + void up_sigdeliver(void) + +This is the a signal handling trampoline. +Logic in ``up_schedule_sigaction()`` forced the signal action to be +delivered to ``up_sigdeliver()``. + +``up_sigdeliver()`` will do such things as: + +1. Set up the state to return to the normal, uninterrupted thread + when the signal handler exits. +2. Make sure that the signal handler runs with interrupts enabled. +3. Invoke the signal handler. + +When the signal handler returns this function will: + +1. Free up resources committed by ``up_schedule_sigaction()``. +2. Restore the interrupt state. +3. And perform a context switch to return to the normal, uninterrupted thread. + +When the uninterrupted thread is next suspended, the return will go back to +``nxsig_deliver()`` which will continue delivering signals. +(Hmmm.. shouldn't any other queued signal actions be handled first +before returning to the normal, uninterrupted thread?) diff --git a/Documentation/implementation/smp.rst b/Documentation/implementation/smp.rst new file mode 100644 index 00000000000..2663db51e2c --- /dev/null +++ b/Documentation/implementation/smp.rst @@ -0,0 +1,872 @@ +.. _smp: + +=============================== +SMP (Symmetric MultiProcessing) +=============================== + +Definition +========== + +According to Wikipedia: + + "Symmetric multiprocessing (SMP) involves a symmetric + multiprocessor system hardware and software architecture where two or more + identical processors connect to a single, shared main memory, have full access + to all I/O devices, and are controlled by a single operating system instance + that treats all processors equally, reserving none for special purposes. + Most multiprocessor systems today use an SMP architecture. + In the case of multi-core processors, the SMP architecture applies to + the cores, treating them as separate processors. + + (..) + + SMP systems are tightly coupled multiprocessor systems with a pool + of homogeneous processors running independently, each processor executing + different programs and working on different data and with capability + of sharing common resources (memory, I/O device, interrupt system and so on) + and connected using a system bus or a crossbar." + + -- Source: https://en.wikipedia.org/wiki/Symmetric_multiprocessing. + + +Development Status +================== + +SMP support is complete and stable in NuttX on several multi-core platforms. + + +Enabling SMP +============ + +SMP can be enabled on NuttX with the following configuration settings: + +* ``CONFIG_SMP`` - Enables support for Symmetric Multi-Processing (SMP) + on a multi-CPU platform. +* ``CONFIG_SMP_NCPUS`` - This value identifies the number of CPUs support + by the processor that will be used for SMP. +* ``CONFIG_SMP_IDLETHREAD_STACKSIZE`` - Each CPU will have its own IDLE task. + System initialization occurs on CPU0 and uses + ``CONFIG_IDLETHREAD_STACKSIZE``. + This setting provides the stack size for the IDLE task on CPUS 1 + through ``(CONFIG_SMP_NCPUS-1)``. + +This section provides the origin design specification for the implemention. +As a result, you may find that the test uses future and conditional tenses +when describing the implementation of SMP on NuttX. + +This design has been maintained and now reflects the current "as-built" +state of SMP in NuttX. + + +Design Requirements +=================== + +The basic design requirements are pretty simple: + +1. Need to be able to bring up NuttX running on multiple CPUs. +2. Need data structures to manage multiple active tasks. +3. Need to be able to schedule tasks on other CPUs. +4. Need to be able to modify tasks running on other CPUs. +5. Need to be able to manage critical sections on all CPUs. +6. Need spinlocks to block on all CPUs in all cases: semaphore, + signal, message queue, etc. +7. Need to understand how some non-standard NuttX operations things + like disabling pre-emption work. + + +Data Structures +=============== + +Task Lists +---------- + +At the core of the NuttX design are data structures called +**Task Control Blocks** or just **TCB**. + +These data structures contain everything-you-need-to-know about +a thread or task. +These TCBs are retained in lists within the RTOS. +The state of a thread or task is then determined by which list +the TCB resides in. + +The Read-To-Run Task List +------------------------- + +On such TCB list is of particular importance in the implementation of SMP. +That is the so-called ready-to-run list, ``g_readytorun``. +That list contains the TCB of every task or thread that is not blocked +in any way and so is, well, ready to run. + +The ``g_readytorun`` is a prioritized list. The lowest priority task +is in the list is the one at the end of the list and that must always +by the IDLE task. + +That is the only task/thread that is permitted to have priority 0. +The highest priority, read-to-run task is always at the head of +``g_readytorun`` and must be the currently executing task. + +All other tasks after this is eligle to run, but not currently running. + +The Assigned Task List +---------------------- + +In order to support SMP, the function of the ``g_readytorun`` list +must change. This ``g_readytorun`` should still exist but it should +now contain only: + +1. Only tasks/threads that are eligible to run, but not currently running, AND +2. Tasks/threads that have not been assigned to a CPU. + +For SMP support there should be an array of assigned tasks like: + +.. code-block:: c + + volatile dq_queue_t g_assignedtasks[CONFIG_SMP_NCPUS]; + +Where ``CONFIG_SMP_NCPUS`` is the configured number of CPUs supported +by the processors. As its name suggests, on ``g_assignedtasks`` queue for +``CPU n`` would contain only tasks/threads that are assigned to CPU n. +Threads would be assigned a particular CPU by one of two mechanisms: + +1. (Semi-)permanently through an RTOS interfaces such as + ``pthread_attr_setaffinity()``, OR +2. Temporarily through new scheduling logic. + +Tasks/threads that are assigned to a CPU via an interface like +``pthread_attr_setaffinity()`` would never go into the ``g_readytorun`` list, +but would only go into the ``g_assignedtasks[n]`` list for the CPU n to which +the thread has been assigned. +Hence, the ``g_readytorun`` list would hold only unassigned tasks/threads. + +An indication within the TCB would indicated whether or not a task/thread +is assigned to a CPU and, if so, which CPU it is assigned to. + +Scheduling logic would temporarily assign a task or thread to a CPU. +The assignment is only temporary because state data in the TCB would indicate +that the task is unassigned when, hence, it could be returned +to the ``g_readytorun`` list later. + +The assigned tasks lists lists would be prioritized. +The highest priority task, and the one currently executing on CPU n would be +the one at the head of ``g_assignedtasks[n]``. +Tasks after the active task are ready-to-run and assigned to this CPU. +The tail of this assigned task list, the lowest priority task, +is always the CPU's IDLE task. + +The CPU n scheduling logic would execute whenever the currently running task +is removed from the head of ``g_assignedtasks[n]``. +The algorithm might be something like: + +.. code-block:: c + + /* Is the assigned task list for the CPU empty? */ + + if (g_assignedtasks[cpu].head == NULL) + { + /* No.. Is the task at the head of the assigned list for the CPU lower + * in priority that the current (unassigned) task at the head of the + * ready-to-run list? + */ + + FAR struct tcb_s *rtcb = (FAR struct tcb_s *)g_readytorun.head ; + FAR struct tcb_s *atcb = (FAR struct tcb_s *)g_assignedtasks[cpu].head; + if (atcb->sched_priority < rtcb->sched_priority) + { + /* Remove the TCB from the head of the g_readytorun list. */ + + /* Add that TCB to the g_assignedtasks[cpu] list (it will go at the + * head of the list). + */ + } + + /* Now activate the task at the head of the g_assignedtasks[cpu] list on + * the CPU. + */ + + } + +The Current Task +---------------- + +There is a lot of logic in the RTOS now that obtains the TCB for the currently +excuting task by examining the head of the ``g_readytorun`` list. +You will see this assignment in many places, both in the core OS logic +in ``nuttx/sched`` but also in architecture-specific logic under +``nuttx/arch``: + +.. code-block:: c + + FAR struct tcb_s *rtcb = this_task(); + +Where ``this_task()`` is a macro defined in ``nuttx/sched/sched.h`` +and expands as follows: + +.. code-block:: c + + #define current_task(cpu) ((FAR struct tcb_s *)g_readytorun.head) + #define this_cpu() (0) + #define this_task() (current_task(this_cpu)) + +Of course, that would not work with the proposed changes. +We would need to then get the TCB of the currently executing task/thread +for CPU n from the head of ``g_assignedtasks[n]``. +I would propose a replacing the above assignment with a macro like +``current_task()`` where that macro might expand to: + +.. code-block:: c + + #ifdef CONFIG_SMP + # define current_task(cpu) ((FAR struct tcb_s *)g_assignedtasks[cpu].head) + # define this_cpu() up_cpu_index() + #else + # define current_task(cpu) ((FAR struct tcb_s *)g_readytorun.head) + # define this_cpu() (0) + #endif + #define this_task() (current_task(this_cpu)) + +where ``up_cpu_index()`` is some new MCU specific interface that will +return an index associated with the currently active CPU. + +.. note:: + + This is a two step operations: Step 1. Get the CPU number and + Step 2: Use the CPU number as an index into the + ``g_assignedtasks[]`` array of lists. **This must be atomic!** + The schedule should be locked to assure that the task + is not suspended after fetching the CPU number then restarted + on a different CPU to access the ``g_assignedtasks[]`` array + of lists. + +The IDLE Task +------------- + +Without SMP, the ``g_readytorun`` list always ends with the TCB of IDLE task. + +It is always guaranteed to be at the end of the list because the list +is prioritized and because the IDLE task has an impossibly low priority +that no other task/thread could have. + +The IDLE task is necessary because it gives the CPU something to execute +when there is nothing else to be done. + +But with SMP, there are multiple CPUs that need something to do when there +is nothing else to do. We are tentatively thinking that each CPU needs its own +IDLE thread whose TCB would reside at the end of each +``g_assignedtasks[cpu]`` list. But that does feel wasteful + +I am not certain the mechanism as of this writing, but I assume that +the ``nx_start()`` initialization logic would need to create an IDLE task +for each CPU and assign each IDLE task to each CPU. + +CPU Index +--------- + +In order to access arrays indexed by a CPU ID value, some method must be +generated to provide the CPU ID that the currently executing task +is running on. To provide this index value, an interface +``up_cpu_index()`` is proposed. + +For ARM, the implementation of ``up_cpu_index()`` can be accomplished +by reading the CP15 Multiprocessor Affinity Register (MPDIR). +That register has a 2 bit field index provides exactly the index that +we need for the SMP implementation. + +Looking at how Linux does this, Linux uses an interface called ``get_cpu()`` +which is analogous to the proposed ``up_cpu_index()``. +``get_cpu()`` maps to ``smp_processor_id()`` and if debug options are +not enabled, this further maps to ``raw_smp_procesor_id()``. +For the case of ARM, this maps to ``(current_thread_info()->cpu)`` +where ``current_thread_info()`` is a location at the far end of the +allocated stack: +``(current_stack_pointer & ~(THREAD_SIZE-1))`` and +``THREAD_SIZE`` is ``(PAGE_SIZE << THREAD_SIZE_ORDER)``. + +So, to make that long story short, Linux solves the problem by putting +some magic information at the base of far end of each stack when +a context switch occurs (and when the CPU is also known). +That magic information can then just be recovered using the thread's +stack pointer at any time. This is part of the basic implementation +of Thread Local Storage (TLS) in Linux. + +Something similar could be done with NuttX and would require: + +1. Special aligned stack allocation, +2. Logic to write the CPU index into the stack when each thread + is [re-]started. + +This would also place an upper limit on the size of the stack: +If we are going to find the far end of the stack by simply ANDing out +the lower bits, then size of that mask would also determine +the maximum size of the stack. + +However, I believe that using the information from the MPIDR register +is a better general solution. Counter-arguments are: + +1. There may be some architectures that do not have such a simple mechanism. +2. TLS has value in any event. +3. The stack-based TLS is in user-accessible memory and could be used + by applications in protected and kernel builds. + + +System Startup +============== + +I assume that initially, only one CPU is active. +System initialization would then occur on that single thread. +At the completion of the initialization of the OS, just before beginning +normal multitasking, the additional CPUs would be started. + +Each CPU would be provided the entry point to is IDLE task when started. +Perhaps the MCU interface would be something like: + +.. code-block:: c + + int up_cpu_start(int cpu, main_t idletask); + +The OS initialization logic would call this function repeatedly +until each CPU is started. + + +Scheduler Interactions +====================== + +In the general case, the scheduler should have full control over the current +state of all tasks. It must make that that if there are N CPUs that the top N +highest priority tasks are running. + +Srict priority scheduling is the requirement, but perhaps the scheduling logic +could do some load balancing to distribute work as evenly as possible +over the CPUs. When a new task or thread becomes ready to run, +the scheduler must include some heuristics for assigning that task to a CPU +to achieve some optimal performance. + +There are complications to how one CPU controls the tasks already running +on another CPU. To determine a task should run, you would need to be able to: + +* Keep the task data structures stable while they are being analyzed. +* Find the lowest priority running task which could be on any CPU. +* If that priority is lower than the priority task, then replace it with + the new task at the head of the ``g_assignedtasks[]`` list. +* If not, find the task with the next lowest priority and compare that one. +* Continue until until the new task is assigned to a CPU or until + it is determined that all of the currently running tasks are higher priority + than the new task. In that base, the new task should be added + to the ``g_readytorun`` list. + +To support this behavior, I think that the following new MCU interfaces +will be needed: + +.. code-block:: c + + int up_cpu_pause(int cpu); + +Which would stop execution on CPU0, saving the state of the currently running +task so that it may be resumed. And: + +.. code-block:: c + + int up_cpu_resume(int cpu); + +Restart the CPU with the task at the head of the ``g_assignedtasks[]`` list. + +.. note:: + + Please also note the the "Signal Handling" paragraph below. + The same issue exists for dispatching signals to threads actively + running on another CPU. + + +Interrupt Handling +================== + +Per-CPU Interrupts +------------------ + +How will interrupts be taken? On one CPU or on multiple CPUs? + +This may work different on different hardware platforms. +This design requires only that: + +* If the processor supports interrupts on only one CPU, then interrupts + cannot be nested; further interrupts must be disabled while that interrupt + handler runs (see Nested Interrupts and High Priority, + Zero Latency Interrupts.). +* If the process supports device interrupts on multiple CPUs, the interrupt + handling on the CPUs is not concurrent: When interrupts are disabled + on one CPU, they are disabled on all CPUs (unless, of course, if interrupts + are needed for inter-CPU communication). + +However, I do not know of any CPU architecture that supports disabling +interrupts on one CPU from another CPU. +Instead, critical sections will need to be supported via spinlocks +as described below. + +If interrupts can be taken by multiple CPUs then any data structures used +for interrupt handling would also need to become and array indexed by the +CPU number. Most architectures current use a data structure defined like: + +.. code-block:: c + + volatile uint32_t *g_current_regs; + +Which would have to become an array like: + +.. code-block:: c + + volatile uint32_t *g_current_regs[CONFIG_SMP_NCPUS]; + + +System Calls +============ + +System Calls are normally implemented via software interrupts. + +The System Call software interrupt should run on the same CPU as does +the logic that generated the System Call or, alternatively, +the design must have some way of obtaining the index of the CPU +that generated the System Call. + + +Critical Sections +================= + +A critical section is a set of statements that must be able to execute +exclusively. Higher level applications will, of course, use OS application +interfaces such as ``sem_wait()`` and ``sem_post()`` to manage critical +sections. But within the OS, for example, in the low level implementation +of ``sem_wait()`` and ``sem_post()``, more primitive, non-standard methods +must be used to implement critical sections. + + +.. _spinlocks: + +Spinlocks +========= + +A spinlock is a lock which causes a thread trying to acquire it to simply wait +in a loop (spin) while repeatedly checking if the lock is available. +The thread remains active but is not performing a useful task. + +The use of such a lock is a kind of busy waiting and is used commonly +in SMP implementations to manage access to resources by multiple CPUs. + +Spinlock Implementation +----------------------- + +In a NuttX implementation, the spinlock would probably involve only: + +* A memory location with one value, say ``SP_LOCKED``, meaning that the lock + is taken and another value, ``SP_UNLOCKED``, meaning that the lock + is available. +* An integer type memory location that contains the number of the CPU + holding the lock. +* An integer type memory location hold the number of counts on the lock. +* And a loop performs a test-and-set operation: The memory location + is read by a thread and set to true in one atomic operation. + If the read value is false, then the thread holds the lock. + Otherwise, it must loop trying repeatedly until the thread gets the lock. + +The meaning of the lock is that CPU holding the lock has exclusive access +to a resource that is shared by multiple CPUs. +So there is never any reason for two threads on the same CPU to spin: +If the CPU already holds the lock, additional threads need simply +only increment the lock count. + +* If the test-and-set fails in the logic that is spinning, but if the lock + is held by the logic that that CPU is running on, then the spin logic + should simply increment the count of locks (which needs to be atomic only + for single processor). + +Could this cause one CPU to hog too much resource time? +Perhaps, been calls to the test-and-set logic, the spinlock should call +``sched_yield()`` which would at least let other threads +of the same priority run. + +* Releasing the lock should be matter of decrementing the lock count + and if the lock count would decrement to zero, setting the lock value to + ``SP_UNLOCKED``. This will, of course, allow another thread spinning + on the lock in a different CPU to take the lock for that CPU. + +The following new, internal OS interfaces are proposed: + +.. code-block:: c + + void spin_lock(FAR spinlock_t *lock); + void spin_unlock(FAR spinlock_t *lock); + +Where the type ``spinlock_t`` is defined in MCU-specific header files. +These new spinlock interfaces would also use the MCU-specific interface: + +.. code-block:: c + + spinlock_t up_testset(FAR spinlock_t *lock); + +.. note:: + + A thread may take the lock while running on one CPU, but then later + be assigned to a different CPU, and then release the lock while + running on that other CPU. Is there a problem in this? + Yes, probably. One solution might be lock the thread + to a CPU if it holds the lock? + +There is also a risk is that the thread holding the lock will be pre-empted +by the OS scheduler while holding the lock. If this happens, other threads +on other CPUs will be left spinning (repeatedly trying to acquire the lock), +while the thread holding the lock is not making progress towards releasing it. +The result is an indefinite postponement until the thread holding the lock +can finish and release it. + +Spinlock logic can be common. However, there must be a unique instance +of that common spinlock logic in each OS operation that requires mutually +exclusive access by a CPU. + +Now, what will we do with these spinlocks? Is there really a need for them? +Yes, probably. We will need examine every place in the OS that uses disabling +of pre-emption or disabling of interrupts to prevent other tasks +(and interrupts) from executing. + +All of those cases need to be reconsidered and, most likely, protected +with spinlocks. + +Let's next examine all of the cases of how resources are managed in NuttX. + +Spinlocks in Semaphores, Signals, and Message Queues +---------------------------------------------------- + +A critical section using ``irqsave()`` and ``irqrestore()`` is already used +in the implementation of these inter-process communications to enforce +a critical section. + +One a single CPU system, disabling interrupts will prevent context switches +(by prevent the asynchronous events that could cause a context switch) +and also prevents conflicts with interrupt level processing. + +I believe that simply replacing ``irqsave()`` and ``irqrestore()`` with +new proposed functions ``enter_critical_section()`` and +``leave_critical_section()``, as described below under Disabling Interrupts, +should be sufficient. + +These proposed functions include a spinlock to assure that they do enforce +a critical section. + +Spinlocks and Data Caches +------------------------- + +If spinlocks are used in a system with a data cache, then there may be +a problem with cache coherency in some CPU architectures. + +When one CPU modifies the spinlock, the changes may not be visible +to another CPU if it does not share the data cache. +That would cause failure in the spinlock logic. + +Flushing the D-cache on writes and invalidating before a read +is not a good option. Spinlocks are normally 8-bits in size and cache +lines are typically 32-bytes so that would have side effects unless +the spinlocks were made to be the same size as one cache line. + +The better option is to add compiler independent "ornamentation" +to the spinlock so that the spinlocks are all linked together +into a separate, non-cacheable memory regions. +Because of region alignment and minimum region mapping sizes +this could still be wasteful of memory. +This would work in systems that have both data cache and either an MPU +(such as Cortex-m7) or an MMU (such as Cortex-Ax). + + +Disabling Pre-emption +===================== + +Pre-emption is disabled via the interface ``sched_lock()``. +``sched_lock()`` currently works by preventing context switches from the +currently executing tasks. + +This prevents other tasks from running (without disabling interrupts) +and gives the currently executing task exclusive access to the (single) +CPU resources. +Thus, ``sched_lock()`` and its companion, ``sched_unlcok()``, +are used to implement some critical sections. + +Currnetly, Pre-emption is disabled using a simple lockcount in the TCB. +When the scheduling is locked, the lockcount is incremented; +when the scheduler is unlocked, the lockcount is decremented. +If the lockcount for the task at the head of the ``g_readytorun`` +list has a ``lockcount > 0``, then pre-emption is disabled. + +No special protection is required since only the executing task +can modify its lockcount. + +Certainly, disabling context switches on one CPU would still be possible +in an SMP model, but it may not be possible to give a task exclusive access +to the (multiple) CPU resources without stopping the other CPUs: +Even though pre-emption is disabled, other threads will still be executing +on the other CPUS. + +The full dynamics of the behavior of the scheduler logic in this case +is not certain. +However, I think that this would be an acceptable behavior provided that: + +* There is a global lock count ``g_cpu_lockset`` that includes a bit + for each CPU: If the bit is ``1``, then the corresponding CPU has + the scheduler locked; if ``0``, then the CPU does not have the scheduler + locked. +* Scheduling logic would set the bit associated with the cpu in + ``g_cpu_lockset`` when the TCB at the head of the + ``g_assignedtasks[cpu]`` list transitions has ``lockount > 0``. + This might happen when ``sched_lock()`` is called, or after + a context switch that changes the TCB at the head of the + ``g_assignedtasks[cpu]`` list. +* Similarly, the cpu bit in the global ``g_cpu_lockset`` would be cleared + when the TCB at the head of the ``g_assignedtasks[cpu]`` list has + ``lockount == 0``. This might happen when ``sched_unlock()`` is called, + or after a context switch that changes the TCB at the head of the + ``g_assignedtasks[cpu]`` list. +* Modification of the global ``g_cpu_lockset`` must be protected + by a simplified spinlock, ``g_cpu_schedlock``. That spinlock would be + taken when ``sched_lock()`` is called, and released when ``sched_unlock()`` + is called. This assures that the scheduler does enforce the critical + section. NOTE: Because of this spinlock, there should never be more + than one bit set in ``g_cpu_lockset`` attempts to set additional bits + should be cause the CPU to block on the spinlock. However, additional + bits could get set in ``g_cpu_lockset`` due to the context switches + on the various CPUs. +* Each the time the head of a ``g_assignedtasks[]`` list changes + and the scheduler modifies ``g_cpu_lockset``, it must also set + ``g_cpu_schedlock`` depending on the new state of ``g_cpu_lockset``. +* Logic that currently uses the currently running tasks lockcount + should instead use the global ``g_cpu_schedlock``. + A value of ``SP_UNLOCKED`` would mean that no CPU has pre-emption disabled; + ``SP_LOCKED`` would mean that at least one CPU has pre-emption disabled. + +Disabling pre-emption is a non-standard feature but the general capability +is common to many RTOS. But since feature is non-standard and perhaps +not realizable in the SMP model, another option would be +to simply eliminate it. + + +Disabling Interrupts +==================== + +Closely related to disabling pre-emption is the practice of disabling +interrupts to get exclusive access to resources. + +Disabling interrupts is not really so different from disabling +pre-emption in practice. +It effectively disables pre-emption by preventing any asynchronous +events that could cause a context switch and, of course, in addition +prevents interrupt level processing. + +So disabling of interrupts is also used in places to implement critical +sections and, because the similarity in behavior to disabling only +pre-emption, suffers from the same issues in the SMP environment. + +Currently, interrupts on the single CPU are enabled and disabled with: + +.. code-block:: c + + irqstate_t irqsave(void); + void irqrestore(irqstate_t flags); + +Those functions disable interrupts on the single CPU. +In the SMP environment, they would need to disable interrupts in all CPUs. + +.. note:: + + The legacy ``irqsave()`` and ``irqrestore()`` have been replaced + with new functions implemented in the OS, + ``enter_critical_section()`` and ``leave_critical_section()``. + +These might be implemented as follows (highly simplified): + +.. code-block:: c + + spinlock_t g_spu_irqlock = SP_UNLOCKED; + + #ifdef CONFIG_SMP + irqstate_t enter_critical_section(void) + { + irqstate_t flags = irqsave(); + + spinlock(&g_cpu_irqlock); + g_cpu_irqset |= (1 << cpu); + + return flags; + } + + void leave_critical_section(irqstate_t flags) + { + g_cpu_irqset &= ~(1 << cpu); + spinunlock(&g_cpu_irqlock); + irqrestore(flags); + } + +There is an unhandled complexities in the above simplified logic. +Consider this scenario: + +1. The thread calls ``enter_critical_section()``, disabling interrupts + on all CPUs and taking the spinlock. +2. The thread then suspends, waiting for an event. This is actually + a very standard behavior to suspend with interrupts disabled: + The system handles this gracefully be simply re-enabling interrupts + (if they were enabled by the next task to run). +3. Later, the event occurs, the task is again made ready-to-run, + and the interrupts are again disabled. + +But, + +1. There must be additional logic to release the spinlock + when the task is suspended. +2. There must be additional logic to re-acquire the spinlock + when the task restarts. +3. Is there any way that the spinlock could already be locked when the task + restarts? No, I don't think this is possible. If interrupts are disabled + and the spinlock is locked, then there should be no context switches. + +There would be additional complexities if ``enter_critical_section()`` were +called during interrupt handling. + +Interrupts are disabled during interrupt level processing, however, interrupt +level logic will attempt to establish critical sections even when +it does not need to do this: It will call ``enter_critical_section()`` anyway +because it will use some common logic with non interrupt level code. + +There are many situations in which use of spinlocks as shown +in the simplified example will result in deadlock conditions. + +As a result of these complexities, the full implementation of +``enter_critical_section()`` and ``leave_critical_section()`` are considerably +more complex. +See the logic in the file ``sched/irq/irq_csection.c`` if you are +really interested in the details. + + +Pre-Emption Controls and Critical Sections +========================================== + +The effect of disabling pre-emption is to prevent to tasks from running +while on task has disabled pre-emption; the effect of entering a critical +section, on the other hand, is to: + +1. Enforce exclusive access to the logic when in the critical section. +2. Keep the system stable while certain operations are performed. +3. Disable competing interrupt level activity when possible. + +In order to keep the system stable within in the critical section +it is necessary, the critical section will modify the behavior of the +pre-emption controls. +The basic result is this modification is that new tasks are not permitted +to be started or resumed if: + +1. Pre-emption is disabled, OR +2. Some other CPU other than the current CPU is in a critical section. + +The CPU that has entered the critical section must have the ability +to start and stop tasks. Attempts to start new tasks from other CPUs when +one CPU is within the critical section is will result in the newly started +task being postponed in a pending task list, ``g_pendingtasks``. + +Such pending tasks will only be allowed to run when: + +1. All CPUs have re-enabled pre-emption, AND +2. All CPUs have left the critical section. + +.. note:: + + It can be determined which CPU(s) have the critical section + by examining ``g_cpu_irqset``. + + +Signal Handlers +=============== + +There will be some issues related to how signals are delivered, +at least in regard to how signal handlers are executed. + +I am thinking of the case where a signal is sent by a thread running +on one CPU to a thread running on another CPU that has a signal handler +installed. This would probably have to work as follows: + +1. Stop the CPU on which the task is running ``using up_cpu_pause()``, +2. Schedule the signal action as is done in the existing logic, then +3. Re-start the CPU with ``up_cpu_resume()`` to resume execution with + the signal handler. + +A special wrapper function for ``up_cpu_pause()`` is provided in the OS +to support this operation: + +.. code-block:: c + + int sched_tcb_pause(FAR struct tcb_s *tcb); + +This function checks if the task associated with tcb is running on another CPU +and, if so, conditionally calls ``up_cpu_pause()`` to pause execution +on that CPU. It returns the CPU index of the paused CPU (or a negated +``errno`` value if no CPU was paused). While the CPU is paused, operations +can be performed on the data structures associated with the task. +Then the non-negative CPU index can then be used with +``up_cpu_resume()`` to restart the paused CPU. + +This same sequence would have to be followed for other functions +that might need to modify the behavior of a running task such as +``task_delete()`` or ``task_restart()``. + + +Thread Affinity +=============== + +By default, a thread may run on any CPU. +There are some semi-standard interfaces that can be used to restrict +the set of CPUs that a thread may run on. +This set of CPUs is referred to as the threads affinity mask. +Semi-standard meaning used on Linux and available in ``GLIBC`` when +``__GNU_SOURCE`` is defined. + +There are interfaces to set and get the affinity mask for a task prototyped +in ``sched.h``. + +``cpusetsize`` is fixed in NuttX and must be equal to ``sizeof(cpu_set_t)``: + +.. code-block:: c + + #ifdef CONFIG_SMP + int sched_setaffinity(pid_t pid, size_t cpusetsize, + FAR const cpu_set_t *mask); + int sched_getaffinity(pid_t pid, size_t cpusetsize, FAR cpu_set_t *mask); + #endif + +There are similar interfaces for a ``pthread`` prototyped in ``phtread.h``: + +.. code-block:: c + + #ifdef CONFIG_SMP + int pthread_setaffinity_np(pthread_t thread, size_t cpusetsize, + FAR const cpu_set_t *cpuset); + int pthread_getaffinity_np(pthread_t thread, size_t cpusetsize, + FAR cpu_set_t *cpuset); + #endif + +.. note:: + + The ``_np`` in the naming is to remind you that + this **interface is non-POSIX**! + +By default, a child task or pthread inherits the affinity mask of its parent. +The thread affinity mask for a ``pthread``, however, can also be set before +the thread is started via ``pthread_create()``: + +.. code-block:: c + + #ifdef CONFIG_SMP + + int pthread_attr_setaffinity_np(FAR pthread_attr_t *attr, + size_t cpusetsize, + FAR const cpu_set_t *cpuset); + + int pthread_attr_getaffinity_np(FAR const pthread_attr_t *attr, + size_t cpusetsize, cpu_set_t *cpuset); + + #endif + +In addition, macros are defined in the header file ``include/sched.h`` +to abstract operations are CPU sets. +There are several such macros with names like ``CPU_ZERO()``, ``CPU_SET()``, +``CPU_CLR()``, etc. diff --git a/Documentation/implementation/syslog.rst b/Documentation/implementation/syslog.rst new file mode 100644 index 00000000000..ae4b681d9c5 --- /dev/null +++ b/Documentation/implementation/syslog.rst @@ -0,0 +1,794 @@ +.. _syslog: + +====== +SysLog +====== + +Standard SysLog Interfaces +========================== + +The NuttX SYSLOG is an architecture for getting debug and status information +from the system. The syslogging interfaces are defined in the header file +``include/syslog.h``. + +The primary interface to SYSLOG sub-system is the function ``syslog()`` and, +to a lesser extent, its companion ``vsyslog()``: + +.. code-block:: c + + /**************************************************************************** + * Name: syslog and vsyslog + * + * Description: + * syslog() generates a log message. The priority argument is formed by + * ORing the facility and the level values (see include/syslog.h). The + * remaining arguments are a format, as in printf and any arguments to the + * format. + * + * The NuttX implementation does not support any special formatting + * characters beyond those supported by printf. + * + * The function vsyslog() performs the same task as syslog() with the + * difference that it takes a set of arguments which have been obtained + * using the stdarg variable argument list macros. + * + ****************************************************************************/ + + int syslog(int priority, FAR const IPTR char *format, ...); + int vsyslog(int priority, FAR const IPTR char *src, va_list ap); + + +The additional ``setlogmask()`` interface can use use to filter +SYSLOG output: + +.. code-block:: c + + /**************************************************************************** + * Name: setlogmask + * + * Description: + * The setlogmask() function sets the logmask and returns the previous + * mask. If the mask argument is 0, the current logmask is not modified. + * + * The SYSLOG priorities are: LOG_EMERG, LOG_ALERT, LOG_CRIT, LOG_ERR, + * LOG_WARNING, LOG_NOTICE, LOG_INFO, and LOG_DEBUG. The bit corresponding + * to a priority p is LOG_MASK(p); LOG_UPTO(p) provides the mask of all + * priorities in the above list up to and including p. + * + * Per OpenGroup.org "If the maskpri argument is 0, the current log mask + * is not modified." In this implementation, the value zero is permitted + * in order to disable all syslog levels. + * + * REVISIT: Per POSIX the syslog mask should be a per-process value but in + * NuttX, the scope of the mask is dependent on the nature of the build: + * + * Flat Build: There is one, global SYSLOG mask that controls all output. + * Protected Build: There are two SYSLOG masks. One within the kernel + * that controls only kernel output. And one in user-space that controls + * only user SYSLOG output. + * Kernel Build: The kernel build is compliant with the POSIX requirement: + * There will be one mask for for each user process, controlling the + * SYSLOG output only form that process. There will be a separate mask + * accessable only in the kernel code to control kernel SYSLOG output. + * + ****************************************************************************/ + + int setlogmask(int mask); + + +These are all standard interfaces as defined at https://www.OpenGroup.org. + + +Debug Interfaces +================ + +**In NuttX syslog output is really synonymous to debug output** and, +therefore, the debugging interface macros defined in the header file +``include/debug.h`` are also syslogging interfaces. +Those macros are simply wrappers around ``syslog()``. + +.. note:: Debug here means "system log" rather than on-chip-debug. + +The debugging interfaces differ from the syslog interfaces in that: + +* They do not take a priority parameter; the priority is inherent + in the debug macro name. +* They decorate the output stream with information such as the file name. +* They can each be disabled via configuration options. + +Each debug macro has a base name that represents the priority and a prefix +that represents the sub-system. +Each macro is individually initialized by both priority and sub-system. +For example, ``uerr()`` is the macro used for error level messages +from the USB subsystem and is enabled with ``CONFIG_DEBUG_USB_ERROR``. + +The base debug macro names, their priority, and configuration variable +are summarized below: + +* ``info()``: The ``info()`` macro is the lowest priority (``LOG_INFO``) + and is intended to provide general information about the flow of program + execution so that you can get an overview of the behavior of the program. + ``info()`` is often very chatty and voluminous and usually more information + than you may want to see. The ``info()`` macro is controlled via + ``CONFIG_DEBUG+subsystem+INFO``. +* ``warn()``: The ``warn()`` macro has medium priority (``LOG_WARN``) + and is controlled by ``CONFIG_DEBUG+subsystem+WARN``. The ``warn()`` is + intended to note exceptional or unexpected conditions that might be + potential errors or, perhaps, minor errors that easily recovered. +* ``err()``: This is a high priority debug macro (``LOG_ERROR``) + and controlled by ``CONFIG_DEBUG+subsystem+ERROR``. + The ``err()`` is reserved to indicate important error conditions. +* ``alert()``: The highest priority debug macro (``LOG_EMERG``) + and is controlled by ``CONFIG_DEBUG_ALERT``. The ``alert()`` macro + is reserved for use solely by assertion and crash handling logic. + It also differs from the other macros in that it is global + and cannot be enabled or disabled per subsystem. + + +SYSLOG Channels +=============== + +SYSLOG Channel Interfaces +------------------------- + +In the NuttX SYSLOG implementation, the underlying device logic the supports +the SYSLOG output is referred to as a SYSLOG channel. +Each SYSLOG channel is represented by an interface defined +in ``include/nuttx/syslog/syslog.h``: + +.. code-block:: c + + /* This structure provides the interface to a SYSLOG device */ + + typedef CODE int (*syslog_putc_t)(int ch); + typedef CODE int (*syslog_flush_t)(void); + + struct syslog_channel_s + { + /* I/O redirection methods */ + + syslog_putc_t sc_putc; /* Normal buffered output */ + syslog_putc_t sc_force; /* Low-level output for interrupt handlers */ + syslog_flush_t sc_flush; /* Flush buffered output (on crash) */ + + /* Implementation specific logic may follow */ + }; + +The channel interface is instantiated by calling ``syslog_channel()``: + +.. code-block:: c + + /**************************************************************************** + * Name: syslog_channel + * + * Description: + * Configure the SYSLOGging function to use the provided channel to + * generate SYSLOG output. + * + * Input Parameters: + * channel - Provides the interface to the channel to be used. + * + * Returned Value: + * Zero (OK)is returned on success. A negated errno value is returned + * on any failure. + * + ****************************************************************************/ + + int syslog_channel(FAR const struct syslog_channel_s *channel); + +``syslog_channel()`` is a non-standard, internal OS interface +and is not available to applications. +It may be called numerous times as necessary to change channel interfaces. +By default, all system log output goes to console (``/dev/console``). + +SYSLOG Channel Initialization +----------------------------- + +The initial, default SYSLOG channel is established with statically initialized +global variables so that some level of SYSLOG output may be available +immediately upon reset. +This initialized data is in the file ``drivers/syslog/syslog_channel.c``. + +The initial SYSLOG capability is determined by the selected SYSLOG channel: + +* In-Memory Buffer (**RAMLOG**). Full SYSLOG capability as available at reset. +* **Serial Console**: If the serial implementation provides the low-level + character output function ``up_putc()``, then that low level serial output + is available as soon as the serial device has been configured. +* For all other SYSLOG channels, all SYSLOG output goes to the bit-bucket + (discarded) until the SYSLOG channel device has been initialized. + +The syslog channel device is initialized when the bring-up logic +calls ``syslog_intialize()``: + +.. code-block:: c + + /**************************************************************************** + * Name: syslog_initialize + * + * Description: + * One power up, the SYSLOG facility is non-existent or limited to very + * low-level output. This function is called later in the initialization + * sequence after full driver support has been initialized. It installs + * the configured SYSLOG drivers and enables full SYSLOGing capability. + * + * This function performs these basic operations: + * + * - Initialize the SYSLOG device + * - Call syslog_channel() to begin using that device. + * + * If CONFIG_ARCH_SYSLOG is selected, then the architecture-specific + * logic will provide its own SYSLOG device initialize which must include + * as a minimum a call to syslog_channel() to use the device. + * + * Input Parameters: + * phase - One of {SYSLOG_INIT_EARLY, SYSLOG_INIT_LATE} + * + * Returned Value: + * Zero (OK) is returned on success; a negated errno value is returned on + * any failure. + * + ****************************************************************************/ + + #ifndef CONFIG_ARCH_SYSLOG + int syslog_initialize(enum syslog_init_e phase); + #else + # define syslog_initialize(phase) + #endif + + +Different types of SYSLOG devices have different OS initialization +requirements. Some are available immediately at reset, some are available +after some basic OS initialization, and some only after OS is fully +initialized. +In order to satisfy these different initialization requirements, +``syslog_initialize()`` is called twice from the boot-up logic: + +1. ``syslog_initialize()`` is called from the architecture-specific + ``up_initialize()`` function as some as basic hardware resources + have been initialized: Timers, interrupts, etc. + In this case, ``syslog_initialize()`` is called with the argument + ``SYSLOG_INIT_EARLY``. +2. ``syslog_initialize()`` is called again from ``nx_start()`` when + the full OS initialization has completed, just before the application + main entry point is spawned. In this case, ``syslog_initialize()`` + is called with the argument ``SYSLOG_INIT_LATE``. + +There are other types of SYSLOG channel devices that may require even further +initialization. For example, the file SYSLOG channel (described below) +cannot be initialized until the necessary file systems have been mounted. + +Interrupt Level SYSLOG Output +----------------------------- + +As a general statement, SYSLOG output only supports normal output from NuttX +tasks. However, for debugging purposes, it is also useful to get SYSLOG +output from interrupt level logic. +In an embedded system, that is often where the most critical operations +are performed. + +There are three conditions under which SYSLOG output generated from interrupt +level processing can a included the SYSLOG output stream: + +1. Low-Level Serial Output. +2. In-Memory Buffering. +3. Serialization Buffer. + +The SYSLOG interrupt buffer is enabled with ``CONFIG_SYSLOG_INTBUFFER``. +When the interrupt buffer is enabled, you must also provide the size +of the interrupt buffer with ``CONFIG_SYSLOG_INTBUFSIZE``. + +Low-Level Serial Output +^^^^^^^^^^^^^^^^^^^^^^^ + +If you are using a SYSLOG console channel (``CONFIG_SYSLOG_CONSOLE``) +with a serial console (``CONFIG_SYSLOG_SERIAL_CONSOLE``) and if the underlying +architecture supports the low-level ``up_putc()`` interface +(``CONFIG_ARCH_LOWPUTC``), then the SYLOG logic will direct the output +to ``up_putc()`` which is capable of generating the serial output +within the context of an interrupt handler. + +There are a few issues in doing this however: + +1. ``up_putc()`` is able to generate debug output in any context because + it disables serial interrupts and polls the hardware directly. + These polls may take many milliseconds and during that time, all interrupts + are disable within the interrupt handler. This, of course, interferes with + the real-time behavior of the RTOS. +2. The output generated by ``up_putc()`` is immediate and in real-time. + The normal SYSLOG output, on the other hand, is buffered in the serial + driver and may be delayed with respect to the immediate output by many + lines. Therefore, the interrupt level SYSLOG ouput provided throug + ``up_putc()`` is grossly out of synchronization with other debug output. + +In-Memory Buffering +^^^^^^^^^^^^^^^^^^^ + +If the RAMLOG SYSLOG channel is supported, then all SYSLOG output is buffered +in memory. Interrupt level SYSLOG output is no different than normal SYSLOG +output in this case. + +Serialization Buffering +^^^^^^^^^^^^^^^^^^^^^^^ + +A final option is the use the an interrupt buffer to buffer the interrupt +level SYSLOG output. In this case: + +1. SYSLOG output generated from interrupt level process in not sent + to the SYSLOG channel immediately. Rather, it is buffered in the + interrupt serialization buffer. +2. Later, when the next normal syslog output is generated, it will + first empty the content of the interrupt buffer to the SYSLOG device + in the proper context. It will then be followed by the normal syslog + output. In this case, the interrupt level SYSLOG output will interrupt + the normal output stream and the interrupt level SYSLOG output will + be inserted into the correct position in the SYSLOG output when + the next normal SYLOG output is generated. + + +SYSLOG Channel Options +====================== + +SYSLOG Console Device +--------------------- + +The typical SYSLOG device is the system console. +If you are using a serial console, for example, then the SYSLOG output +will appear on that serial port. + +This SYSLOG channel is automatically selected by ``syslog_initialize()`` +in the LATE initialization phase based on configuration options. +The configuration options that affect this channel selection include: + +* ``CONFIG_DEV_CONSOLE``: This setting indicates that the system supports + a console device, i.e., that the character device ``/dev/console`` exists. +* ``CONFIG_SERIAL_CONSOLE``: This configuration option is automatically + selected when a UART or USART is configured as the system console. + There is no user selection. +* ``CONFIG_SYSLOG_CONSOLE``: This configuration option is manually selected + from the SYSLOG menu. This is the option that acutally enables the SYSLOG + console device. It depends on ``CONFIG_DEV_CONSOLE`` and it will + automatically select ``CONFIG_SYSLOG_SERIAL_CONSOLE`` if + ``CONFIG_SERIAL_CONSOLE`` is selected. +* ``CONFIG_ARCH_LOWPUTC``: This is an indication from the architecture + configuration that the platform supports the ``up_putc()`` interface. + ``up_putc()`` is a very low level UART interface that can even be used + from interrupt handling. +* ``CONFIG_SYSLOG_SERIAL_CONSOLE``: This enables certain features + of the SYSLOG operation that depend on a serial console. + If ``CONFIG_ARCH_LOWPUTC`` is also selected, for example, + then ``up_putc()`` will be used for the forced SYSLOG output. + +Interrupt level SYSLOG output will be lost unless: + +1. The interrupt buffer is enabled to support serialization, or +2. A serial console is used and ``up_putc()`` is supported. + +.. note:: + + The console channel uses the fixed character device at + ``/dev/console``. The console channel is not synonymous with + ``stdout`` (or file descriptor ``1``). ``stdout`` is the + current output from a task when, say, ``printf()`` if used. + Initially, ``stdout`` does, indeed, use the ``/dev/console`` + device. However, ``stdout`` may subsequently be redirected + to some other device or file. + This is always the case, for example, when a transient device + is used for a console – such as a USB console or a Telnet + console. + The SYSLOG channel is not redirected as ``stdout`` is; the SYSLOG + channel will stayed fixed (unless it is explicitly changed + via ``syslog_channel()``). + +References: ``drivers/syslog/syslog_consolechannel.c`` and +``drivers/syslog/syslog_device.c``. + + +SYSLOG Character Device +----------------------- + +The system console device, ``/dev/console``, is a character driver with +some special properties. However, any character driver may be used as the +SYSLOG output channel. For example, suppose you have a serial console +on ``/dev/ttyS0`` and you want SYSLOG output on ``/dev/ttyS1``. +Or suppose you support only a Telnet console but want to capture +debug output ``/dev/ttyS0``. + +This SYSLOG device channel is selected with ``CONFIG_SYSLOG_CHAR`` and has +no other dependencies. Differences fromthe SYSLOG console channel include: + +1. ``CONFIG_SYSLOG_DEVPATH``: This configuration option string must be set + provide the full path to the character device to be used. +2. The forced SYSLOG output always goes to the bit-bucket. + This means that interrupt level SYSLOG output will be lost unless + the interrupt buffer is enabled to support serialization. +3. ``CONFIG_SYSLOG_CHAR_CRLF``: If ``CONFIG_SYSLOG_CHAR_CRLF`` is selected, + then inefeeds in the SYSLOG output will be expanded to + Carriage Return + Linefeed. Since the character device is not a console + device, the addition of carriage returns to line feeds would + not be performed otherwise. + You would probably want this expansion if you use a serial terminal + program with the character device output. + +References: ``drivers/syslog/syslog_devchannel.c`` and +``drivers/syslog/syslog_device.c``. + +SYSLOG File Device +------------------ + +Files can also be used as the sink for SYSLOG output. +There is, however, a very fundamental difference in using a file as opposed +the system console, a RAM buffer, or character device: +You must first mount the file system that supports the SYSLOG file. +That difference means that the file SYSLOG channel cannot be supported during +the boot-up phase but can be instantiated later when board level logic +configures the application environment, including mounting of the file systems. + +The interface ``syslog_file_channel()`` is used to configure +the SYSLOG file channel: + +.. code-block:: c + + /**************************************************************************** + * Name: syslog_file_channel + * + * Description: + * Configure to use a file in a mounted file system at 'devpath' as the + * SYSLOG channel. + * + * This tiny function is simply a wrapper around syslog_dev_initialize() + * and syslog_channel(). It calls syslog_dev_initialize() to configure + * the character file at 'devpath then calls syslog_channel() to use that + * device as the SYSLOG output channel. + * + * File SYSLOG channels differ from other SYSLOG channels in that they + * cannot be established until after fully booting and mounting the target + * file system. This function would need to be called from board-specific + * bring-up logic AFTER mounting the file system containing 'devpath'. + * + * SYSLOG data generated prior to calling syslog_file_channel will, of + * course, not be included in the file. + * + * NOTE interrupt level SYSLOG output will be lost in this case unless + * the interrupt buffer is used. + * + * Input Parameters: + * devpath - The full path to the file to be used for SYSLOG output. + * This may be an existing file or not. If the file exists, + * syslog_file_channel() will append new SYSLOG data to the end of the + * file. If it does not, then syslog_file_channel() will create the + * file. + * + * Returned Value: + * Zero (OK) is returned on success; a negated errno value is returned on + * any failure. + * + ****************************************************************************/ + + #ifdef CONFIG_SYSLOG_FILE + int syslog_file_channel(FAR const char *devpath); + #endif + + +References: ``drivers/syslog/syslog_filechannel.c``, +``drivers/syslog/syslog_device.c``, and ``include/nuttx/syslog/syslog.h``. + +SYSLOG RAMLOG Device +-------------------- + +The RAMLOG is a standalone feature that can be used to buffer any character +data in memory. There are, however, special configurations that can be used +to configure the RAMLOG as a SYSLOG channel. +The RAMLOG functionality is described in a more general way +in the following paragraphs. + +RAM Logging Device +^^^^^^^^^^^^^^^^^^ + +The RAM logging driver is a driver that was intended to support debugging +output (SYSLOG) when the normal serial output is not available. +For example, if you are using a Telnet or USB serial console, +the debug output will get lost – or worse. +For example, what if you want to debug the network over Telnet? + +The RAM logging driver can also accept debug output data from interrupt +handler with no special serialization buffering. +As an added benefit, the RAM logging driver is much less invasive. +Since no actual I/O is performed with the debug output is generated, +the RAM logger tends to be much faster and will interfere much less +when used with time critical drivers. + +The RAM logging driver is similar to a pipe in that it saves the debugging +output in a circular buffer in RAM. +It differs from a pipe in numerous details as needed to support logging. + +This driver is built when CONFIG_RAMLOG is defined in the Nuttx configuration. + +dmesg +^^^^^ + +When the RAMLOG (with SYSLOG) is enabled, a new NuttShell (NSH) command +will appear: ``dmesg``. The dmsg command will dump the contents of the +circular buffer to the console (and also clear the circular buffer). + +RAMLOG Configuration options +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +* ``CONFIG_RAMLOG`` - Enables the RAM logging feature. +* ``CONFIG_RAMLOG_CONSOLE`` - Use the RAM logging device as a system console. + If this feature is enabled (along with ``CONFIG_DEV_CONSOLE``), + then all console output will be re-directed to a circular buffer in RAM. + This might be useful, for example, if the only console is a Telnet console. + Then in that case, console output from non-Telnet threads will go to the + circular buffer and can be viewed using the NSH ``dmesg`` command. + This optional is not useful in other scenarios. +* ``CONFIG_RAMLOG_SYSLOG`` - Use the RAM logging device for the syslogging + interface. If this feature is enabled, then all debug output will be + re-directed to the circular buffer in RAM. This RAM log can be viewed + from NSH using the ``dmesg`` command. + NOTE: Unlike the limited, generic character driver SYSLOG device, + the RAMLOG can be used to capture debug output from interrupt level handlers. +* ``CONFIG_RAMLOG_NPOLLWAITERS`` - The number of threads than can be waiting + for this driver on ``poll()``. Default: 4. + +If ``CONFIG_RAMLOG_CONSOLE`` or ``CONFIG_RAMLOG_SYSLOG`` is selected, +then the following must also be provided: + +* ``CONFIG_RAMLOG_BUFSIZE`` - The size of the circular buffer to use. + Default: 1024 bytes. + +Other miscellaneous settings: + +* ``CONFIG_RAMLOG_CRLF`` - Pre-pend a carriage return before every linefeed + that goes into the RAM log. +* ``CONFIG_RAMLOG_NONBLOCKING`` - Reading from the RAMLOG will never block + if the RAMLOG is empty. If the RAMLOG is empty, then zero is returned + (usually interpreted as end-of-file). If you do not define this, the NSH + ``dmsg`` command will lock up when called! So you probably do want this! +* ``CONFIG_RAMLOG_NPOLLWAITERS`` - The maximum number of threads that + may be waiting on the poll method. + + +SYSLOG (Input) Character Device +=============================== + +If the option ``CONFIG_SYSLOG_CHARDEV`` is selected then support for a special +character device at ``/dev/syslog`` is supported. +The function ``syslog_register()`` can be used to register +that character device: + +.. code-block:: c + + /**************************************************************************** + * Name: syslog_register + * + * Description: + * Register a simple character driver at /dev/syslog whose write() method + * will transfer data to the SYSLOG device. This can be useful if, for + * example, you want to redirect the output of a program to the SYSLOG. + * + * NOTE that unlike other syslog output, this data is unformatted raw + * byte output with no time-stamping or any other SYSLOG features + * supported. + * + ****************************************************************************/ + + void syslog_register(void); + + +.. note:: Careful... there is overloaded naming here! + +A character device can serve as the SYSLOG output channel as described above. +There we were referring to a data path like:: + + syslog() -> SYSLOG channel layer -> Character driver output channel. + +Here we are talking about a different SYSLOG character devices that provides +input data to the SYSLOG channel. +That is a data path like:: + + SYSLOG Character driver -> SYSLOG channel layer -> Output channel. + +Very confusing and begs for some re-naming. + + +Using SYSLOG for Debug +====================== + +Lately, I have starting thinking a little more about the ``SYSLOG`` functions +in general (as well as all of those debug macros). I think I am coming to the +conclusion that there needs to be some things done. + +The original design of ``syslog()`` was just some hooks for simple serial +debug output. These simple debug hooks were standardized and renamed +``syslog()`` so that they provide a portable debug interface. +As NuttX has increased in complexity and sophistication over the years, +the implementation of ``syslog()`` "under the hood" is still that mindlessly +simple serial debug logic from the original NuttX release. +Perhaps the time as come to assess what is wrong with the solution +and to implement some improvements in the ``syslog()`` design? + +Here are some of the issues of the current implementation that bother me: + +Use of File Descriptors +----------------------- + +In the default case, the debug output goes out on file descriptor ``1`` +(``stdout``). But if you think about that, it is a little crazy. + +Each task can have I/O redirected in various ways. In most simple systems, +a serial console is used for stdout in all tasks and the debug output goes +to the serial console so everything is seamless. But if you are using +a mixture of serial consoles, USB serial consoles, telnet sessions, etc. +then who know were where the output is going to go in the most general case. +Bits and pieces could go to different devices. + +If you are redirecting stdout to a file, for example, then the debug +information could go into your file, corrupting the output that you wanted. +As another perverse example, try enabling network debug output using a Telnet +session. That is an interesting exercise for anyone who like to see +infinite loops: Telnet I/O generates debug output, the debug output goes +to the Telnet network connection, which generates more network debug output, +and on and one. + +Interrupt Handlers +------------------ + +Output from interrupt handlers, of course, cannot use the console device +at all. File descriptor I/O is not permitted from interrupt handlers. +Attempts to do "normal" SYSLOG output from interrupt handlers will +just result in the output going to the bit bucket. + +Low-Level Serial Driver +----------------------- + +The usual workaround to get debug output from interrupt handlers is to use +the low-level serial I/O from interrupt handlers. +But there are issues with this as well: + +* First, the console may not be the same serial device. + It might be something else althogether. That means that non-interrupt + SYSLOG output goes output one way and interrupt level SYSLOG output + always goes out the serial port. + Potentially very strange behavior could result. +* Second, the low-level serial output does a busy wait poll! + That interferes badly with the behavior of the interrupt handler – + it has to wait within the interrupt handler while the serial output + is performed. That wait would will be many milliseconds with + interrupts disabled! +* The low level serial output also interacts the serial driver itself + making other use of the console impossible. Even, in some cases + on some platforms, locking up the serial driver. + +Interrupt Buffer +---------------- + +Another way to handle interrupt level output is supported. +This is the only other option available if a Serial Console is not being used. +This second option is enabled with ``CONFIG_SYSLOG_INTBUFFER``. + +In this case, syslog output generated from interrupt level logic will simply +be buffered in memory. Then, later, when the next non-interrupt level syslog +output is generated, the buffer interrupt level output will performed. +This works because it essentially defers the syslog output generated +from interrupt handlers until the next opportunity to perform normal output. + +Asynchronous Output +------------------- + +Another syslog related issue is the asynchronous behavior with debug output +from interrupt handlers. The normal debug output goes to the serial driver +and is buffered for sending there. The size of the serial RX buffer +is configurable. At any given point in time, the current output is behind +realtme depending on that buffer size and the serial BAUD. + +The interrupt debug output does not use the serial driver but immdiately +commandeers the serial port and outputs dara in real time. +The result is it appears to happen earlier in the output. + +This is also why you lose the last debug output on a crash... +the last debug data is stranded in the serial drivers RX buffer. + +Interleaved Output +------------------ + +Because the output is done character at a time, the debug output +from different tasks may get multiplexed and unreadable in the most critical +of cases. The interleaved output can become totally useless, usually +in the most complex situations where you need the debug output the most. +Many times my plans to debug a problem with SYSLOG output has been thwarted +because the output is just uninterpretable in a highly multitasking context. + +Buffer Overrun +-------------- + +The root cause of this problem is the RX buffering in the serial driver: +Character output is done one character at a time. That is not usually +an issue. But when the system is very busy and he serial RX buffer becomes +full, then each character output causes the caller to suspending, +waiting for space in the RX buffer. It suspends and is moved to the last +of the FIFO for that priority. + +When this happens, many tasks may be suspended waiting space to put the next +byte in the serial RX buffer. If the tasks are the same priority, then +the output will be interleaved as described since each task gets essentially +round-robin access to the serial driver. If one of the tasks is lower +priority, then its its output may be deferred for some time until all +of the higher priority tasks complete their output. +Again, leaving a big time skew in the output data. + +When there is is a big time skew in the debug output – whether from +asynchronous output from interrupt handlers or from blocked, +lower priority tasks – This can lead to misinterpretation of the +debug output since our instinct is to treat the output as if it were +in sequential order in time. + +.. note:: + + Although a simple working around for a partical debug scenario + is simply to increase the size of the RX buffer in the serial + driver. Assuming that the RX buffer overrun is only the result + of short-term, bursty behavior, then the large buffer migh prevent + tasks blocking waiting for space to write the next byte. + +Of course, there is no work around for the perverse case where the debug +output is generated at a higher rate than can be transferred +on the serial port. You are just basically out-of-luck in that case. + +Solutions +--------- + +Serialization Buffer +^^^^^^^^^^^^^^^^^^^^ + +Some of these asynchrony problems could also be reduced or eliminated +if all debug output were buffered in an in-memory FIFO. That buffer +would serialize output from diffrent sources: +Various tasks and interrupt level logic. + +There is already a outgoing, RX buffer in the serial driver. +Why would an additional layer of buffering help? Only because then +the interrupt handler debug output could also be serialized. +The input to the FIFO is debug output from all concurrent tasks and interrupt +handlers; the single output of the FIFO would be the serial driver. + +Serialization via syslog buffer has been recently implemented in NuttX, +this option is enabled with ``CONFIG_SYSLOG_BUFFER``. + +Crash Dump +^^^^^^^^^^ + +It might even be possible to flush that serialization buffer at the time +of the crash. This has ability has not yet been implemented +as of the time of this writing. + +CONFIG_SYSLOG and the RAMLOG +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +There is a partial solution for all of these issues when ``CONFIG_SYSLOG=y``. +In that case, stdout is not used but instead some custom logic is used +that is established by the configuration. Currently that option is only used +with the ``RAMLOG``. The RAMLOG actually works very nicely and eliminates +most of the above issue (except for the interleaving issue). +But the ``RAMLOG`` also requires some additional logic to get the debug +information out of the ``RAMLOG``. In ``NSH``, you can use the dmesg command +to dump the content of the ``RAMLOG`` to the ``NSH`` console. + +But I am thinking that some in-memory buffering and serialization +is the solution to most of the issues mentioned about: +That would end the reliance on stdout. It would be compatible with +debug output from interrupt handlers; interrupt handler output +would be serialized in the in-memory buffer in the correct position. + +The same interleaving problem could still occur. Adding more buffering +can always eliminate the interleaving problem, provided that the problem +is due to bursty, high-volume output. But you can just increase the size +of the serial RX buffer to accomplish that; any special serialization +is of no help (other than it effectively increases the size of the RX buffer). + +The existing ``RAMLOG`` code could be extended so that the buffered data +is agressively dumped to the ``SYSLOG`` device and I think all problems +would be eliminated. The buffer could be dumped that the end of each +non-interrupt level execution of ``syslog()`` (and also on a crash). +Interrupt level output would have to pend in the buffer until the next, +non-interrupt level debug output pushes it out to the serial driver. diff --git a/Documentation/implementation/task_exit_sequence.png b/Documentation/implementation/task_exit_sequence.png new file mode 100644 index 00000000000..5bc27c70cb6 Binary files /dev/null and b/Documentation/implementation/task_exit_sequence.png differ diff --git a/Documentation/implementation/tasks_vs_threads.rst b/Documentation/implementation/tasks_vs_threads.rst new file mode 100644 index 00000000000..4bd14efb6f4 --- /dev/null +++ b/Documentation/implementation/tasks_vs_threads.rst @@ -0,0 +1,217 @@ +.. _tasks-vs-threads: + +================= +Tasks vs. Threads +================= + +What is the difference between a thread and a task in Nuttx? +============================================================ + +Tasks and threads in NuttX try to emulate processes and threads +in the standard Unix environment. +I think of a process as a "container" of resources that are shared +by the threads that execute within the context of the process. + +The process has one special thread, the "main" thread. +This is the special thread that is started when the process was created. + +Since there are no processes in NuttX, the distinction is not so simple. +But in a similar way, tasks under NuttX have mostly private resources: +Their own file descriptors, their own ``FILE`` streams, their own ``errno`` +variable, there own environment, etc. + +Threads under NuttX, on the other hand, share task resources in the same way +that threads within the same process share the process resources. + +The task resources were created when a task was created (``task_create()``, +``execv()``, ``posix_spawn()``, etc.). +When the task creates new ``pthreads``, those pthreads share the resources +of the parent task. +There is little to distinguish the main thread in NuttX. +All threads are essentially equal (as they were in old LinuxThreads library). + +All task resources that are shared amongst threads reside in a "break-away", +reference-counted structure called struct ``task_group_s``. +The Task Control Block (TCB) of each thread that is a member of the task group +holds a reference to the same instance of this breakaway structure +(see ``include/nuttx/sched.h``). + +The reference count of the shared, +task group resources is initialized to one when the task is created +(via ``task_create()``, for example); that task is the first member +of the task group. +The reference count is incremented when each additional thread is created +(via ``pthread_create()``) and joins task group. +The reference count equal to the number of members of the task group. + +If any thread in the task group creates a new task (vs. thread), +that establishes a new task group but has no effect on the creator's +task group. The first task creates the task group; +the group membership only grows when threads are created via +``pthread_create()``. + +As threads exit, they leave the task group and the reference count +on the struct ``task_group_s`` resources used by the thread is decremented. +When the reference count goes to zero, there are no members in the task group +and so the task group resources are finally deleted. +In this way, the life of a task resource begins when the task is created +and persists until the last thread in the task group exits. + +Here is the definitive list of what is shared. Most of these resources +reside within the task group structure struct ``task_group_s``: + +* Child task exit status. +* Pthread join data. +* Environment variables. +* File descriptors. +* FILE streams. +* Sockets. +* Opened message queues. +* ``pthread`` keys. +* Support data for ``atexit()``, ``on_exit()``, and/or ``waitpid()``. + +The exception is the PIC address space used with NXFLAT. +It currently has its own data structure (struct ``dspace_s``, +but logically also belongs in ``task_group_s``): + +* PIC data space and address environments. + +.. note:: + + The ``errno`` is not one of the shared resources. + In NuttX ``errno`` is thread-private; each thread has + its own ``errno``. But for bug-for-bug compatibility, the ``errno`` + variable really should also be moved into ``task_group_s``. + +NuttX tasks and threads are discussed in much more detail in the +:ref:`nuttx-tasking` section. + + +When would I want to use a task? When would I want to use a thread? +=================================================================== + +Threads are very light weight; the memory cost for the thread is the cost +of the task control block plus the size of the stack. +That is perhaps 0.8-2KB, depending mostly on the configured stack size. + +The memory cost of a task is higher because it includes most of the costs +of a pthread plus the cost of the task group structure. +The cost of the task group structure varies with such things as: + +* The number of file descriptors you have configured (controlled by + ``CONFIG_NFILE_DESCRIPTORS``), +* The number of streams that you have configured for C buffered I/O + (controlled by ``CONFIG_NFILE_STREAMS``), +* The size of the I/O buffer configured for for each stream + (controlled by ``CONFIG_STDIO_BUFFER_SIZE``). This setting is especially + important because there is an I/O buffer for each opened stream. +* The number of sockets that you have configured + (controlled by ``CONFIG_NSOCKET_DESCRIPTORS``). +* The size of dynamic data such as environment variables + (size is determined by your usage). + +The size of task group can then become largely depending upon how these things +are configured, typically in the range 0.5-1KB. + +So why would you ever use a task if pthreads use so much less memory? +**Pthreads and tasks share resources. So they are not independent.** +There is strong coupling between the threads in a task group +and what one thread does can affect the behavior of another thread. + +Normally, each major block of functionality is implemented with a separate +task. With each of those ``_task group_s``, however, you may also want several +helper threads to implement asynchronous and concurrent behaviors. + + +How do signals work in a task group with many pthreads? +======================================================= + +The behavior of signals in the multi-thread task group is complex. +NuttX emulates a process model with task ``group_s`` and follows the POSIX +rules for signalling behavior. + +Normally when you signal the ``_task`` group you would signal using +the task ID of the main task that created the group (in practice, a different +task should not know the IDs of the internal threads created within +the task group); that ID is remembered by the task group +(even if the main task thread exits). + +Here are some of the things that should happen when you signal +a multi-threaded task group: + +* If a task group receives a signal then one and only one indeterminate thread + in the task group which is not blocking that signal will receive the signal. +* If a task group receives a signal and more than one thread is waiting + on that signal, then one and only one indeterminate thread out of that + waiting group will receive the signal. + +You can mask out that signal using ``sigprocmask()`` +(or ``pthread_sigmask()``). That signal will then be effectively disabled +and will never be received in those threads that have the signal masked. +On creation of a new thread, the new thread will inherit the signal mask +of the parent thread that created it. + +So you if block signal signals on one thread then create new threads, +those signals will also be blocked in the new threads as well. + +You can control which thread receives the signal by controlling +the signal mask. +You can, for example, create a single thread whose sole purpose +it is to catch a particular signal and respond to it: +Simply block the signal in the main task; then the signal will be blocked +in all of the pthreads in the group too. + +In the one "signal processing" pthread, enable the blocked signal. +This thread will then be only thread that will receive the signal. + + +How do ``atexit()`` and ``on_exit()`` work with task groups? +============================================================ + +``atexit()`` and ``on_exit()`` callbacks must be registered using the task ID +of the main task (which makes sense since pthreads do not have task IDs +of type ``pid_t``). +The callback is not necessarily made when the task thread exits; +the ``atexit()`` and ``on_exit()`` callbacks will be executed when +the task group terminates, that is, when the final thread +of the task group terminates. + + +How does ``waitpid()`` work with task groups? +============================================= + +In a single-thread task group, ``waitpid()`` will wait until the single, +main thread of the task group exits (i.e., the one created +by ``task_start()``). This is the intuitive behavior. +But the behavior may be less intuitive for multi-threaded task groups. +In a multi-threaded task group ``waitpid()`` will wait until +every thread in the task group exits. +Nothing special happens when the main thread of the task group exits. + + +What are privileged threads? How are threads handled differently when NuttX is built as a Kernel +================================================================================================ + +NuttX supports a build mode where it is built as a monolithic kernel. +This mode is selected with the configuration option +``CONFIG_BUILD_PROTECTED=y`` and is currently only supported +for a few architectures. + +When used this way, NuttX is built as a separate kernel mode "blob" +and the applications are built as another separate user mode "blob". +The kernel runs in kernel mode and the applications run in user mode +(with the MPU restricting user mode accesses). +Access to the kernel from the user blob is only via system calls (SVCalls). + +Thread and tasks that execute within the user mode "blob" +are all unprivileged, user mode threads. Exactly what "unprivileged" +means depends upon the memory protection architecture. +But it generally means that there are regions of memory where unprivileged +threads are prohibited from reading and/or writing data and/or from executing +code. Tasks created in the user-space are unprivileged; +all pthreads are unprivileged. + +Certain threads are created within the kernel space to perform OS housekeeping +operations. Those are referred to as "kernel threads". +They are essentially the same as user-mode tasks but run in a privileged mode +and have full access to all of the restricted resources. diff --git a/Documentation/implementation/tickless_os.rst b/Documentation/implementation/tickless_os.rst new file mode 100644 index 00000000000..d33f2e3c148 --- /dev/null +++ b/Documentation/implementation/tickless_os.rst @@ -0,0 +1,375 @@ +.. _tickless: + +=========== +Tickless OS +=========== + +Default System Timer +==================== + +By default, a NuttX configuration includes a periodic timer interrupt +that drives all system timing. +The timer is provided by architecture-specific code that calls into NuttX +at a rate controlled by ``CONFIG_USEC_PER_TICK``. +The default value of ``CONFIG_USEC_PER_TICK`` is 10000 microseconds which +corresponds to a timer interrupt rate of 100Hz. + +On each timer interrupt, NuttX does these things: + +* Increments a counter. This counter is the system time and has a resolution + of ``CONFIG_USEC_PER_TICK`` microseconds. +* Checks if it is time to perform time-slice operations on tasks that + have select round-robin scheduling. +* Checks for expiration of timed events. + +What is wrong with this default system timer? Nothing really. +It is reliable and uses only a small fraction of the CPU band width. +But we can do better. Some limitations of default system timer are, +in increasing order of importance: + +1. **Overhead:** Although the CPU usage of the system timer interrupt + at 100Hz is really very low, it is still mostly wasted processing time. + On most timer interrupts, there is really nothing that needs be done + other than incrementing the counter. +2. **Resolution:** Resolution of all system timing is also determined + by ``CONFIG_USEC_PER_TICK``. So nothing can be timed with resolution + finer than 10 milliseconds by default. If you want to increase this + resolution, then you can reduce ``CONFIG_USEC_PER_TICK``. + However, then the system timer interrupts use more of the CPU bandwidth + processing useless interrupts. +3. **Power Usage:** But the biggest issue is power usage. + When the system is IDLE, it enters a light, low-power mode (for ARMs, + this mode is entered with the ``wfi`` instruction for example). + But each interrupt awakens the system from this low power mode. + Therefore, higher rates of interrupts cause greater power consumption. + + +Tickless OS Advantages +====================== + +The so-called **Tickless OS** provides one solution to issue. +The basic concept here is that the periodic, timer interrupt is eliminated +and replaced with a one-shot, interval timer. +It becomes event driven instead of polled: The default system timer +is a polled design. On each interrupt, the NuttX logic checks if it needs +to do anything and, if so, it does it. + +Using an interval timer, you can anticipate when the next interesting +OS event will occur, program the interval time and wait for it to fire. +When the interval time fires, then the scheduled activity is performed. + + +Platform Support +================ + +In order to use the Tickless OS, you must provide special support +from the platform-specific code. Just as with the default system timer, +the platform-specific code must provide the timer resources to support +the OS behavior. + +Currently these timer resources are only provided on a few platforms. +You can see that implementation for the simulation in +``nuttx/arch/sim/src/up_tickless.c``. There is another implementation +for the Atmel SAMA5 at ``nuttx/arch/arm/src/sama5/sam_tickless.c``. +This Wiki will explain how you can provide the Tickless OS support +for your platform. + + +Configuration Options +===================== + +* ``CONFIG_ARCH_HAVE_TICKLESS``: If your platform provides support + for the Tickless OS, then you should set this option in the ``Kconfig`` + file for your board. Here is what the selection looks in the + ``arch/Kconfig`` file for the simulated platform:: + + config ARCH_SIM + bool "Simulation" + select ARCH_HAVE_TICKLESS + ---help--- + Linux/Cywgin user-mode simulation. + +* ``CONFIG_SCHED_TICKLESS``: If ``CONFIG_ARCH_HAVE_TICKLESS`` is selected, + then you will be able to use this option to enable the Tickless OS features + in NuttX. +* ``CONFIG_SCHED_TICKLESS_ALARM``: The tickless option can be supported either + via a simple interval timer (plus elapsed time) or via an alarm. + The interval timer allows programming events to occur after an interval. + With the alarm, you can set a time in the future and get an event when + that alarm goes off. This option selects the use of an alarm. + The advantage of an alarm is that it avoids some small timing errors; + the advantage of the use of the interval timer is that the hardware + requirement may be less. +* ``CONFIG_USEC_PER_TICK``: This is not a new option, but changes + its relevance when the Tickless OS is selected. + +In the default configuration where system time is provided by a periodic timer +interrupt, the default system timer is configure the timer for 100Hz or +``CONFIG_USEC_PER_TICK=10000``. If ``CONFIG_SCHED_TICKLESS`` is selected, +then there are no system timer interrupt. In this case, +``CONFIG_USEC_PER_TICK`` does not control any timer rates. +Rather, it only determines the resolution of time reported by +``clock_systimer()`` and the resolution of times that can be set for certain +delays including watchdog timers and delayed work. + +In this case there is still a trade-off: +It is better to have the ``CONFIG_USEC_PER_TICK`` as low as possible +for higher timing resolution. However, the the time is currently held +in ``unsigned int``. On some systems, this may be ``16-bit`` in width +but on most contemporary systems it will be ``32-bit``. +In either case, smaller values of ``CONFIG_USEC_PER_TICK`` will reduce +the range of values that delays that can be represented. +So the trade-off is between range and resolution (you could also modify +the code to use a 64-bit value if you really want both). + +.. note:: Recently NuttX switched to ``int64_t`` for time stamps. + +The default, 100 microseconds, will provide for a range of delays +up to 120 hours. + +This value should never be less than the underlying resolution of the timer. +Error may ensue. + + +System Interfaces +================= + +Implementation Notes +-------------------- + +Notice that with the ``CONFIG_SCHED_TICKLESS`` option, an implementation +might require two hardware timers: +(1) An interval timer to satisfy the requirements for ``up_timer_start()`` +and ``up_timer_cancel()``, and +(2) a counter to handle the requirement of ``up_timer_gettime()``. + +Since timers are a limited resource, the use of two timers could be an issue +on some systems. The job could be done with a single timer if, for example, +the single timer were kept in a free-running at all times. +Some timer/counters have the capability to generate a compare interrupt when +the timer matches a comparison value but also to continue counting without +stopping. If your hardware supports such counters, one might used the +``CONFIG_SCHED_TICKLESS_ALARM`` option and be able to simply set the +comparison count at the value of the free running timer PLUS the desired +delay. Then you could have both with a single timer: +An alarm and a free-running counter with the same timer! + +Imported Intefaces +------------------ + +The interfaces that must be provided by the platform specified code +are defined in ``include/nuttx/arch.h`` and summarized below. + +``up_timer_intialize()`` +^^^^^^^^^^^^^^^^^^^^^^^^ + +.. code:: c + + void up_timer_initialize(void); + +* **Description:** Initializes all platform-specific timer facilities. + This function is called early in the initialization sequence by + ``up_intialize()``. On return, the current up-time should be available + from ``up_timer_gettime()`` and the interval timer is ready for use + (but not actively timing). +* **Input Parameters:** None. +* **Returned Value:** None. +* **Assumptions:** Called early in the initialization sequence before + any special concurrency protections are required. + +``up_timer_gettime()`` +^^^^^^^^^^^^^^^^^^^^^^ + +.. code:: c + + int up_timer_gettime(FAR struct timespec *ts); + +* **Description:** Return the elapsed time since power-up (or, more correctly, + since ``up_timer_initialize()`` was called). This function is functionally + equivalent to ``clock_gettime()`` for the clock ID ``CLOCK_MONOTONIC``. + This function provides the basis for reporting the current time and also + is used to eliminate error build-up from small errors + in interval time calculations. +* **Input Parameters:** ``*ts`` - Provides the location in which + to return the up-time. +* **Returned Value:** Zero (``OK``) is returned on success; + a negated ``errno`` value is returned on any failure. +* **Assumptions:** Called from the the normal tasking context. The implementation must provide whatever mutual exclusion is necessary for correct operation. This can include disabling interrupts in order to assure atomic register operations. + +``up_alarm_cancel()`` +^^^^^^^^^^^^^^^^^^^^^ + +.. code:: c + + int up_alarm_cancel(FAR struct timespec *ts); + +* **Description:** Cancel the alarm and return the time of cancellation + of the alarm. These two steps need to be as nearly atomic as possible. + ``sched_timer_expiration()`` will not be called unless the alarm + is restarted with ``up_alarm_start()``. If, as a race condition, + the alarm has already expired when this function is called, + then time returned is the current time. +* **Input Parameters:** ``*ts`` - Location to return the expiration time. + The current time should be returned if the timer is not active. + ``ts`` may be ``NULL`` in which case the time is not returned. +* **Returned Value:** Zero (``OK``) is returned on success; + a negated ``errno`` value is returned on any failure. +* **Assumptions:** May be called from interrupt level handling or from the + normal tasking level. Interrupts may need to be disabled internally + to assure non-reentrancy. + +.. note:: This function is only required when ``CONFIG_SCHED_TICKLESS_ALARM`` + is defined. + +``up_alarm_start()`` +^^^^^^^^^^^^^^^^^^^^ + +.. code:: c + + int up_alarm_start(FAR const struct timespec *ts); + +* **Description:** Start the alarm. ``sched_timer_expiration()`` will be + called at alarm expires (unless ``up_alarm_cancel()`` is called to stop + the alarm. +* **Input Parameters:** ``*ts`` - The time in the future at the alarm + is expected to occur. When the alarm occurs the timer logic will call + ``ched_timer_expiration()`` is called. +* **Returned Value:** Zero (``OK``) is returned on success; + a negated ``errno`` value is returned on any failure. +* **Assumptions:** May be called from interrupt level handling + or from the normal tasking level. Interrupts may need to be disabled + internally to assure non-reentrancy. + +.. note:: This function is only required when ``CONFIG_SCHED_TICKLESS_ALARM`` + is defined. + +``up_timer_cancel()`` +^^^^^^^^^^^^^^^^^^^^^ + +.. code:: c + + int up_timer_cancel(FAR struct timespec *ts); + +* **Description:** Cancel the interval timer and return the time remaining + on the timer. These two steps need to be as nearly atomic as possible. + ``sched_timer_expiration()`` will not be called unless the timer + is restarted with ``up_timer_start()``. If, as a race condition, + the timer has already expired when this function is called, + then that pending interrupt must be cleared so that + ``up_timer_start()`` and the remaining time of zero should be returned. +* **Input Parameters:** ``*ts`` - Location to return the remaining time. + Zero should be returned if the timer is not active. ``ts`` may be ``NULL`` + in which case the time remainging is not returned +* **Returned Value:** Zero (``OK``) is returned on success; + a negated ``errno`` value is returned on any failure. +* **Assumptions:** May be called from interrupt level handling + or from the normal tasking level. Interrupts may need to be disabled + internally to assure non-reentrancy. + +.. note:: This function is only required when ``CONFIG_SCHED_TICKLESS_ALARM`` + is not defined. + +``up_timer_start()`` +^^^^^^^^^^^^^^^^^^^^ + +.. code:: c + + int up_timer_start(FAR const struct timespec *ts); + +* **Description:** Start the interval timer. ``sched_timer_expiration()`` + will be called at the completion of the timeout (unless + ``up_timer_cancel()`` is called to stop the timing. +* **Input Parameters:** ``*ts`` - Provides the time interval until + ``sched_timer_expiration()`` is called. +* **Returned Value:** Zero (``OK``) is returned on success; + a negated ``errno`` value is returned on any failure. +* **Assumptions:** May be called from interrupt level handling or from the + normal tasking level. Interrupts may need to be disabled internally + to assure non-reentrancy. + +.. note:: This function is only required when ``CONFIG_SCHED_TICKLESS_ALARM`` + is not defined. + + +Exported Interfaces +------------------- + +In addition, the following interface is provided by the RTOS for use +by the platform specific code: + +``sched_alarm_expiration()`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. code:: c + + void sched_timer_expiration(FAR const timespec *tc); + +* **Description:** if ``CONFIG_SCHED_TICKLESS`` is defined, then this + function is provided by the RTOS base code and called from + platform-specific code when the alaram used to implement + the tick-less OS expires. +* **Input Parameters:** ``*ts`` - The time when the alarm expired. +* **Returned Value:** None. +* **Assumptions:** Base code implementation assumes that this function + is called from interrupt handling logic with interrupts disabled. + +.. note:: This function is only provided when ``CONFIG_SCHED_TICKLESS_ALARM`` + is defined. + +``sched_timer_expiration()`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. code:: c + + void sched_timer_expiration(void); + +* **Description:** if ``CONFIG_SCHED_TICKLESS`` is defined, then this + function is provided by the RTOS base code and called from + platform-specific code when the interval timer used to implement + the tick-less OS expires. +* **Input Parameters:** None. +* **Returned Value:** None. +* **Assumptions:** Base code implementation assumes that this function + is called from interrupt handling logic with interrupts disabled. + +.. note:: This function is only provided when ``CONFIG_SCHED_TICKLESS_ALARM`` + is not defined. + + +Issues +====== + +Let me describe an experience to illustrate some of the problems +that you may run into: + +I implemented the tickless mode on an MCU with ``16-bit`` timers. +Most of the clock sources were too fast for use with the 16-bit timer, +so I tried using a 32.768 kHz clock source. +This resulted in a time resolution of about 30.518 microseconds. + +That time, 30.518 microseconds, can not be represented accurately with +``CONFIG_USEC_PER_TICK`` so I used the value 31 which is in error +by about 0.6%. + +What I found on a busy system is that the delay I get when executing +this NSH command was actually about 5.5 seconds:: + + nsh> sleep 10 + +This was very puzzling to me and cost me a lot of debug time. +Finally, I disabled all competing usage of the interval timer by disabling +features (the high priority work queue, by the way, is the most significant +user of the interval timer). +After disabling all competing usage, the NSH command sleep 10 resulted +in a delay of about 10.3 seconds. Still not accurate but not so bad. + +I concluded that the gross inaccuracy in the first case was due to +the inaccuracies in the representation of the clock rate. +30.518 usec cannot be represented accurately. +Each timing calculation results in a small error. +When the interval timer is very busy, long delays will be divided +into many small pieces and each small piece has a large error +in the calculation. The cumulative error is the cause of the problem. + +.. caution:: The moral of this story: Do not use timer sources + that cannot be represented by ``CONFIG_USEC_PER_TICK``. diff --git a/Documentation/implementation/tls.rst b/Documentation/implementation/tls.rst new file mode 100644 index 00000000000..0c57b0ca9d7 --- /dev/null +++ b/Documentation/implementation/tls.rst @@ -0,0 +1,358 @@ +.. _thread-local-storage: + +========================= +TLS: Thread Local Storage +========================= + +Historical Background +===================== + +Thread Local Storage (TLS) was originally implemented to support +pthread-specific data and a per-thread ``errno`` variable. + +``errno`` +========= + +The user ``errno`` value was originally kept in the TCB and could be accessed +via an OS system call called ``__errno()``: + +.. code-block:: c + + # define errno *__errno() + +The ``errno`` value was kept in the thread's TCB in the ``pterrno`` field. +The ``__errno()`` function finds the TCB associated with the current thread +and returns a pointer to that errno storage location. + +This worked in the FLAT build mode because it dereferenced the address +of the errno in TCB as both an ``RVALUE``: + +.. code-block:: c + + errcode = errno; + +And as an ``LVALUE``: + +.. code-block:: c + + errno = errcode; + +That works, however, it is rather inefficient. + +However, that will not work at all in the PROTECTED or KERNEL build modes +because the memory holding the TCB is protected and cannot be directly +accessed from the user mode application. +Instead, it is accessed through accessor functions: + +.. code-block:: c + + void set_errno(int errcode); + int get_errno(void); + +Accessing the ``errno`` via these functions in PROTECTED or KERNEL mode +is a huge performance hit, because the calls must go through a system call +which is implemented as a software interrupt. + +And in this case, the errno is defined to be: + +.. code-block:: c + + # define errno get_errno() + +This works fine as an ``RVALUE``: + + errcode = errno; + +But will cause a compilation error if used as an ``LVALUE``: + +.. code-block:: c + + errno = errcode; + +Instead user code must explicitly call ``set_errno(errcode)``. +This a a violation of the accepted usage of the POSIX ``errno`` variable. + +TLS-Based ``errno`` +------------------- + +The current solution has moved the ``errno`` storage location out of the +protected TCB memory and into the unprotected application thread's stack +memory. Then it can be accessed with the appropriate TLS +(Thread Local Storage) interfaces. + + +Task-Specific Globals in the FLAT and PROTECTED builds +====================================================== + +A special case is the use of TLS to support task-specific +(vs. thread specific) global data. + +Task-specific differs from thread-specific in that all of the threads +in the task group share the global data, however, each task group has +its own set of task-specific global variables. + +This feature permits a high-fidelity emulation of process global data +as you would see in the KERNEL build where each process has its own copy +of all global variables. + +Task-specific data is implemented by keeping such globals in the stack +of the main thread. A single main thread is always present and, hence, +provides user-accessible, per-task storage. + +Task-specific data is accessed like TLS except that the task-specific data +is accessed from the stack of the main thread, +rather than the stack stack of the main thread. + + +Re-Entrant ``getopt()`` +======================= + +One use of task-specific data is for global variables used by the ``getopt()`` +function. ``getopt()`` is an important C library function used by applications +for parsing of task command line parameters. + +It is, however, not re-entrant due to the use of global variables: +``optarg``, ``opterr``, ``optind``, and ``optopt``. +There are several, non-standard implementations for a re-entrant version +of called ``getopt_r()``, however, these are all wildly different +and do not conform to any standard. + +.. important:: Non-standard interfaces are not desirable in NuttX. + +In the FLAT and PROTECTED builds, this non-reentrant limitation +poses a problem. Unlike the KERNEL build mode, there is one instance +of of the globals ``optarg``, ``opterr``, ``optind``, and ``optopt`` +shared across all task groups. +It is not a problem in the KERNEL build where there is a separate instance +of the globals in each process address space. + +Using task-specific data, however, it is possible to make ``getopt()`` +thread-safe. This amounts to keeping the ``getopt()`` globals in the main +thread's TLS so that there is an individual copy for each thread. +That would keep both the standard form of the ``getopt()`` interfaces +as well as making ``getopt()`` full re-entrant with respect to task groups. + + +Unaligned TLS +============= + +Historically, TLS worked by aligning the stack base address then +simply AND'ing the current stack pointer to obtain the base address +of the stack where the TLS data can be found (as struct ``tls_info_s``). +This mechanism is very efficient because no OS system call is required, +as application logic can obtain the TLS data directly form its stack +via AND'ing and casting. + +This works well in the KERNEL build mode where the stack resides at highly +aligned virtual address, but does not work well in FLAT and PROTECTED modes: + +1. The alignment must be large. That is because it also determines + the maximum size of the thread's stack. If the size of the thread's stack + exceeds the maximum value determined by the alignment, then + the AND operation will alias to the wrong address. +2. If all of the addresses are highly aligned then + (1) you need to have much more memory available for stack allocation. + This is because (2) the large alignment causes bad memory fragmentation + and degraded use of memory. + +An alternative way to get the stack base address is to call into the OS +to get the unaligned stack base address. +This involves a system call, but is more usable that other alternatives +in the FLAT and PROTECTED modes. + +We have implemented the configuration: ``CONFIG_TLS_ALIGNED`` that selects +the legacy aligned stack for TLS access. If this is not defined, then +the new unaligned stack TLS logic is used. +We have also implemented aligned and unaligned TLS support +for every architecture. + +Addition implementation steps: + +1. Moved the ``errno`` storage location out of the TCB and in TLS + (into the struct ``tls_info_s``). +2. Modified the ``errno`` access definitions and logic that was + in ``sched/errno`` to use the TLS logic in user space. + That logic now resides in ``libs/libc/errno`` since it is now + a user library interface, not a core OS interface. +3. TLS is now enabled by default. It is enabled in the unaligned mode + for the FLAT and PROTECTED build modes but in the highly efficient + aligned mode for KERNEL build mode. +4. There are no longer an OS system calls related to the error + (with the exception of a call to get the struct ``tls_info_s`` + in the unaligned TLS mode). + + +Push-Up vs. Push-Down Stacks +============================ + +TLS data is always located at the beginning thread's stack. +This is true for both CPUs with push-up stacks and CPUs with push-down stacks. +This location required in order to access the TLS by ANDing the aligned stack +pointer address. +The stack memory maps, differ only in the usage of the available stack:: + + Push Down Push Up + +-------------+ +-------------+ <- Stack memory allocation + | TLS Data | | TLS Data | + +-------------+ +-------------+ + | Task Data* | | Task Data* | + +-------------+ +-------------+ + | Arguments | | Arguments | + +-------------+ +-------------+ | + | | | | v + | Available | | Available | + | Stack | | Stack | + | | | | + | | | | + | | ^ | | + +-------------+ | +-------------+ + +\*) Task data is allocated in the main's thread's stack only + + +TLS interfaces +============== + +TLS is a non-standard, but more general interface. +It differs from pthread-specific data only in that its semantics are general; +the semantics of the pthread-specific data interfaces are focused on pthreads. +But they really should share the same common underlying logic. + +Currently, there are four TLS interfaces: + +.. code-block:: c + + int tls_alloc(void); + int tls_free(int tlsindex); + uintptr_t tls_get_value(int tlsindex); + int tls_set_value(int tlsindex, uintptr_t tlsvalue); + +as prototyped and documented in ``include/nuttx/tls.h``. + +These interfaces are basically adaptations from the Windows TLS interfaces +which are directly analogous to the POSIX pthread-specific data interfaces, +adapted to NuttX coding standards (see, for example, links available at +https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-tlsalloc +). + +There are no POSIX TLS interfaces and Linux (actual GLIBC) does not provide +a good mechanism. It relies on a storage class that is specific +to ELF binaries. That is not useful in an embedded system where +no ELF information is present. + + +pthread-specific data +===================== + +Pthread-specific data is another mechanism for accessing thread-specific data. +It consists of these POSIX standard interfaces: + +.. code-block:: c + + int pthread_key_create(FAR pthread_key_t *key, + CODE void (*destructor)(FAR void *)); + int pthread_setspecific(pthread_key_t key, FAR const void *value); + FAR void *pthread_getspecific(pthread_key_t key); + int pthread_key_delete(pthread_key_t key); + +This, historically, was separate from TLS data and managed internal +to the OS. pthread-specific data was only accessible through OS system calls. + +But it is a natural extension of the TLS logic to support both +non-standard TLS access and standard pthread-specific data access. +This would amount to: + +1. Moving the pthread-specific storage out of the TCB + and into the TLS data. +2. Moving the pthread-specific data accessors out of ``sched/pthread`` + and into ``libs/libc/pthread``. +3. Remove the pthread-specific data system calls. + + +Accessing the ``errno`` from kernel space +========================================= + +.. important:: We should never access the errno from kernel space. + +This is especially dangerous from kernel space because we can't be certain +which user address environment is in effect or which stack is being used +(if we were to use separate kernel mode stacks in the future as Linux does, +for security reasons, etc). + +Most OS interfaces have two parts: + +1. A user callable function, say ``osapi()`` that sets the ``errno`` value + is necessary (and may implement cancellation points) and +2. An internal OS version which might then be ``nx_osapi()`` which + does not modify the ``errno`` value. + +Ideally, all of the user callable OS interfaces (the ``osapi()``) should be +moved to the location in ``libs/libc`` the system call (``sycall/``) should be +redirected to ``nx_osapi()``. + +A complication to doing this is that the OS interfaces which are also +cancellation points call special internal interfaces, +``enter_cancellation_point()`` and ``leave_cancellation_point()`` +that are not available from user space. +So some addition partitioning would be need to accomplish this. + +This separation of user and OS interfaces has not been implemented +as of this writing. + + +Wild Ideas +========== + +The primary beneficiary of TLS is the C library. +**The TLS interfaces are non-standard and should not be used directly +by any portable application code.** +However, these non-standard TLS interfaces can be used within the C library +to implement improved, standard user interfaces. + +Local Process ID +---------------- + +Another data item that should, eventually, be included in the TLS data +is the process ID (``pid``) of the currently executing thread. +In some analysis of PROTECTED and KERNEL builds, it was found that +``getpid()`` was the most highly accessed OS interface. +By moving the PID into TLS, we could eliminate this system call overhead +(at least in the aligned TLS case). +The same mechanism would be used by ``pthread_self()`` which, in NuttX, +would be the equivalent function, but following pthread semantics. + +Streams +------- + +In NuttX, C buffered I/O streams are represented with a pre-group array +of type FILE. A stream is accessed from the OS via the +``nxched_get_streams()`` interface (system call). +The streams array is not used by OS and would be another candidate +to move into TLS. + +.. note:: In retrospect, this is not such a good idea because the I/O streams + are not unique per-thread; they are common across all threads within + a task group: That includes main thread of the group and all child + pthreads whose parent, grandparent, of great-grandparent is the + main thread. They all share the same I/O stream array. + As a result, I/O streams must be one per task-group. + + This works now because the I/O stream array is a part + of the internal OS group structure which is one per task group. + While it would be nice to get the I/O stream around out of the OS, + TLS is probably not the right solution to do that. + + +Code References +=============== + +Things to look at: + +* ``include/errno.h`` - Defines current errno access. +* ``include/nuttx/tls.h`` - Defines the tls_info_s structure. +* ``include/nuttx/sched.h`` - Group related TLS structures. +* ``libs/libc/errno`` - The TLS-based errno logic. +* ``libs/libc/tls`` - The implementation of the most TLS interfaces. +* ``sched/group`` - Group-related implementation of certain TLS interfaces. +* ``libs/libc/pthread`` - The implementation of the pthread-specific dta interfaces. + diff --git a/Documentation/implementation/usb.rst b/Documentation/implementation/usb.rst new file mode 100644 index 00000000000..65523c6950b --- /dev/null +++ b/Documentation/implementation/usb.rst @@ -0,0 +1,147 @@ +========================== +USB (Universal Serial Bus) +========================== + +NAKing USB OUT/IN Tokens +------------------------ + +USB supports a NAKing mechanism to pace incoming OUT data from the USB host. +The USB device class driver can control this NAKing only indirectly +by the manner in which it manages read requests. + +Similarly, USB will refuse to perform the USB host's requests for IN data +by NAKing the host's IN tokens. + +.. note:: USB direction naming is host-centric. This makes life difficult for + people working USB device drivers: + IN refers to data coming from the device INto the host. + But for the device, this is a write operation (write being + peripheral-centric: from-memory-to-peripheral). + OUT refers to data sent from the host OUT to the device. + For the device this is a read (peripheral-to-memory) operation. + + +NAKing USB OUT Tokens +^^^^^^^^^^^^^^^^^^^^^ + +Below is how the NAKing is implemented in the interaction between the USB +device class driver and USB Device Controller Drivers (DCDs): + +1. At initialization time when the USB device class driver is enumerated + by the host, the class driver will allocates a number of read requests + (which contain request buffers). +2. The USB device class driver sends each allocated read request to the device + DCD via ``DRVR_EPSUBMIT()``. +3. Upon receipt, the DCD will keep all of these empty read requests in a list. +4. When the OUT token is received what happens next depends upon the state + of the list of empty read requests: + + 4a. If there are no requests in the list of empty read requests, the DCD + will have configured the endpoint to NAK the OUT token. This NAKing + will continue until the list of read requests is non-empty and the DCD + disables the NAKing to signal that it can accept the OUT packet data. + + 4b. If the list of read requests is not empty, the DCD will remove the next + read request from the list, receive the DATA accompanying the OUT + token, copy the received packet data into the read request's buffer, + and return the filled reqd request to the USB device class driver. + The data is returned using a callback function pointer that is + contained inside of the read request. + +5. When the filled read request if returned, the USB device class driver will + add the newly filled read request to the TAIL of a list of filled read + requests. This will then be available to the the driver read logic when + the application requests more data. + This step will also wake up read logic that has been suspended waiting + for the availability of incoming data. +6. The receive logic starts when the application requests data (or it is + resumed after the receipt of new data) it will look at the read request + at the HEAD of the list of filled read requests. If will return the data + payload to the application by copying it from the read request's buffer + into the application-provided receive buffer: + + 6a. If the application's receive buffer becomes full and it cannot accept + more data, the receive logic will save the offset in the data buffer + where it left off. If the application asks for more data, it will + resume transferring at the point where it left off. + + 6b. If the receive logic transfers all of the data from the receive + buffer. It will remove the read request from the HEAD of the list + of filled read requests and return the receive buffer to the driver + via ``DRVR_EPSUBMIT()`` (See step 3). + The receive logic will then continue at step 6. + + 6c. If the receive logic needs more data in order to complete + the application read operation, it will wait until it is awakened + by the receipt of a newly filled. + + +CDC/ACM driver +~~~~~~~~~~~~~~ + +The CDC/ACM driver differs from all other drivers because it is REQUIRED +to discard data if there is no application ready to receive the data. +This is how a serial device works. So there is nothing like step 5 +in the CDC/ACM driver and in step 6 the CDC/ACM driver will not remember +the point where it left off; if the application cannot accept any further +data, it will return the partially emptied read request to the DCD +immediately to be filled – dropping an untransferred data into the bit bucket. + +This is not an error or an oversight in the CDC/ACM. This is the intentional +design. This is the CORRECT behavior of the CDC/ACM driver. Data overrun can +only be prevented on a serial connection by using either XON/XOFF software +flow control or hardware flow control. A special endpoint 0 control request +is reserved for the CDC/ACM implementation. + +.. note:: As of this writing, the logic that implements the CDC/ACM + _hardware flow control has not been implemented. + There is a place holder for such logic, but the implementation + is missing._ + + +NAKing USB IN Tokens +^^^^^^^^^^^^^^^^^^^^ + +The DCD will NAK IN tokens whenever it has nothing to send to the host PC. +Without going into as much detail, suffice it to say that the logic flow +is very similar to the case off OUT tokens with the following differences: + +1. The list of empty write requests are not provided to the DCD at + initialization time, but are instead are retained in a list of empty write + requests in the USB device class driver. + +2. When the USB class driver has data to send, it will remove a write request + from this list, copy the outgoing application data into the write buffer, + and send the filled write request to the DCD via ``DRVR_EPSUBMIT()``. + If there are no available write requests, the application thread may block + waiting for the next write request to be returned from the DCD. + +3. The DCD will keep a list of filled write requests for each endpoint. + When that list becomes non-empty, the DCD will disable the NAKing and + set up to transfer data to the host. Normally this involves copying + the IN data into an outgoing FIFO or setting up some write DMA transfers: + + 3a. After the data has been transferred to the USB device hardware, + the emptied write request will be returned to the USB device class driver. + This is done through a callback function pointer within the write request. + + 3b. If the DCD's list of write requests becomes empty, it will set up + the endpoint to NAK any further IN tokens (after the final IN transfer + completes). + +4. When the USB device class driver receives the emptied write request from + the DCD, it will return the write request to its list of empty write + request and may, perhaps, wake up any thread that was waiting + for an available write request to send more data. + +So the fundamental different between IN and OUT processing is that for +IN transfers, empty write requests are saved in the USB device class driver +and then given to the DCD when they are filled with outgoing data. +For OUT transfers empty read requests are retained in the DCD and returned +to the USB device class driver when they filled with incoming data. + +Similarly, filled read requests are queued for application read processing +in the USB device class driver; filled write requests are queued for transfer +to the host in the DCD. + +But in either case, NAKing occurs when associated DCD request queue is empty. diff --git a/Documentation/reference/user/13_boardctl.rst b/Documentation/reference/user/13_boardctl.rst index e9f09849af2..6cd765862d8 100644 --- a/Documentation/reference/user/13_boardctl.rst +++ b/Documentation/reference/user/13_boardctl.rst @@ -1,3 +1,5 @@ +.. _board-ioctl: + =========== Board IOCTL ===========